Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21463 )
Change subject: IMPALA-13106: Support larger imported query profile sizes through compression ...................................................................... Patch Set 4: (8 comments) http://gerrit.cloudera.org:8080/#/c/21463/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21463/2//COMMIT_MSG@14 PS2, Line 14: pako > Done Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl File www/queries.tmpl: http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@317 PS3, Line 317: : : profileParseWorker.onmessage = (e) => { : if (e.data.error) { : showImportedQueriesStatusMessage("Error parsing some JSON profiles"); : setTimeout(setScrollReload, 1500); : console.log(e.data.error); : return; : } : var profileStore = db.transaction("profiles", "readwrite").objectStore("profiles"); : profileStore.put(e.data).onsuccess = () => { : raw_total_size += json_profile_chooser.files[upload_count].size; : compressed_total_size += e.data.profile.length; : upload_count++; : if (upload_count >= json_profile_chooser.files.length) { : console.log("Raw total size : " + getReadableSize(raw_total_size, 2)); : console.log("Compressed total size : " : + getReadableSize(compressed_total_size, 2)); : console.log("Query Profile(s) Processing time : " + getReadableTimeMS(Date.now() : - query_processor_start_time)); : setTimeout(setScrollReload, 2000); : } else { : // Recursively call uploadProfile() until all selected JSON profiles : // are parsed and stored : uploadProfile(); : } : }; : }; : : dbOpenR > Done Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@364 PS3, Line 364: lt > Done Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@368 PS3, Line 368: : } > I have renamed the attribute. Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@372 PS3, Line 372: ileStore.index( > Done Done http://gerrit.cloudera.org:8080/#/c/21463/2/www/query_stmt.tmpl File www/query_stmt.tmpl: http://gerrit.cloudera.org:8080/#/c/21463/2/www/query_stmt.tmpl@68 PS2, Line 68: db.transaction("profiles", "readonly").objectStore("profiles"); > Done Done http://gerrit.cloudera.org:8080/#/c/21463/4/www/scripts/tests/queries/profileParseWorker.test.js File www/scripts/tests/queries/profileParseWorker.test.js: http://gerrit.cloudera.org:8080/#/c/21463/4/www/scripts/tests/queries/profileParseWorker.test.js@26 PS4, Line 26: ../../.. nit: Maybe use $IMPALA_HOME env var instead? I found this: https://stackoverflow.com/questions/53041159/cant-read-file-using-nodejs-fs-module-when-file-name-is-stored-in-environment-v http://gerrit.cloudera.org:8080/#/c/21463/4/www/scripts/tests/queries/profileParseWorker.test.js@28 PS4, Line 28: import("../../../pako.min.js").then((pako) => { : pako = pako.default; : expect(pako.inflate(pako.deflate(exampleJSONProfileText, {level : 3}), {to : "string"})) : .toBe(exampleJSONProfileText); : }); Instead of just testing the compress-decompress, what if this test instantiate profileParseWorker like https://gerrit.cloudera.org/c/21463/4/www/queries.tmpl#268, and then inflate the result from Worker with inflateParseJSON function? -- To view, visit http://gerrit.cloudera.org:8080/21463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4f31beb9cac89051460bf764b6d50c3933bd03 Gerrit-Change-Number: 21463 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Comment-Date: Tue, 04 Jun 2024 21:21:27 +0000 Gerrit-HasComments: Yes
