Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/21463 )
Change subject: IMPALA-13106: Support larger imported query profile sizes through compression ...................................................................... Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl File www/queries.tmpl: http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@317 PS3, Line 317: : : profileParseWorker.onmessage = (e) => { : if (e.data.error) { : showImportedQueriesStatusMessage("Error parsing some JSON profiles"); : setTimeout(setScrollReload, 1500); : console.log(e.data.error); : return; : } : var profileStore = db.transaction("profiles", "readwrite").objectStore("profiles"); : profileStore.put(e.data).onsuccess = () => { : raw_total_size += json_profile_chooser.files[upload_count].size; : compressed_total_size += e.data.profile.length; : upload_count++; : if (upload_count >= json_profile_chooser.files.length) { : console.log("Raw total size : " + getReadableSize(raw_total_size, 2)); : console.log("Compressed total size : " : + getReadableSize(compressed_total_size, 2)); : console.log("Query Profile(s) Processing time : " + getReadableTimeMS(Date.now() : - query_processor_start_time)); : setTimeout(setScrollReload, 2000); : } else { : // Recursively call uploadProfile() until all selected JSON profiles : // are parsed and stored : uploadProfile(); : } : }; : }; : : dbOpenR > Is it better to move these to compressionWorker as well? I'm guessing JSON. Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@364 PS3, Line 364: lt > Use '>=' just to be safe. Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@368 PS3, Line 368: : } > "Compression time" should be exclusive for total time doing compression onl I have renamed the attribute. I think total time consumed for both parsing and compressing is more important. http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@372 PS3, Line 372: ileStore.index( > Add comment, "Recursively call uploadProfile until all selected JSON profil Done http://gerrit.cloudera.org:8080/#/c/21463/3/www/scripts/tests/queries/compressionWorker.test.js File www/scripts/tests/queries/compressionWorker.test.js: http://gerrit.cloudera.org:8080/#/c/21463/3/www/scripts/tests/queries/compressionWorker.test.js@31 PS3, Line 31: > Can you use ./impala-profiles/impala_profile_log_tpcds_compute_stats.expect Done -- To view, visit http://gerrit.cloudera.org:8080/21463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4f31beb9cac89051460bf764b6d50c3933bd03 Gerrit-Change-Number: 21463 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Comment-Date: Tue, 04 Jun 2024 11:48:28 +0000 Gerrit-HasComments: Yes
