Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21463 )
Change subject: IMPALA-13106: Support larger imported query profile sizes through compression ...................................................................... Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl File www/queries.tmpl: http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@317 PS3, Line 317: var profile = JSON.parse(e.target.result).contents; : var val = profile.profile_name; : query.id = val.substring(val.indexOf("=") + 1, val.length - 1); : query.user = profile.child_profiles[0].info_strings : .find(({key}) => key === "User").value; : query.default_db = profile.child_profiles[0].info_strings : .find(({key}) => key === "Default Db").value; : query.type = profile.child_profiles[0].info_strings : .find(({key}) => key === "Query Type").value; : query.start_time = profile.child_profiles[0].info_strings : .find(({key}) => key === "Start Time").value; : query.end_time = profile.child_profiles[0].info_strings : .find(({key}) => key === "End Time").value; : query.bytes_read = profile.child_profiles[2].counters : .find(({counter_name}) => counter_name === "TotalBytesRead").value; : query.bytes_read = getReadableSize(query.bytes_read, 2); : query.bytes_sent = profile.child_profiles[2].counters : .find(({counter_name}) => counter_name === "TotalBytesSent").value; : query.bytes_sent = getReadableSize(query.bytes_sent, 2); : query.state = profile.child_profiles[0].info_strings : .find(({key}) => key === "Query State").value; : query.rows_fetched = profile.child_profiles[1].counters : .find(({counter_name}) => counter_name === "NumRowsFetched").value; : query.resource_pool = profile.child_profiles[0].info_strings : .find(({key}) => key === "Request Pool").value; : query.statement = profile.child_profiles[0].info_strings : .find(({key}) => key === "Sql Statement").value; : if (query.statement.length > 250) { : query.statement = query.statement.substring(0, 250) + "..."; : } Is it better to move these to compressionWorker as well? I'm guessing JSON.parse() for large profiles will take long time as well. http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@364 PS3, Line 364: == Use '>=' just to be safe. http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@368 PS3, Line 368: getReadableTimeMS(Date.now() : - compression_start_time "Compression time" should be exclusive for total time doing compression only. Current calculation will include other things such as JSON parsing from L312 to L346. http://gerrit.cloudera.org:8080/#/c/21463/3/www/queries.tmpl@372 PS3, Line 372: uploadProfile() Add comment, "Recursively call uploadProfile until all selected JSON profiles are stored." http://gerrit.cloudera.org:8080/#/c/21463/3/www/scripts/tests/queries/compressionWorker.test.js File www/scripts/tests/queries/compressionWorker.test.js: http://gerrit.cloudera.org:8080/#/c/21463/3/www/scripts/tests/queries/compressionWorker.test.js@31 PS3, Line 31: generateRandomText() Can you use ./impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty_extended.json for testcase? -- To view, visit http://gerrit.cloudera.org:8080/21463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c4f31beb9cac89051460bf764b6d50c3933bd03 Gerrit-Change-Number: 21463 Gerrit-PatchSet: 3 Gerrit-Owner: Surya Hebbar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Comment-Date: Mon, 03 Jun 2024 19:00:17 +0000 Gerrit-HasComments: Yes
