Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20131 )
Change subject: IMPALA-12152: Add query options to wait for HMS events sync up ...................................................................... Patch Set 35: (12 comments) http://gerrit.cloudera.org:8080/#/c/20131/35//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20131/35//COMMIT_MSG@45 PS35, Line 45: A succeeded wait: : Query Compilation: 937.279ms : - Synced events from Metastore: 909.162ms (909.162ms) : - Metadata of all 1 tables cached: 911.005ms (1.843ms) : - Analysis finished: 919.600ms (8.595ms) : : A failed wait: : Query Compilation: 1s321ms : - Continuing without syncing Metastore events: 40.883ms (40.883ms) : - Metadata load started: 41.618ms (735.633us) Any test validating existence of new event timeline in query profile? http://gerrit.cloudera.org:8080/#/c/20131/35//COMMIT_MSG@85 PS35, Line 85: Currently, UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT, : OPEN_TXN Unrelated question: Do you know if updating table property will raise HMS event? If yes, how CatalogD deal with it? Asking for https://gerrit.cloudera.org/c/22474/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#2824 http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/catalog/catalog-server.cc@103 PS35, Line 103: when waiting Who is waiting? Maybe tell us something about this flag in commit message? http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/catalog/catalog-server.cc@471 PS35, Line 471: void WaitForHmsEvent(TWaitForHmsEventResponse& resp, : const TWaitForHmsEventRequest& req) override { Please add gauge / metric on how much WaitForHmsEvent RPC is currently hung up waiting. http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/service/impala-server.cc@2322 PS35, Line 2322: if (timeline != nullptr) { : timeline->MarkEvent("Detected change in catalog service ID"); : } This pattern if timeline != nullptr check followed by timeline->MarkEvent() repeated in multiple places. Maybe worth to make an inline function for it. http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/service/query-options.h File be/src/service/query-options.h: http://gerrit.cloudera.org:8080/#/c/20131/35/be/src/service/query-options.h@367 PS35, Line 367: QUERY_OPT_FN(sync_hms_events_wait_time_s, SYNC_HMS_EVENTS_WAIT_TIME_S, \ : TQueryOptionLevel::DEVELOPMENT) \ : QUERY_OPT_FN(sync_hms_events_strict_mode, SYNC_HMS_EVENTS_STRICT_MODE, \ : TQueryOptionLevel::DEVELOPMENT) \ Is this strict for DEVELOPMENT / testing, or do you envision this to become ADVANCED? Need documentation? http://gerrit.cloudera.org:8080/#/c/20131/31/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/20131/31/fe/src/main/java/org/apache/impala/service/Frontend.java@2294 PS31, Line 2294: warnings.add(errorMsg); > Modified ImpalaServer::UpdateExecSummary() to also add the analysis_warning Done http://gerrit.cloudera.org:8080/#/c/20131/35/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/20131/35/tests/common/impala_test_suite.py@1429 PS35, Line 1429: execute_query_with_hms_sync make this return result? http://gerrit.cloudera.org:8080/#/c/20131/35/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/20131/35/tests/custom_cluster/test_events_custom_configs.py@436 PS35, Line 436: execute_scalar Why not use the new execute_query_with_hms_sync? http://gerrit.cloudera.org:8080/#/c/20131/35/tests/custom_cluster/test_kudu.py File tests/custom_cluster/test_kudu.py: http://gerrit.cloudera.org:8080/#/c/20131/35/tests/custom_cluster/test_kudu.py@323 PS35, Line 323: execute_query Why not use the new execute_query_with_hms_sync? http://gerrit.cloudera.org:8080/#/c/20131/35/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/20131/35/tests/metadata/test_event_processing.py@293 PS35, Line 293: @SkipIfCatalogV2.hms_event_polling_disabled() hms_event_polling is enabled by default, right? http://gerrit.cloudera.org:8080/#/c/20131/35/tests/metadata/test_event_processing_base.py File tests/metadata/test_event_processing_base.py: http://gerrit.cloudera.org:8080/#/c/20131/35/tests/metadata/test_event_processing_base.py@30 PS35, Line 30: def _run_test_insert_events_impl(cls, unique_database, is_transactional=False): Except for EventProcessorUtils.wait_for_event_processing_impl removal in favor of EVENT_SYNC_QUERY_OPTIONS, other changes in this method is just indentation adjustment, correct? -- To view, visit http://gerrit.cloudera.org:8080/20131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a Gerrit-Change-Number: 20131 Gerrit-PatchSet: 35 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]> Gerrit-Comment-Date: Thu, 13 Feb 2025 23:05:49 +0000 Gerrit-HasComments: Yes
