Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22231 )
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py File tests/util/compute_table_stats.py: http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@98 PS3, Line 98: tables_to_compute = (all_tables.intersection(selected_tables) > Why would running compute stats on individual tables cause lock contention I wrote my analysis here: https://issues.apache.org/jira/browse/IMPALA-13277?focusedCommentId=17906891&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17906891 Then, I suspect that there is a DB-level contention in HMS when 1-2 COMPUTE STATS query out of 24 concurrent ones fail with such HMS error. But looking some more, it could be that the root cause is in the underlying RDBMS (ie., the default postgresql max_connection, 100, is too small). http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@99 PS3, Line 99: - excluded_tables) : db_table_map[db] = tables_to_compute : total_tables += len(tables_to_compute) : : # Round robin between DB to reduce chance HMS locking over single DB object. : while db_table_map and not is_stop_on_error(): : for db in list(db_table_map.keys()): : if is_stop_on_error(): : break : if db_table_map[db]: : table = db_table_map[db].pop() : # Submit command to threadpool : pool.apply_async( : compute_stats_table, (client_factory, db, table, continue_on_error,), : callback=append_success_table, error_callback=append_error) : else: : del db_table_map[db] > I'm not exactly understanding this algorithm, but it seems to be essentiall Changed to iterate over map. -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Thu, 19 Dec 2024 21:08:05 +0000 Gerrit-HasComments: Yes
