Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22231 )
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py File tests/util/compute_table_stats.py: http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@98 PS3, Line 98: # Round robin between DB to reduce chance HMS locking over single DB object. Why would running compute stats on individual tables cause lock contention at the db level? http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@99 PS3, Line 99: db_table_pairs = [] : empty = 0 : while empty < len(db_table_list): : empty = 0 : for table_list in db_table_list: : if table_list: : db_table_pairs.append(table_list.pop(0)) : else: : empty += 1 : : for db, table in db_table_pairs: : if not continue_on_error and len(exceptions) > 0: : break : # Submit command to threadpool : pool.apply_async( : compute_stats_table, (client_factory, db, table, continue_on_error,), : callback=append_success_table, error_callback=append_error) I'm not exactly understanding this algorithm, but it seems to be essentially a breadth-first traversal. Please look at using a dictionary with key of db and value of set(tables). Then, this code could iterate through all the dictionary keys adding one table at a time to the thread pool. -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Thu, 19 Dec 2024 20:12:26 +0000 Gerrit-HasComments: Yes
