Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22231 )
Change subject: IMPALA-13620: Refresh compute_table_stats.py script ...................................................................... Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py File tests/util/compute_table_stats.py: http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@98 PS3, Line 98: tables_to_compute = (all_tables.intersection(selected_tables) > I wrote my analysis here: Looking at IMPALA-13277, that error does seem to be a connection issue between HMS and its backend data store. Exhausting the connection pool could definitely be the cause behind the exception "javax.jdo.JDODataStoreException: objectstore - Connection is not available, request timed out after 30000ms". It's not going to hurt anything to round robin over the databases but may not help either. What are the implications to increasing the postgres connection pool count? Is that something worth doing in another Jira? http://gerrit.cloudera.org:8080/#/c/22231/3/tests/util/compute_table_stats.py@99 PS3, Line 99: - excluded_tables) : db_table_map[db] = tables_to_compute : total_tables += len(tables_to_compute) : : # Round robin between DB to reduce chance HMS locking over single DB object. : while db_table_map and not is_stop_on_error(): : for db in list(db_table_map.keys()): : if is_stop_on_error(): : break : if db_table_map[db]: : table = db_table_map[db].pop() : # Submit command to threadpool : pool.apply_async( : compute_stats_table, (client_factory, db, table, continue_on_error,), : callback=append_success_table, error_callback=append_error) : else: : del db_table_map[db] > Changed to iterate over map. Done -- To view, visit http://gerrit.cloudera.org:8080/22231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ebf02f95b957e7dda3a30622b87e8fca3197699 Gerrit-Change-Number: 22231 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Fri, 20 Dec 2024 17:00:46 +0000 Gerrit-HasComments: Yes
