Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17253 )
Change subject: IMPALA-6671: Change wait for sync ddl timeout ...................................................................... IMPALA-6671: Change wait for sync ddl timeout When skip locked tables from topic updates is enabled (topic_update_tbl_max_wait_time_ms > 0), it is possible that a thread waiting for a topic update during a sync ddl execution to terminate earlier. This is because the waitForSyncDdlVersion currently statically counts the number of instances of topic updates against a maximum and bails out when the maximum is reached. With topic update thread skipping locked tables, this number of instances of topic updates is more likely to hit the maximum attempt limit. This commit changes the logic so that when locked tables are skipped from topic updates, the sync ddl operation waits until a configurable timeout. This timeout value is specified in seconds using the configuration max_wait_time_for_sync_ddl_s. The default value of this configuration is 0 which means it waits indefinitely. This makes sure that only the sync ddl queries on a table which has been locked wait for extended durations while the other unrelated sync ddl queries can finish appropriately. Also this commit changes the default values of following flags which were introduced in the earlier patch for IMPALA-6671. The current default value of topic_update_tbl_max_wait_time_ms of 500ms is too low and may skip the locked tables more aggressively than needed. The new defaults were set based on analysis of a real world deployment. topic_update_tbl_max_wait_time_ms = 120000 catalog_max_lock_skipped_topic_updates = 3 Testing: 1. Ran existing test test_topic_update_frequency. 2. Added a new test for the newly added flag. Change-Id: I79e64cdec0e6aa7b597a47851b4b5c5441ca5528 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_topic_update_frequency.py 6 files changed, 88 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17253/3 -- To view, visit http://gerrit.cloudera.org:8080/17253 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I79e64cdec0e6aa7b597a47851b4b5c5441ca5528 Gerrit-Change-Number: 17253 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
