Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22980
Change subject: IMPALA-14075: Add CatalogOpExecutor.icebergExecutorService_ ...................................................................... IMPALA-14075: Add CatalogOpExecutor.icebergExecutorService_ Before this patch, Impala executes EXPIRE_SNAPSHOTS operation on a single thread. It can be really slow on cloud storage systems, especially if the operation needs to remove lots of files. This patch parallelize it by passing org.apache.iceberg.util.ThreadPools.getWorkerPool() to ExpireSnapshots.executeDeleteWith() API. Max num thread in worker pool is controlled through Java system properties 'iceberg.worker.num-threads' or default to max(2, num available CPU cores) if unset. Testing: - Lower sleep time between insert queries from 5s to 1s in test_expire_snapshots and test_describe_history_params to speed up tests. - Manually verify that 'iceberg-worker-pool' threads are visible in /jvm-threadz page of CatalogD. - Pass test_iceberg.py. Change-Id: I6dcbf1e406e1732ef8829eb0cd627d932291d485 --- M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M tests/query_test/test_iceberg.py 2 files changed, 6 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/22980/1 -- To view, visit http://gerrit.cloudera.org:8080/22980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6dcbf1e406e1732ef8829eb0cd627d932291d485 Gerrit-Change-Number: 22980 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com>
