Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18126
Change subject: IMPALA-11068: Add tuning flag to reduce scanner thread launch. ...................................................................... IMPALA-11068: Add tuning flag to reduce scanner thread launch. Under heavy decompression workload, Impala running with scanner thread parallelism (MT_DOP=0) can still hit OOM error due to launching too many threads too soon. We have logic in ScannerMemLimiter to limit the number of scanner threads by calculating the thread's memory requirement and estimating the memory growth rate of all threads. However, it does not prevent a scanner node from quickly launching many threads and immediately reaching the memtracker's spare capacity. Even after ScannerMemLimiter rejects a new thread launch, some existing threads might continue increasing their non-reserved memory for decompression work until finally the memory limit is exceeded. IMPALA-7096 adds 'hdfs_scanner_thread_max_estimated_bytes' flag as a heuristic to count for non-reserved memory growth. Increasing this flag value can help reduce thread count, but might severely regress other queries that do not have heavy decompression characteristics. Similarly with lowering the NUM_SCANNER_THREADS query option. This patch adds one more flag as an alternative to mitigate OOM called 'hdfs_scanner_thread_headroom_count'. This flag intent to lower the threshold to reject a new scanner thread launch. For example, setting hdfs_scanner_thread_headroom_count=1 means ScannerMemLimiter will reject a new thread from launching if the memtracker does not have spare capacity to fit at least 2 threads. Testing: - Add custom cluster test TestScanMemLimitCustom.test_hdfs_scanner_thread_mem_scaling_headroom Change-Id: I03cadf1230eed00d69f2890c82476c6861e37466 --- M be/src/runtime/scanner-mem-limiter.cc A testdata/workloads/functional-query/queries/QueryTest/hdfs-scanner-thread-mem-scaling-headroom.test M tests/custom_cluster/test_mem_reservations.py 3 files changed, 85 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18126/1 -- To view, visit http://gerrit.cloudera.org:8080/18126 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I03cadf1230eed00d69f2890c82476c6861e37466 Gerrit-Change-Number: 18126 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
