Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10573 )
Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads ...................................................................... IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads This gives a tighter bound on memory consumption when running with a lower num_scanner_threads value. With IMPALA-7096 we'll revisit the approach to reliably avoid OOM. Cap the maximum row batch queue size at 5 * max_num_scanner_threads_ so that we guarantee lower amounts of memory in the row batch queue when num_scanner_threads is set, rather than just achieving it statistically because of the producer running slower relative to consumer. It does not reduce the default significantly on typical server configurations that would have 24+ cores except under high concurrency or low memory environments where the number of scanner threads is limited. We should evaluate reducing the default further or otherwise better controlling memory consumption in a follow-up, based on experiments. Testing: Tested along with Part 1. Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Reviewed-on: http://gerrit.cloudera.org:8080/10573 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/hdfs-scan-node.cc 1 file changed, 19 insertions(+), 6 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
