Hello Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10573
to look at the new patch set (#4).
Change subject: IMPALA-7078: Part 2: reduce queue size based on
num_scanner_threads
......................................................................
IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads
This gives a tighter bound on memory consumption when running with
a lower num_scanner_threads value. With IMPALA-7096 we'll revisit
the approach to reliably avoid OOM.
Cap the maximum row batch queue size at 5 * max_num_scanner_threads_
so that we guarantee lower amounts of memory in the row batch queue
when num_scanner_threads is set, rather than just achieving it
statistically because of the producer running slower relative to
consumer. It does not reduce the default significantly on typical
server configurations that would have 24+ cores except under high
concurrency or low memory environments where the number of scanner
threads is limited. We should evaluate reducing the default further
or otherwise better controlling memory consumption in a follow-up,
based on experiments.
Testing:
Tested along with Part 1.
Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b
---
M be/src/exec/hdfs-scan-node.cc
1 file changed, 19 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/10573/4
--
To view, visit http://gerrit.cloudera.org:8080/10573
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b
Gerrit-Change-Number: 10573
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Dan Hecht <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
