Michael Ho has uploaded a new patch set (#3). Change subject: IMPALA-2831: Bound the number of scanner threads per scan node. ......................................................................
IMPALA-2831: Bound the number of scanner threads per scan node. Our current code base allows a scan node to spin up as many as 3x the number of logical cpu cores of scanner threads. However, the scanner threads are cpu bound so there is diminishing return for starting more scanner threads than the number of logical cores. In fact, it may be detrimental due to context switching overhead. This change bounds the number of scanner threads spun up by a scan node to the number of logical cpu cores unless the query option 'num_scanner_threads' is set. The total number of available thread tokens is unchanged. With this change, the peak memory usage of the following query on a single node Impala cluster running on a machine with 8 logical cores reduces from 287MB to 101MB. select count(*) from tpch100_parquet.lineitem where l_orderkey > 20 The reduction comes mostly from the fewer outstanding IO buffers. The IO for scan ranges will be scheduled by the scanner threads which pick them up. There will be at least an IO buffer of 8 to 16MB associated with each scan range. So, more threads we start up, more memory will be consumed by the IO buffers, leading to the higher peak memory usages. Change-Id: I191988ad18d6b4caf892fc967258823edcf9681f --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h 2 files changed, 23 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/4174/3 -- To view, visit http://gerrit.cloudera.org:8080/4174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I191988ad18d6b4caf892fc967258823edcf9681f Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
