Michael Ho has posted comments on this change. Change subject: IMPALA-2831: Bound the number of scanner threads per scan node. ......................................................................
Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/4174/1//COMMIT_MSG Commit Message: Line 20: with 8 logical cores reduces from 287MB to 101MB. > could you add a little more explanation about why the mem usage goes down ( Done http://gerrit.cloudera.org:8080/#/c/4174/1/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: Line 906: if (!ranges.empty()) ThreadTokenAvailableCb(runtime_state_->resource_pool()); > and isn't this already handled by: As discussed offline, 'progress_' doesn't take into account of file format so it will be non-zero. This ends up causing a bunch of scanner threads to be started. In the worst case, those threads may just quit when they realize they cannot get a scan range. I find this behavior a bit confusing. -- To view, visit http://gerrit.cloudera.org:8080/4174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I191988ad18d6b4caf892fc967258823edcf9681f Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
