Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8966 )

Change subject: IMPALA-4835: Part 3: switch I/O buffers to buffer pool
......................................................................


Patch Set 22: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8966/20/be/src/exec/hdfs-parquet-scanner.cc
File be/src/exec/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/8966/20/be/src/exec/hdfs-parquet-scanner.cc@1766
PS20, Line 1766:       }
> My motivation for all this complication was mostly the case where you have 
Right, but don't we reserve 3 * 8MB per column anyway? Oh, but then we'll 
reduce it to the hdfs split size, so then this more complicated logic does kick 
in.  But in that case, we have enough reservation to buffer the entire row 
group and therefore can assign buffers that are as large as each column (and 
then some). But then I guess we want to limit the buffer size to overlap cpu 
and IO. So, hmm, okay.


http://gerrit.cloudera.org:8080/#/c/8966/20/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/8966/20/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1315
PS20, Line 1315: idealScanRangeReservation_
> Not right now. I thought about adding it to the explain output. It would be
How are those questions answered then? And is it easy to see how much memory 
each scanner thread got?



--
To view, visit http://gerrit.cloudera.org:8080/8966
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic09c6196b31e55b301df45cc56d0b72cfece6786
Gerrit-Change-Number: 8966
Gerrit-PatchSet: 22
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Feb 2018 18:26:09 +0000
Gerrit-HasComments: Yes

Reply via email to