Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/15370 )
Change subject: IMPALA-6636: Use async IO in ORC scanner ...................................................................... Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/15370/16/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/15370/16/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@2179 PS16, Line 2179: if (orcAsyncRead) columnByteSizes.add(DEFAULT_COLUMN_SCAN_RANGE_RESERVATION); I think we can do better on counting min mem reservation here and other places for orcAsyncRead. DEFAULT_COLUMN_SCAN_RANGE_RESERVATION is 4MB, and by keep appending this much per stream, we potentially going to over provision memory by a lot. Instead, we can split this 4MB allocation to all streams that belong to the column. -- To view, visit http://gerrit.cloudera.org:8080/15370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I348ad9e55f0cae7dff0d74d941b026dcbf5e4074 Gerrit-Change-Number: 15370 Gerrit-PatchSet: 16 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Wed, 08 Dec 2021 17:56:35 +0000 Gerrit-HasComments: Yes
