Alex Behm has posted comments on this change. Change subject: IMPALA-5304: reduce transfer of Parquet decompression buffers ......................................................................
Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/6876/2/be/src/exec/parquet-column-readers.cc File be/src/exec/parquet-column-readers.cc: Line 1029: Status BaseScalarColumnReader::ReadDataPage() { > We already do that in HdfsParquetScanner::InitColumns() if the file is comp Ahh right. Thanks. Line 1037: decompressed_data_pool_->FreeAll(); > You mean Clear()? I don't want to make that change - it's really difficult Makes sense. We could selectively Clear() and FreeAll() depending on the uncompressed_size of the page and the allocated bytes in the mempool (I think we can safely assume the mempool only has a single chunk). Change seems easy enough, but fine with me if you want to defer. http://gerrit.cloudera.org:8080/#/c/6876/2/be/src/exec/parquet-column-readers.h File be/src/exec/parquet-column-readers.h: Line 476: && slot_desc_ != nullptr && slot_desc_->type().IsVarLenType(); I think checking IsStringType() is more accurate because IsVarLenType() includes collection types which are not really var len in the sense used here. -- To view, visit http://gerrit.cloudera.org:8080/6876 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2dbd749f43078b222ff8e1ddcec840986c466de6 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
