Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-2399: Check for mem limit in allocations in parquet scanner and decompressor ......................................................................
IMPALA-2399: Check for mem limit in allocations in parquet scanner and decompressor This change replaces all calls to MemPool::Allocate() with MemPool::TryAllocate() in the parquet scanner and the decompressor. Also streamline CheckQueryState() to avoid unnecessary spinlock acquisition for the common case when there is no error. Also removes some dead code in the text converter. MemPool::Allocate() is also updated to return a valid pointer instead of NULL when the allocation size is zero. NULL is only returned during allocation failure. This change also updates CollectionValueBuilder::GetFreeMemory() to return Status in case it exceeds memory limit. As part of the change, the max allocation limit (2 GB) is also removed from it as 64-bit allocations are supported in MemPool with this change. Change-Id: Ic70400407b7662999332448f4d1bce2cc344ca89 Reviewed-on: http://gerrit.cloudera.org:8080/2203 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Internal Jenkins --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/text-converter.cc M be/src/exec/text-converter.h M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/collection-value-builder-test.cc M be/src/runtime/collection-value-builder.h M be/src/runtime/collection-value.h M be/src/runtime/mem-pool.cc M be/src/runtime/mem-pool.h M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/util/codec.h M be/src/util/decompress.cc M be/src/util/runtime-profile.h M common/thrift/generate_error_codes.py 23 files changed, 332 insertions(+), 195 deletions(-) Approvals: Internal Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/2203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic70400407b7662999332448f4d1bce2cc344ca89 Gerrit-PatchSet: 17 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Skye Wanderman-Milne <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
