Michael Ho has uploaded a new change for review. http://gerrit.cloudera.org:8080/2781
Change subject: IMPALA-1619: Support 64-bit allocations. ...................................................................... IMPALA-1619: Support 64-bit allocations. This change extends MemPool, FreePool and StringBuffer to support 64-bit allocations, fixes a bug decompressor and extends various places in the code to support 64-bit allocation sizes. With this change, the text scanner can now decompress compressed files larger than 1GB. This change also adds a function MemPool::TryFree() which will free the chunk corresponding to the input buffer if the buffer covers the entire range of a chunk. This is useful for reducing the memory consumption of StringBuffer which will double its buffer without freeing the old buffer and the memory is accumulated till the end of the query. Note that the UDF interfaces FunctionContext::Allocate() and FunctionContext::Reallocate() still use 32-bit for the input argument to avoid breaking compatibility. In addition, the byte size of a tuple is still assumed to be within 32-bit. If it needs to be upgraded to 64-bit, it will be done in a separate change. Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20 --- M be/src/exec/delimited-text-parser.cc M be/src/exec/delimited-text-parser.h M be/src/exec/delimited-text-parser.inline.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/hdfs-text-scanner.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/collection-value-builder.h M be/src/runtime/free-pool-test.cc M be/src/runtime/free-pool.h M be/src/runtime/mem-pool-test.cc M be/src/runtime/mem-pool.cc M be/src/runtime/mem-pool.h M be/src/runtime/string-buffer-test.cc M be/src/runtime/string-buffer.h M be/src/udf/udf-internal.h M be/src/udf/udf.cc M be/src/udf/udf.h M be/src/util/codec.cc M be/src/util/codec.h M be/src/util/decompress-test.cc M be/src/util/decompress.cc M infra/python/bootstrap_virtualenv.py M infra/python/deps/download_requirements M infra/python/deps/requirements.txt M tests/query_test/test_compressed_formats.py 28 files changed, 344 insertions(+), 267 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/81/2781/1 -- To view, visit http://gerrit.cloudera.org:8080/2781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7ed28083d809a86d801a9c063a0aa32c50d32b20 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Michael Ho <[email protected]>
