Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-1886/IMPALA-2154: Add support for multi-stream bz2/gzip compressed files. ......................................................................
IMPALA-1886/IMPALA-2154: Add support for multi-stream bz2/gzip compressed files. Fix a bug in which Impala only reads the first stream of a multi-stream bz2/gzip file. Changes the bz2 decoder to read the file in a streaming fashion rather than reading the entire file into memory before it can be decompressed. Change-Id: Icbe617d03a69953f0bf3aa0f7c30d34bc612f9f8 (cherry picked from commit b6d0b4e059329633dc50f1f73ebe35b7ac317a8e) Reviewed-on: http://gerrit.cloudera.org:8080/2219 Reviewed-by: Juan Yu <[email protected]> Tested-by: Internal Jenkins --- M be/src/exec/hdfs-text-scanner.cc M be/src/exec/hdfs-text-scanner.h M be/src/util/codec.cc M be/src/util/codec.h M be/src/util/decompress-test.cc M be/src/util/decompress.cc M be/src/util/decompress.h M common/thrift/generate_error_codes.py M testdata/data/README A testdata/data/data-bzip2.bz2 A testdata/data/data-pbzip2.bz2 A testdata/data/large_bzip2.bz2 A testdata/data/large_pbzip2.bz2 M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test A testdata/workloads/functional-query/queries/QueryTest/text-bzip-scan.test M tests/query_test/test_compressed_formats.py 18 files changed, 530 insertions(+), 188 deletions(-) Approvals: Juan Yu: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/2219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Icbe617d03a69953f0bf3aa0f7c30d34bc612f9f8 Gerrit-PatchSet: 19 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.5.0_5.7.0 Gerrit-Owner: Juan Yu <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Juan Yu <[email protected]> Gerrit-Reviewer: Skye Wanderman-Milne <[email protected]>
