Hello Quanlong Huang, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20482
to look at the new patch set (#10).
Change subject: IMPALA-12431: Support reading compressed JSON file
......................................................................
IMPALA-12431: Support reading compressed JSON file
This patch adds the functionality to read compressed JSON files for the
JSON scanner. Because the decompression code can largely be reused from
HdfsTextScanner, this patch moves that part of the code from
HdfsTextScanner to HdfsScanner so that HdfsJsonScanner can also call it.
As it reuses the relevant code from the TEXT scanner, the compression
formats supported by the Json scanner are the same as those supported by
the TEXT scanner.
Tests
- Most of the existing end-to-end JSON format tests can run on
compressed JSON format too.
Change-Id: I2471855d97d4cdd51363b321055e6b06aa6d81e8
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/json/hdfs-json-scanner.cc
M be/src/exec/json/hdfs-json-scanner.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-query/functional-query_exhaustive.csv
M testdata/workloads/tpcds/tpcds_core.csv
M testdata/workloads/tpcds/tpcds_exhaustive.csv
M testdata/workloads/tpcds/tpcds_pairwise.csv
M testdata/workloads/tpch/tpch_core.csv
M testdata/workloads/tpch/tpch_exhaustive.csv
M testdata/workloads/tpch/tpch_pairwise.csv
M tests/query_test/test_compressed_formats.py
15 files changed, 258 insertions(+), 215 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/20482/10
--
To view, visit http://gerrit.cloudera.org:8080/20482
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2471855d97d4cdd51363b321055e6b06aa6d81e8
Gerrit-Change-Number: 20482
Gerrit-PatchSet: 10
Gerrit-Owner: Zihao Ye <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Zihao Ye <[email protected]>
