Impala Public Jenkins has submitted this change and it was merged. ( )

Change subject: IMPALA-5717: Support for reading ORC data files

IMPALA-5717: Support for reading ORC data files

This patch integrates the orc library into Impala and implements
HdfsOrcScanner as a middle layer between them. The HdfsOrcScanner
supplies input needed from the orc-reader, tracks memory consumption of
the reader and transfers the reader's output (orc::ColumnVectorBatch)
into impala::RowBatch. The ORC version we used is release-1.4.3.

A startup option --enable_orc_scanner is added for this feature. It's
set to true by default. Setting it to false will fail queries on ORC

Currently, we only support reading primitive types. Writing into ORC
table has not been supported neither.

 - Most of the end-to-end tests can run on ORC format.
 - Add tpcds, tpch tests for ORC.
 - Add some ORC specific tests.
 - Haven't enabled test_scanner_fuzz for ORC yet, since the ORC library
   is not robust for corrupt files (ORC-315).

Change-Id: Ia7b6ae4ce3b9ee8125b21993702faa87537790a4
Reviewed-by: Quanlong Huang <>
Reviewed-by: Tim Armstrong <>
Tested-by: Impala Public Jenkins <>
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/codegen/
M be/src/exec/CMakeLists.txt
A be/src/exec/
A be/src/exec/hdfs-orc-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-parquet-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-scanner.h
M be/src/util/
M bin/
M bin/
A cmake_modules/FindOrc.cmake
M common/thrift/BackendGflags.thrift
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/main/java/org/apache/impala/planner/
M fe/src/main/java/org/apache/impala/service/
M fe/src/main/java/org/apache/impala/service/
M fe/src/main/jflex/sql-scanner.flex
M testdata/LineItemMultiBlock/README.dox
A testdata/LineItemMultiBlock/lineitem_orc_multiblock_one_stripe.orc
A testdata/LineItemMultiBlock/lineitem_sixblocks.orc
A testdata/LineItemMultiBlock/lineitem_threeblocks.orc
M testdata/bin/
M testdata/bin/
M testdata/bin/
M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl
A testdata/data/chars-formats.orc
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/functional-query_core.csv
M testdata/workloads/functional-query/functional-query_dimensions.csv
M testdata/workloads/functional-query/functional-query_exhaustive.csv
M testdata/workloads/functional-query/functional-query_pairwise.csv
M testdata/workloads/tpcds/tpcds_core.csv
M testdata/workloads/tpcds/tpcds_dimensions.csv
M testdata/workloads/tpcds/tpcds_exhaustive.csv
M testdata/workloads/tpcds/tpcds_pairwise.csv
M testdata/workloads/tpch/tpch_core.csv
M testdata/workloads/tpch/tpch_dimensions.csv
M testdata/workloads/tpch/tpch_exhaustive.csv
M testdata/workloads/tpch/tpch_pairwise.csv
M tests/common/
M tests/common/
M tests/common/
M tests/comparison/
M tests/query_test/
M tests/query_test/
M tests/query_test/
M tests/query_test/
M tests/query_test/
62 files changed, 1,745 insertions(+), 307 deletions(-)

  Quanlong Huang: Looks good to me, but someone else must approve
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia7b6ae4ce3b9ee8125b21993702faa87537790a4
Gerrit-Change-Number: 9134
Gerrit-PatchSet: 21
Gerrit-Owner: Quanlong Huang <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Impala Public Jenkins <>
Gerrit-Reviewer: Joe McDonnell <>
Gerrit-Reviewer: Quanlong Huang <>
Gerrit-Reviewer: Tim Armstrong <>

Reply via email to