This is an automated email from the ASF dual-hosted git repository. boroknagyz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/impala.git.
from 1e49b6a IMPALA-2029. Implement our own getJNIEnv equivalent new 9075099 Drop statestore update frequency during data loading new d423979 IMPALA-5843: Use page index in Parquet files to skip pages The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/common/global-flags.cc | 2 +- be/src/exec/hdfs-scan-node-base.cc | 26 +- be/src/exec/hdfs-scan-node-base.h | 14 +- be/src/exec/parquet/CMakeLists.txt | 3 + be/src/exec/parquet/hdfs-parquet-scanner.cc | 258 +++++++- be/src/exec/parquet/hdfs-parquet-scanner.h | 69 +- be/src/exec/parquet/parquet-column-readers.cc | 312 ++++++++- be/src/exec/parquet/parquet-column-readers.h | 133 +++- be/src/exec/parquet/parquet-column-stats.cc | 50 +- be/src/exec/parquet/parquet-column-stats.h | 12 + be/src/exec/parquet/parquet-common-test.cc | 122 ++++ be/src/exec/parquet/parquet-common.cc | 99 +++ be/src/exec/parquet/parquet-common.h | 79 +++ be/src/exec/parquet/parquet-level-decoder.h | 29 +- be/src/exec/parquet/parquet-page-index-test.cc | 108 ++++ be/src/exec/parquet/parquet-page-index.cc | 147 +++++ be/src/exec/parquet/parquet-page-index.h | 83 +++ be/src/exprs/literal.cc | 16 +- be/src/runtime/scoped-buffer.h | 4 +- be/src/service/query-options.cc | 7 +- be/src/service/query-options.h | 4 +- common/thrift/ImpalaInternalService.thrift | 3 + common/thrift/ImpalaService.thrift | 5 + testdata/bin/create-load-data.sh | 3 +- testdata/data/README | 124 +++- testdata/data/alltypes_tiny_pages.parquet | Bin 0 -> 454233 bytes testdata/data/alltypes_tiny_pages_plain.parquet | Bin 0 -> 811756 bytes testdata/data/decimals_1_10.parquet | Bin 0 -> 3874 bytes testdata/data/double_nested_decimals.parquet | Bin 0 -> 3846 bytes testdata/data/nested_decimals.parquet | Bin 0 -> 2369 bytes .../QueryTest/nested-types-parquet-page-index.test | 704 +++++++++++++++++++++ ...rquet-page-index-alltypes-tiny-pages-plain.test | 234 +++++++ .../parquet-page-index-alltypes-tiny-pages.test | 234 +++++++ .../QueryTest/parquet-page-index-large.test | 357 +++++++++++ .../queries/QueryTest/parquet-page-index.test | 219 +++++++ .../queries/QueryTest/stats-extrapolation.test | 12 +- tests/query_test/test_parquet_stats.py | 24 + 37 files changed, 3399 insertions(+), 97 deletions(-) create mode 100644 be/src/exec/parquet/parquet-common-test.cc create mode 100644 be/src/exec/parquet/parquet-page-index-test.cc create mode 100644 be/src/exec/parquet/parquet-page-index.cc create mode 100644 be/src/exec/parquet/parquet-page-index.h create mode 100644 testdata/data/alltypes_tiny_pages.parquet create mode 100644 testdata/data/alltypes_tiny_pages_plain.parquet create mode 100644 testdata/data/decimals_1_10.parquet create mode 100644 testdata/data/double_nested_decimals.parquet create mode 100644 testdata/data/nested_decimals.parquet create mode 100644 testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test create mode 100644 testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages-plain.test create mode 100644 testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages.test create mode 100644 testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-large.test create mode 100644 testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test