Hello Lars Volker, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12163
to look at the new patch set (#2).
Change subject: IMPALA-7087: Read Parquet decimal columns with lower
precision/scale than table metadata
......................................................................
IMPALA-7087: Read Parquet decimal columns with lower precision/scale than table
metadata
Allows Impala to read Parquet files that have been written with a lower
precision / scale compared to the precision / scale specified during
table creation. Currently, if the precision / scale in the table does
not exactly match the precision / scale in the Parquet files, Impala
will throw an error and the table will be unqueryable.
This patch only allows reading Parquet files with a *lower* precision /
scale compared to the table metadata. Users still get an error if they
try to read higher precision / scale data into a lower precision /
scale.
The ability to read lower precision / scale data is allowed by the SQL
Standard and several other engines, such as Hive, MySQL, and Postgres,
allow for this behavior.
If the underlying Parquet files contain rows that cannot be converted to
the higher scale without overflow, the decimal slot is set to NULL and a
warning is printed in the logs. This is consistent with what other engines,
such as Hive do. If abort_on_error is set to true, then instead of setting
the slot to NULL, the query fails.
Testing:
* Ran core tests
* Added new tests in test_scanners.py
Change-Id: Iafc8efd12379a39756e3e70f022a81a636dadb61
---
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M common/thrift/generate_error_codes.py
M testdata/data/README
A testdata/data/binary_decimal_precision_and_scale_widening.parquet
A
testdata/workloads/functional-query/queries/QueryTest/parquet-decimal-precision-and-scale-widening-abort-on-error.test
A
testdata/workloads/functional-query/queries/QueryTest/parquet-decimal-precision-and-scale-widening.test
M tests/common/test_result_verifier.py
M tests/query_test/test_scanners.py
10 files changed, 422 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/12163/2
--
To view, visit http://gerrit.cloudera.org:8080/12163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iafc8efd12379a39756e3e70f022a81a636dadb61
Gerrit-Change-Number: 12163
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <stak...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com>
