Sahil Takiar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12163


Change subject: IMPALA-7087: Read Parquet decimal columns with lower 
precision/scale
......................................................................

IMPALA-7087: Read Parquet decimal columns with lower precision/scale

Allows Impala to read Parquet files that have been written with a lower
precision / scale compared to the precision / scale specified during
table creation. Currently, if the precision / scale in the table does
not exactly match the precision / scale in the underlying Parquet files,
Impala will throw an error during any attempt to read data from the
table.

This patch only allows reading Parquet files with a *lower* precision /
scale compared to the table metadata. Users still get an error if they
try to read higher precision / scale data into a lower precision /
scale.

The ability to read lower precision / scale data is allowed by the SQL
Standard and several other engines, such as Hive, MySQL, and Postgres,
allow for this behavior.

If the underlying Parquet files contain rows that cannot be converted to
the higher scale without overflow, an error is thrown and the query
fails. This is different from other engines such as Hive which set the
row to NULL and allow the query to run to completion without error.

Testing:
* Ran core tests
* Added new tests in test_scanners.py

Change-Id: Iafc8efd12379a39756e3e70f022a81a636dadb61
---
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M common/thrift/generate_error_codes.py
M testdata/data/README
A testdata/data/binary_decimal_precision_and_scale_widening.parquet
A 
testdata/workloads/functional-query/queries/QueryTest/parquet-decimal-precision-and-scale-widening.test
M tests/query_test/test_scanners.py
7 files changed, 357 insertions(+), 9 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/12163/1
--
To view, visit http://gerrit.cloudera.org:8080/12163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iafc8efd12379a39756e3e70f022a81a636dadb61
Gerrit-Change-Number: 12163
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <stak...@cloudera.com>

Reply via email to