[
https://issues.apache.org/jira/browse/IMPALA-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692242#comment-16692242
]
Sahil Takiar commented on IMPALA-7087:
--------------------------------------
For text files with extra scale, Impala throws an error:
{code}
[localhost:21000] default> create table dec_test_high_scale (col decimal(10,2))
stored as textfile;
[localhost:21000] default> insert into table dec_test_high_scale values (1),
(1.1), (1.11);
[localhost:21000] default> select * from dec_test_high_scale;
+------+
| col |
+------+
| 1.00 |
| 1.10 |
| 1.11 |
+------+
[localhost:21000] default> create table dec_test_low_scale (col decimal(10,1))
stored as textfile location
'hdfs://localhost:20500/test-warehouse/dec_test_high_scale';
[localhost:21000] default> select * from dec_test_low_scale;
+------+
| col |
+------+
| NULL |
| NULL |
| NULL |
+------+
WARNINGS: Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb2400000000_381290695_data.0.,
before offset: 15
Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb2400000000_381290695_data.0.,
before offset: 15
Error converting column: 0 to DECIMAL(10,1)
Error parsing row: file:
hdfs://localhost:20500/test-warehouse/dec_test_high_scale/44b64756e1edb4d-aa75eb2400000000_381290695_data.0.,
before offset: 15
Fetched 3 row(s) in 4.25s
{code}
MySQL and Postgres both support the following:
{code}
create table dec_test (dec_col decimal(10,2));
insert into dec_test values (1.1), (1.11), (1.111);
select * from dec_test;
dec_col
1.10
1.11
1.11
{code}
I don't have access to the SQL standard, so I'm not sure if it has anything to
say about this behavior.
> Impala is unable to read Parquet decimal columns with lower precision/scale
> than table metadata
> -----------------------------------------------------------------------------------------------
>
> Key: IMPALA-7087
> URL: https://issues.apache.org/jira/browse/IMPALA-7087
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Sahil Takiar
> Priority: Major
> Labels: decimal, parquet
>
> This is similar to IMPALA-2515, except relates to a different precision/scale
> in the file metadata rather than just a mismatch in the bytes used to store
> the data. In a lot of cases we should be able to convert the decimal type on
> the fly to the higher-precision type.
> {noformat}
> ERROR: File '/hdfs/path/000000_0_x_2' column 'alterd_decimal' has an invalid
> type length. Expecting: 11 len in file: 8
> {noformat}
> It would be convenient to allow reading parquet files where the
> precision/scale in the file can be converted to the precision/scale in the
> table metadata without loss of precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
