[
https://issues.apache.org/jira/browse/IMPALA-11839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677754#comment-17677754
]
Noemi Pap-Takacs edited comment on IMPALA-11839 at 1/17/23 2:30 PM:
--------------------------------------------------------------------
{{'Decimal'}} is a logical type in AVRO. It can be represented in AVRO
'{{{}bytes{}}}' or '{{{}fixed'{}}} types.
According to the Iceberg specification, the implementations and the Iceberg
library can read both, but write '{{{}decimal'{}}} with '{{{}fixed'{}}} type,
and so does Hive, when we create and insert into an Iceberg table. However,
Impala's avro scanner can only read '{{{}bytes{}}}'. {{'Fixed'}} type is
currently not supported, that is why the error message says:
{code:java}
Avro enum, array, map, union, and fixed types are not supported. Got type:
fixed{code}
was (Author: noemi):
{{'Decimal'}} is a logical type in AVRO. It can be represented in AVRO
'{{{}bytes{}}}' or '{{{}fixed'{}}} types.
According to the Iceberg specification, the implementations and the Iceberg
library can read both, but write '{{{}decimal'{}}} with '{{{}fixed'{}}} type,
and so does Hive, when we create and insert into an Iceberg table. However,
Impala can only read '{{{}bytes{}}}'. {{'Fixed'}} type is currently not
supported, that is why the error message says:
{code:java}
Avro enum, array, map, union, and fixed types are not supported. Got type:
fixed{code}
> impala fails to read iceberg table stored as avro which has decimal(15,2)
> datatype created by hive
> --------------------------------------------------------------------------------------------------
>
> Key: IMPALA-11839
> URL: https://issues.apache.org/jira/browse/IMPALA-11839
> Project: IMPALA
> Issue Type: Bug
> Reporter: Riddhi jain
> Assignee: Noemi Pap-Takacs
> Priority: Major
> Labels: iceberg
>
> Create table using Hive
> CREATE EXTERNAL TABLE decimal_table_txt (decimal_col decimal(15,2)) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '|' stored as TEXTFILE LOCATION
> 's3a://dwx-testdata/tmp/tpch/data/customer';CREATE TABLE decimal_table STORED
> BY iceberg STORED AS avro AS SELECT * FROM decimal_table_txt;
> When trying to read iceberg table (stored as avro) using impala with
> decimal(15,2) as datatype it fails to return the rows.
> {code:java}
> select * from decimal_table;{code}
> Output:
> {code:java}
> Query 88420c9d3ee0e2f1:4ef29b2a00000000 100% Complete (2 out of 2)
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed
> Query 88420c9d3ee0e2f1:4ef29b2a00000000 100% Complete (2 out of 2)
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed
> Query 88420c9d3ee0e2f1:4ef29b2a00000000 100% Complete (2 out of 2)
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed
> Avro enum, array, map, union, and fixed types are not supported. Got type:
> fixed {code}
> {code:java}
> Done. 0 results. 88420c9d3ee0e2f1:4ef29b2a00000000{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
