[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973654#comment-13973654
]
Anthony Hsu commented on HIVE-6835:
-----------------------------------
On a side note: If you create an Avro table and store the schema in the
TBLPROPERTIES -
{code}
CREATE TABLE ... TBLPROPERTIES ('avro.schema.literal'='...');
{code}
\- everything works fine with partitions because TBLPROPERTIES are NOT copied
to the partition, so the partition will end using the TBLPROPERTIES for
initializing the Avro SerDe.
It's only when you store the schema in the SERDEPROPERTIES -
{code}
CREATE TABLE ... WITH SERDEPROPERTIES ('avro.schema.literal'='...');
{code}
\- that problems arise. SERDEPROPERTIES DO get copied to the partitions, so if
you then end up changing the SERDEPROPERTIES stored at the table level, the
SERDEPROPERTIES in the table and the partitions get out of sync and this
sometimes leads to ClassCastExceptions with the AvroSerDe.
> Reading of partitioned Avro data fails if partition schema does not match
> table schema
> --------------------------------------------------------------------------------------
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.12.0
> Reporter: Anthony Hsu
> Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array<string>);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"}
> } ] }') STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ {"name":"intfield","type":"int","default":0},{
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
