[jira] [Updated] (HIVE-7049) Unable to deserialize AVRO data when file schema and record schema are different and nullable
[ https://issues.apache.org/jira/browse/HIVE-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-7049: Fix Version/s: 1.2.0 > Unable to deserialize AVRO data when file schema and record schema are > different and nullable > - > > Key: HIVE-7049 > URL: https://issues.apache.org/jira/browse/HIVE-7049 > Project: Hive > Issue Type: Bug >Reporter: Mohammad Kamrul Islam >Assignee: Daniel Dai > Fix For: 1.2.0 > > Attachments: HIVE-7049.1.patch, HIVE-7049.2.patch, Statistic, > Statistics10Min.avsc > > > It mainly happens when > 1 )file schema and record schema are not same > 2 ) Record schema is nullable but file schema is not. > The potential code location is at class AvroDeserialize > > {noformat} > if(AvroSerdeUtils.isNullableType(recordSchema)) { > return deserializeNullableUnion(datum, fileSchema, recordSchema, > columnType); > } > {noformat} > In the above code snippet, recordSchema is verified if it is nullable. But > the file schema is not checked. > I tested with these values: > {noformat} > recordSchema= ["null","string"] > fielSchema= "string" > {noformat} > And i got the following exception mu debugged code version>. > {noformat} > org.apache.avro.AvroRuntimeException: Not a union: "string" > at org.apache.avro.Schema.getTypes(Schema.java:272) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeNullableUnion(AvroDeserializer.java:275) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.worker(AvroDeserializer.java:205) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.workerBase(AvroDeserializer.java:188) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserialize(AvroDeserializer.java:174) > at > org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyNullableType(TestAvroDeserializer.java:487) > at > org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.canDeserializeNullableTypes(TestAvroDeserializer.java:407) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7049) Unable to deserialize AVRO data when file schema and record schema are different and nullable
[ https://issues.apache.org/jira/browse/HIVE-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7049: - Assignee: Daniel Dai (was: Mohammad Kamrul Islam) Unable to deserialize AVRO data when file schema and record schema are different and nullable - Key: HIVE-7049 URL: https://issues.apache.org/jira/browse/HIVE-7049 Project: Hive Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Daniel Dai Attachments: HIVE-7049.1.patch, HIVE-7049.2.patch, Statistic, Statistics10Min.avsc It mainly happens when 1 )file schema and record schema are not same 2 ) Record schema is nullable but file schema is not. The potential code location is at class AvroDeserialize {noformat} if(AvroSerdeUtils.isNullableType(recordSchema)) { return deserializeNullableUnion(datum, fileSchema, recordSchema, columnType); } {noformat} In the above code snippet, recordSchema is verified if it is nullable. But the file schema is not checked. I tested with these values: {noformat} recordSchema= [null,string] fielSchema= string {noformat} And i got the following exception line numbers might not be the same due to mu debugged code version. {noformat} org.apache.avro.AvroRuntimeException: Not a union: string at org.apache.avro.Schema.getTypes(Schema.java:272) at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeNullableUnion(AvroDeserializer.java:275) at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.worker(AvroDeserializer.java:205) at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.workerBase(AvroDeserializer.java:188) at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserialize(AvroDeserializer.java:174) at org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyNullableType(TestAvroDeserializer.java:487) at org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.canDeserializeNullableTypes(TestAvroDeserializer.java:407) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)