[
https://issues.apache.org/jira/browse/AVRO-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212467#comment-17212467
]
Ryan Skraba commented on AVRO-2817:
-----------------------------------
I took a look at the Hive JIRA -- it's definitely the same problem.
There's some possible solutions in the previous comment, but I'd like to
suggest a different one: what if we just always turned
*{{setValidateDefault(false)}}* when reading the schema out of a file? At one
point, when writing to the file, the schema was valid enough to save data as
binary. We can probably safely turn off validation when reading the actual
schema metadata from the file for all cases. In any case, the *{{default}}*
attributes in the schema are never used in the actual/writer schema stored in
the file during schema resolution, only with the expected/desired output schema.
This shouldn't change any existing behaviour for 1.9.x or 1.10.x readers
running with older avro files, and should solve the problem both here and in
Hive.
> Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for
> certian type of schemas
> ------------------------------------------------------------------------------------------------------
>
> Key: AVRO-2817
> URL: https://issues.apache.org/jira/browse/AVRO-2817
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.9.2
> Reporter: Manoj Kumar Jha
> Priority: Blocker
> Attachments: records.avro
>
>
> Hello,
> We observed avro file generated using avro-1.8.2 is not readable while
> using avro-1.9.2 if schema contains a field "default:null". Please see
> below.
>
> --------------------------------------------------------------------------
> [mxj142:Jars]$ ls -ltr
> total 168520
> -rwx------@ 1 mxj142 staff 34798932 Jan 16 14:45 avro-tools-1.8.2.jar
> -rwxr-xr-x@ 1 mxj142 staff 51303364 Mar 6 17:43 avro-tools-1.9.2.jar
> -rw-r--r-- 1 mxj142 staff 715 Apr 24 11:19 records.avro
> [mxj142:Jars]$ java -jar avro-tools-1.8.2.jar tojson records.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {"enrichmentHeader":\{"correlationId":"1","sourceId":"09b5d770-7373-462d-9517-8e5ce957793f"},"cx":\{"dateFirst":"20200419","dateLast":"20200419"}}
> {"enrichmentHeader":\{"correlationId":"2","sourceId":"0285dc49-cc5c-4717-8ea2-45e50de2f65f"},"cx":\{"dateFirst":"20200420","dateLast":"20200420"}}
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$ java -jar avro-tools-1.9.2.jar tojson records.avro
> 20/04/24 11:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" org.apache.avro.AvroTypeException: Invalid default
> for field enrichmentHeader: null not a
> \{"type":"record","name":"EnrichmentHeader","namespace":"com.mxj142.commons.domain","fields":[{"name":"correlationId","type":{"type":"string","avro.java.string":"String"}},\{"name":"sourceId","type":{"type":"string","avro.java.string":"String"}}]}
> at org.apache.avro.Schema.validateDefault(Schema.java:1540)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.<init>(Schema.java:521)
> at org.apache.avro.Schema.parse(Schema.java:1647)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1382)
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130)
> at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90)
> at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:93)
> at org.apache.avro.tool.Main.run(Main.java:66)
> at org.apache.avro.tool.Main.main(Main.java:55)
> [mxj142:Jars]$
> --------------------------------------------------------------------------
>
> The file "records.avro" is generated using avro-1.8.2 . It contains two
> records and using "avro-tools-1.8.2.jar" one can read it. If we use
> "avro-tools-1.9.2.jar", then "records.avro" is not readable (as shown
> above). The file "records.avro" is also attached for your convenience.
>
> Let me know if you need any more information.
>
> Thanks,
> Manoj
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
