[
https://issues.apache.org/jira/browse/AVRO-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111128#comment-17111128
]
Ryan Skraba commented on AVRO-2817:
-----------------------------------
Hello! Thanks for the investigation – I'm not entirely sure what you're asking
for!
A quick note, it was actually AVRO-2035 that started validating defaults in
1.9.x.
The schema that worked in Avro Java 1.8 was faulty. Tightening up this
behaviour is important for interoperability (assuming other languages are
following the spec). I don't think it's a good idea to restore the old
behaviour or turn off default validation for all users!
We definitely *should* be providing a path to move to the more strict Java SDK,
however. The
[{{Schema.Parser.setValidateDefaults(false)}}|https://avro.apache.org/docs/1.9.2/api/java/org/apache/avro/Schema.Parser.html#setValidateDefaults(boolean)]
is OK if the user is explicitly parsing the schema themselves. Would one of
the following a sufficient solution for you?
# A command line flag for avro-tool?
# A validateDefault parameter to the Avro file reader?
# A system property for the "default" {{validateDefault}} value?
# A hadoop conf (if using the built-in MapReduce classes)?
If you can provide a bit more detail about how you're storing and accessing
your existing Avro data, we can probably make a useful fix! Ideally, there'd
be a way to move your environment to a valid Avro schema.
> Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for
> certian type of schemas
> ------------------------------------------------------------------------------------------------------
>
> Key: AVRO-2817
> URL: https://issues.apache.org/jira/browse/AVRO-2817
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.9.2
> Reporter: Manoj Kumar Jha
> Priority: Blocker
> Attachments: records.avro
>
>
> Hello,
> We observed avro file generated using avro-1.8.2 is not readable while
> using avro-1.9.2 if schema contains a field "default:null". Please see
> below.
>
> --------------------------------------------------------------------------
> [mxj142:Jars]$ ls -ltr
> total 168520
> -rwx------@ 1 mxj142 staff 34798932 Jan 16 14:45 avro-tools-1.8.2.jar
> -rwxr-xr-x@ 1 mxj142 staff 51303364 Mar 6 17:43 avro-tools-1.9.2.jar
> -rw-r--r-- 1 mxj142 staff 715 Apr 24 11:19 records.avro
> [mxj142:Jars]$ java -jar avro-tools-1.8.2.jar tojson records.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {"enrichmentHeader":\{"correlationId":"1","sourceId":"09b5d770-7373-462d-9517-8e5ce957793f"},"cx":\{"dateFirst":"20200419","dateLast":"20200419"}}
> {"enrichmentHeader":\{"correlationId":"2","sourceId":"0285dc49-cc5c-4717-8ea2-45e50de2f65f"},"cx":\{"dateFirst":"20200420","dateLast":"20200420"}}
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$ java -jar avro-tools-1.9.2.jar tojson records.avro
> 20/04/24 11:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" org.apache.avro.AvroTypeException: Invalid default
> for field enrichmentHeader: null not a
> \{"type":"record","name":"EnrichmentHeader","namespace":"com.mxj142.commons.domain","fields":[{"name":"correlationId","type":{"type":"string","avro.java.string":"String"}},\{"name":"sourceId","type":{"type":"string","avro.java.string":"String"}}]}
> at org.apache.avro.Schema.validateDefault(Schema.java:1540)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.<init>(Schema.java:521)
> at org.apache.avro.Schema.parse(Schema.java:1647)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1382)
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130)
> at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90)
> at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:93)
> at org.apache.avro.tool.Main.run(Main.java:66)
> at org.apache.avro.tool.Main.main(Main.java:55)
> [mxj142:Jars]$
> --------------------------------------------------------------------------
>
> The file "records.avro" is generated using avro-1.8.2 . It contains two
> records and using "avro-tools-1.8.2.jar" one can read it. If we use
> "avro-tools-1.9.2.jar", then "records.avro" is not readable (as shown
> above). The file "records.avro" is also attached for your convenience.
>
> Let me know if you need any more information.
>
> Thanks,
> Manoj
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
