[
https://issues.apache.org/jira/browse/AVRO-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110570#comment-17110570
]
Sidharth Mohan commented on AVRO-2817:
--------------------------------------
HI [~rskraba]
Below is the code snippet from 1.8 and 1.9 from org.apache.avro.Schema
*1.8*
/**
* @param defaultValue the default value for this field specified using the
mapping
* in \{@link JsonProperties}
*/
public Field(String name, Schema schema, String doc,
Object defaultValue, Order order) {
this(name, schema, doc, JacksonUtils.toJsonNode(defaultValue), order);
-----------------
*1.9*
Field(String name, Schema schema, String doc, JsonNode defaultValue, boolean
validateDefault, Order order) {
super(FIELD_RESERVED);
[this.name = v|http://this.name/]alidateName(name);
this.schema = schema;
this.doc = doc;
this.defaultValue = validateDefault ? validateDefault(name, schema,
defaultValue) : defaultValue;
this.order = Objects.requireNonNull(order, "Order cannot be null");
}
The defaultValue true was added in 1.9 , we have billions of data which were
converted using 1.8 where the issue reported originally exist .. There is a
security vulnerability with Jackson databind which is forcing us to move to 1.9
We are not able to move to the newer version of avro as all of the old data
cannot be parsed because of the defaultValue true is enabled by default and
hence not backward compatible ..
Can we get a fix with the validation set to false by default , without this we
will be stuck to 1.8 and its impossible to convert billions of old data with
defaultValue null as the newer version is not backward compatible ..
> Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for
> certian type of schemas
> ------------------------------------------------------------------------------------------------------
>
> Key: AVRO-2817
> URL: https://issues.apache.org/jira/browse/AVRO-2817
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.9.2
> Reporter: Manoj Kumar Jha
> Priority: Blocker
> Attachments: records.avro
>
>
> Hello,
> We observed avro file generated using avro-1.8.2 is not readable while
> using avro-1.9.2 if schema contains a field "default:null". Please see
> below.
>
> --------------------------------------------------------------------------
> [mxj142:Jars]$ ls -ltr
> total 168520
> -rwx------@ 1 mxj142 staff 34798932 Jan 16 14:45 avro-tools-1.8.2.jar
> -rwxr-xr-x@ 1 mxj142 staff 51303364 Mar 6 17:43 avro-tools-1.9.2.jar
> -rw-r--r-- 1 mxj142 staff 715 Apr 24 11:19 records.avro
> [mxj142:Jars]$ java -jar avro-tools-1.8.2.jar tojson records.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {"enrichmentHeader":\{"correlationId":"1","sourceId":"09b5d770-7373-462d-9517-8e5ce957793f"},"cx":\{"dateFirst":"20200419","dateLast":"20200419"}}
> {"enrichmentHeader":\{"correlationId":"2","sourceId":"0285dc49-cc5c-4717-8ea2-45e50de2f65f"},"cx":\{"dateFirst":"20200420","dateLast":"20200420"}}
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$
> [mxj142:Jars]$ java -jar avro-tools-1.9.2.jar tojson records.avro
> 20/04/24 11:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" org.apache.avro.AvroTypeException: Invalid default
> for field enrichmentHeader: null not a
> \{"type":"record","name":"EnrichmentHeader","namespace":"com.mxj142.commons.domain","fields":[{"name":"correlationId","type":{"type":"string","avro.java.string":"String"}},\{"name":"sourceId","type":{"type":"string","avro.java.string":"String"}}]}
> at org.apache.avro.Schema.validateDefault(Schema.java:1540)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.<init>(Schema.java:521)
> at org.apache.avro.Schema.parse(Schema.java:1647)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1382)
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130)
> at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90)
> at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:93)
> at org.apache.avro.tool.Main.run(Main.java:66)
> at org.apache.avro.tool.Main.main(Main.java:55)
> [mxj142:Jars]$
> --------------------------------------------------------------------------
>
> The file "records.avro" is generated using avro-1.8.2 . It contains two
> records and using "avro-tools-1.8.2.jar" one can read it. If we use
> "avro-tools-1.9.2.jar", then "records.avro" is not readable (as shown
> above). The file "records.avro" is also attached for your convenience.
>
> Let me know if you need any more information.
>
> Thanks,
> Manoj
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
