sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469563201 @jerrypeng correct. I added one more to complete the picture. ``` log.info("Pulsar AllowNull :{}", new String(AvroSchema.of(User.class).getSchemaInfo().getSchema())); log.info("Pulsar shaded AVRO NOT AllowNull :{}", org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.get().getSchema(User.class)); log.info("Avro AllowNull :{}", ReflectData.AllowNull.get().getSchema(User.class)); log.info("Avro Not AllowNull :{}", ReflectData.get().getSchema(User.class)); ``` ``` Pulsar AllowNull :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":["null","string"],"default":null},{"name":"favorite_number","type":["null","int"],"default":null},{"name":"favorite_color","type":["null","string"],"default":null},{"name":"age","type":"int"}]} Pulsar shaded AVRO NOT AllowNull :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":"int"},{"name":"favorite_color","type":"string"},{"name":"age","type":"int"}]} Avro AllowNull :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":"string"},{"name":"age","type":"int","default":19}]} Avro Not AllowNull :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":"string"},{"name":"age","type":"int","default":19}]} ``` As you described above, AllowNull doesn't matter if the POJO is generated by AVRO, because AVRO will try to use `SCHEMA$` field to retrieve the schema. But If the POJO is not generated by AVRO, AVRO will attempt to parse it and AllowNull will generating two different incompatible schemas. my tests avoid covered all the cases. 1) `Avro AllowNull` and `Avro Not AllowNull` are using unshaded avro library to parse avro generated POJO. in this way, they will always produce the same schema no matter `AllowNull` used or not. 2) `Pulsar AllowNull` and `Pulsar shade avro Not Allow Null` are using shaded avro library to parse avro generated POJO. In this way, they will produce a completely different schema. `AllowNull` matters here. The 2) indicates two problems: a) for normal POJO (not generated by AVRO), `AllowNull` is producing incompatible schemas. so we need #3752 b) for avro generated POJO, shaded AVRO library (the one used by Pulsar) doesn't recognize it is an AVRO generated POJO. We need a separate fix to make sure Pulsar shaded AVRO working with avro generated POJO.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
