Hi Prakash, Which version of Pulsar are you using?
If you are using 2.3.x or earlier versions, the pulsar client doesn't use the schema version associated with messages as the writer schema. So you will encounter this EOF issue. The schema version is used in latest master, which will be released as part of 2.4.0. If you want to test this feature, try to compile the latest master and test the schema evolution. The pulsar team is about to start the release process for 2.4.0. Hopefully it will come out next week. Stay tuned. Also Jenifer is working on putting up a comprehensive documentation for schema in 1~2 weeks or so. Once that is out, it will make users easier to use Schema in Pulsar. Thanks, Sijie On Thu, Jun 20, 2019 at 8:46 AM Prakash Saraf <[email protected]> wrote: > Hi, > > We are trying to evaluate the "Schema evolution" with Apache Pulsar and > AVRO. > We tried the below use-case - > > Created a "TestMessage" with id,string1 attributes. Published/consumed > 100k messages on a topic. This essentially uploaded first schema version > for the topic. > > We now produced another 100k messages for the topic. It is not consumed > yet. > > Now, we changed "TestMessage" and added another attribute "string2". > Produced another 100k messages on same topic. > > If we now start consumer on the topic, we get the below error - > > org.apache.pulsar.client.api.SchemaSerializationException: > java.io.EOFException > > at org.apache.pulsar.client.impl.schema.AvroSchema.decode( > AvroSchema.java:115) > > at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:247 > ) > > at com.x.pulsarclient.demo.PulsarProducer$PulsarConsumer.run( > PulsarProducer.java:94) > > at java.base/java.lang.Thread.run(Thread.java:834) > > Caused by: java.io.EOFException > > at org.apache.pulsar.shade.org.apache.avro.io.BinaryDecoder.ensureBounds( > BinaryDecoder.java:473) > > > > On some search on open issues, came across > *https://issues.apache.org/jira/browse/AVRO-1440 > <https://issues.apache.org/jira/browse/AVRO-1440>* which mentions that > reader must have access to writers schema. Given, we wanted to leverage the > "Reflection" way of runtime AVRO schema generation and usage, how can we do > this? Both producer and consumer are on same JVM. Even, if they were on > different, the topic will have messages generated by two different schemas > - how will consumer consume both old and new messages? > > Can anyone shed some light on this topic? What am I missing? > Any help is much appreciated. > > Thanks, > Prakash >
