Re: Avro schema doesn't honor backward compatibilty

Ryan Blue Tue, 02 Feb 2016 13:54:12 -0800

Raghvendra,

Yes, you have to keep track of the schema that a blob of bytes waswritten with if you want to read those bytes back correctly. If that'sinconvenient to keep track of, then I recommend keeping old versionsaround (1.avsc, 2.avsc, ...) and adding the schema fingerprint to thestart your serialized payload. That's commonly done when people use Avroto serialize Kafka messages and we're considering standardizing thepractice for interoperability.


rb

On 02/02/2016 11:44 AM, Raghvendra Singh wrote:

Hi Ryan

Thanks for your answer. Here is what i am doing in my environment

1. Write the data using the old schema

*SpecificDatumWriter<ControllerPayload> datumWriter = new
SpecificDatumWriter<>(SCHEMA_V1)*

2. Now trying to read the data written by the old schema using the new
schema

*DatumReader<ControllerPayload> payloadReader = new SpecificDatumReader<>(*
*SCHEMA_V2**)*

In this case *SCHEMA_V1 *is the old schema which doesn't have the field
while SCHEMA_V2 is the new one which has the extra field.

Your suggestion *"You should run setSchema on your SpecificDatumReader to
set the schema the data was written with"*  is kind of work around where i
have to read the data with the schema it was written with and hence this is
not exactly backward compatible. Note that if i do this then i have to
maintain all the schemas while reading and somehow know which version the
data was written with and hence this will make schema evolution pretty
painful.

Please let me know if i didn't understand your email correctly or their is
something i missed.

-raghu

On Tue, Feb 2, 2016 at 9:19 AM, Ryan Blue <[email protected]> wrote:

Hi Raghvendra,

It looks like the problem is that you're using the new schema in place of
the schema that the data was written with.  You should run setSchema on
your SpecificDatumReader to set the schema the data was written with.

What's happening is that the schema you're using, the new one, has the new
field so Avro assumes it is present and tries to read it. By setting the
schema that the data was actually written with, the datum reader will know
that it isn't present and will use your default instead. When you read data
encoded with the new schema, you need to use it as the written schema
instead so the datum reader knows that the field should be read.

Does that make sense?

rb

On 02/01/2016 12:31 PM, Raghvendra Singh wrote:

down votefavorite
<
http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#



I have this avro schema

{
   "namespace": "xx.xxxx.xxxxx.xxxxx",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
       {"name": "filed1",  "type": "string"},
       {"name": "filed2",     "type": "long"},
       {"name": "filed3",  "type": "boolean"},
       {
            "name" : "metrics",
            "type":
            {
               "type" : "array",
               "items":
               {
                   "name": "MyRecord",
                   "type": "record",
                   "fields" :
                       [
                         {"name": "min", "type": "long"},
                         {"name": "max", "type": "long"},
                         {"name": "sum", "type": "long"},
                         {"name": "count", "type": "long"}
                       ]
               }
            }
       }
    ]}

Here is the code which we use to parse the data

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
          DatumReader<MyPayLoad> payloadReader = new
SpecificDatumReader<>(MyPayLoad.class);
          Decoder decoder = DecoderFactory.get().binaryDecoder(payload,
null);
          MyPayLoad myPayLoad = null;
          try {
              myPayLoad = payloadReader.read(null, decoder);
          } catch (IOException e) {
              logger.log(Level.SEVERE, e.getMessage(), e);
          }

          return myPayLoad;
      }

Now i want to add one more field int the schema so the schema looks like
below

   {
   "namespace": "xx.xxxx.xxxxx.xxxxx",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
       {"name": "filed1",  "type": "string"},
       {"name": "filed2",     "type": "long"},
       {"name": "filed3",  "type": "boolean"},
       {
            "name" : "metrics",
            "type":
            {
               "type" : "array",
               "items":
               {
                   "name": "MyRecord",
                   "type": "record",
                   "fields" :
                       [
                         {"name": "min", "type": "long"},
                         {"name": "max", "type": "long"},
                         {"name": "sum", "type": "long"},
                         {"name": "count", "type": "long"}
                       ]
               }
            }
       }
       {"name": "agentType",  "type": ["null", "string"], "default":
"APP_AGENT"}
    ]}

Note the filed added and also the default is defined. The problem is that
if we receive the data which was written using the older schema i get this
error

java.io.EOFException: null
      at org.apache.avro.io
.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io
.BinaryDecoder.readIndex(BinaryDecoder.java:423)
~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io
.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
~[avro-1.7.4.jar:1.7.4]
      at org.apache.avro.io
.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
~[avro-1.7.4.jar:1.7.4]
      at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
~[avro-1.7.4.jar:1.7.4]
      at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
~[avro-1.7.4.jar:1.7.4]
      at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
~[avro-1.7.4.jar:1.7.4]
      at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
~[avro-1.7.4.jar:1.7.4]
      at
com.appdynamics.blitz.shared.util.XXXXXXXXXXXXX.parseBinaryPayload(BlitzAvroSharedUtil.java:38)
~[blitz-shared.jar:na]

What i understood from this
<
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

document
that this should have been backward compatible but somehow that doesn't
seem to be the case. Any idea what i am doing wrong?


--
Ryan Blue
Software Engineer
Cloudera, Inc.



--
Ryan Blue
Software Engineer
Cloudera, Inc.

Re: Avro schema doesn't honor backward compatibilty

Reply via email to