Re: Avro union compatibility mode enhancement proposal

I had the same thoughts as Yacine and I was surprised that adding a new union type or a new enum value is not a forward compatible change (given that there is a default value, e.g. null). https://stackoverflow.com/questions/54439315/avro-union-forward-compatibility

Re: Avro union compatibility mode enhancement proposal

We are facing exactly the same forward compatibility issue for unions. What is the status of this work? Is there a related jira? -- Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Developers-f679485.html

Re: Avro union compatibility mode enhancement proposal

Is there a JIRA for Matthieu Monsch proposal of writer aliases? -- View this message in context: http://apache-avro.679487.n3.nabble.com/Avro-union-compatibility-mode-enhancement-proposal-tp4034377p4037426.html Sent from the Avro - Developers mailing list archive at Nabble.com.

Re: Avro union compatibility mode enhancement proposal

Thanks for the feedback Doug! I think the change would still fit under the "formerly known as" umbrella. More specifically, the current implementation implements a "formerly read as" and the proposed suggestion would also allow the "formerly written as" counterpart. Perhaps a new example would

Re: Avro union compatibility mode enhancement proposal

Matthieu, Thanks for the example. First, is this really an alias, or is it something else? In other words, would a reader ever map a written Vehicle to a Bus? If the use cases are exclusive, perhaps we should call it something different rather than overload the alias concept? Second, would

Re: Avro union compatibility mode enhancement proposal

Just realized I forgot to change the `id` fields to long in the first union (all IDs should be longs). Apologies for the confusion, they don’t matter at all in the example. > On Jun 11, 2016, at 7:43 PM, Matthieu Monsch wrote: > > Happy to provide an example. Let’s

Re: Avro union compatibility mode enhancement proposal

Happy to provide an example. Let’s assume that we have a Kafka producer emitting the following values: union { record Vehicle { int id; }, record Car { int id; boolean selfDriving; } } At a later point in time, a new vehicle becomes supported by the system and must be added

Re: Avro union compatibility mode enhancement proposal

Matthieu, Can you please provide an example of how this would work? Thanks, Doug On Thu, Jun 9, 2016 at 6:47 PM, Matthieu Monsch wrote: > Thinking about this a bit more (and a couple months later…), maybe there > is a simpler alternative. > > Currently, a reason why

Re: Avro union compatibility mode enhancement proposal

Thinking about this a bit more (and a couple months later…), maybe there is a simpler alternative. Currently, a reason why writer evolution is hard (the union issue described below is a special case of this) is that aliases are only used on the reader side. Why not also allow readers to use

Re: Avro union compatibility mode enhancement proposal

The second solution sounds like a great alternative. Branch aliases are more straightforward than an implicit order-sensitive policy. They also have the additional benefit of giving users a bit more flexibility: since defaults are specified on the branches’ types, it is possible to have

Re: Avro union compatibility mode enhancement proposal

On Wed, Apr 20, 2016 at 9:09 PM, Ryan Blue wrote: > Making the default a property of an > inner schema makes me think that we will have to deal with multiple schemas > with such a label at some point. On Thu, Apr 21, 2016 at 6:54 AM, Matthieu Monsch

Re: Avro union compatibility mode enhancement proposal

I worry that an implicit resolution logic will cause confusion. According to the spec, records should only match if their names (or one of the aliases) match. From what I understand, some implementations are more flexible when names are missing (which is fine since user-generated records should

Re: Avro union compatibility mode enhancement proposal

Interesting idea, Doug. I was thinking about just updating the logic like the fix in AVRO-1590 , but it would be nice to have a way to do this in the spec. What do you think about dealing with this by updating the union branch matching rules (and

Re: Avro union compatibility mode enhancement proposal

I was responding to the "For unions, we will add an optional catch-all attribute" part. The syntax of union schemas is unfortunately hard to modify compatibly. Here's a way around that. Recall that every kind of schema, except union, supports arbitrary properties, and unions cannot be directly

Re: Avro union compatibility mode enhancement proposal

Doug, I don't think I understand. Why would this change a union representation? This wouldn't change the schema format, other than to add an attribute to enum types that is ignored by older readers. New readers will use that attribute to determine which symbol to use when the written symbol is

Re: Avro union compatibility mode enhancement proposal

Perhaps then its sufficient to only write the new schema format when the new attribute is specified, so existing apps will continue to represent unions as JSON arrays? If so, this should probably be written into the spec. On Mon, Apr 18, 2016 at 12:52 PM, Ryan Blue

Re: Avro union compatibility mode enhancement proposal

Isn't the problem that these changes aren't compatible right now anyway? If I need to add an entry to an enum right now, older readers fail when trying to handle that data. This creates a way to avoid that failure in new versions. On Mon, Apr 18, 2016 at 12:48 PM, Doug Cutting

Re: Avro union compatibility mode enhancement proposal

On Sun, Apr 17, 2016 at 2:00 PM, Matthieu Monsch wrote: > + For unions, we will add an optional catch-all attribute to mark a branch as > resolution target when no names or aliases match (and come up with the > corresponding syntax). Can this be compatible? If you add a

Re: Avro union compatibility mode enhancement proposal

Great! Summarizing: + For `enum`s, we can go with the approach described in AVRO-1340 [1]. The only missing step is to agree on the `.avsc` and `.avdl` syntax (can be discussed in that ticket). + For unions, we will add an optional catch-all attribute to mark a branch as resolution target

Re: Avro union compatibility mode enhancement proposal

+1 On Sat, Apr 16, 2016 at 5:54 PM, Matthieu Monsch wrote: > > I think it may make sense for > > enums to add a special value for it. > > That would work great. To give slightly more flexibility to users, we > could even allow the reader’s schema to explicitly specify which

Re: Avro union compatibility mode enhancement proposal

> I think it may make sense for > enums to add a special value for it. That would work great. To give slightly more flexibility to users, we could even allow the reader’s schema to explicitly specify which symbol to use when it reads an unknown symbol. If not specified, resolution would fail

Re: Avro union compatibility mode enhancement proposal

Matthieu, how would that work with enums? I think it may make sense for enums to add a special value for it. rb On Sat, Apr 16, 2016 at 1:23 PM, Matthieu Monsch wrote: > > I think that substituting different data for > > unrecognized branches in a union isn't the way to

Re: Avro union compatibility mode enhancement proposal

> I think that substituting different data for > unrecognized branches in a union isn't the way to fix the problem, but I > have a proposal for a way to fix it that looks more like you'd expect in > the OO example above: by adding the Vehicle class to your read schema. > > Right now, unions are

Fwd: Avro union compatibility mode enhancement proposal

Yacine, Thanks for the extra information. Sorry for my delay in replying, I wanted to time think about your suggestion. I see what you mean that you can think of a union as the superclass of its options. The reflect object model has test code that does just that, where classes B and C inherit

Re: Avro union compatibility mode enhancement proposal

Hi Yacine, I believe Ryan was mentioning that if you start from a schema `[“null”, “Car”]` then rather than add a new bus branch to the union, you could update the car’s schema to include new bus fields. For example (using IDL notation): > // Initial approach > // Evolved to union { null, Car,

Re: Avro union compatibility mode enhancement proposal

Hi Ryan, Thank you for giving feedback. I will try in the following to provide you some more details about the addressed problem. But before that, just a brief reminder of the context. Avro has been chosen in this project (and by many other ones for sure) especially for a very important feature:

Re: Avro union compatibility mode enhancement proposal

Hi Yacine, Thanks for the proposal. It sounds interesting, but I want to make sure there's a clear use case for this because it's a significant change to the current evolution rules. Right now we guarantee that a reader will get an error if the data has an unknown union branch rather than getting

Avro union compatibility mode enhancement proposal

Hi all, In order to provide a solution to the union schema evolution problem, as it was earlier notified in the thread "add a type to a union " of the user mailing list, we decided, for the needs of the reactive