Re: Effort towards Avro 2.0?

2013-12-12 Thread Doug Cutting
On Wed, Dec 4, 2013 at 11:40 PM, Christophe Taton < christophe.ta...@gmail.com> wrote: > Well, I guess one can always handle such things externally to Avro. > This needn't be done externally. When an extension schema is encountered, the schema compiler can generate Object references, the DatumWr

Re: Effort towards Avro 2.0?

2013-12-04 Thread Christophe Taton
On Wed, Dec 4, 2013 at 12:18 PM, Douglas Creager wrote: > >- unwieldy because as a user, I'll have to encode and decode the bytes > >field manually everytime I want to access this field from the original > >record, unless I keep track of the decoded extension externally to the > >A

Re: Effort towards Avro 2.0?

2013-12-04 Thread Douglas Creager
>- inefficient because you'll end up serializing your data twice, once >from the actual type into the bytes field, then a second type as a >bytes field; I don't think it's as inefficient as you might think — the second serialization just blits the raw bytes content into some destinatio

Re: Effort towards Avro 2.0?

2013-12-04 Thread Christophe Taton
Hi Douglas, When you write a middleware that lets users define custom types, extensions are pretty much required. Middleware doesn't need to, and shouldn't need to know these user-defined custom types ahead of time : you don't want to rebuild and restart your middleware everytime a user define a

Re: Effort towards Avro 2.0?

2013-12-04 Thread Christophe Taton
On Tue, Dec 3, 2013 at 7:49 AM, Doug Cutting wrote: > On Mon, Dec 2, 2013 at 1:42 PM, Christophe Taton > wrote: > > - New extension data type, similar to ProtocolBuffer extensions > (incompatible change). > > Extensions might be implemented as something like: > > {"type":"record", "name":"exte

Re: Effort towards Avro 2.0?

2013-12-04 Thread Douglas Creager
On Tue, Dec 3, 2013, at 07:49 AM, Doug Cutting wrote: > On Mon, Dec 2, 2013 at 1:42 PM, Christophe Taton > wrote: > > - New extension data type, similar to ProtocolBuffer extensions > > (incompatible change). > > Extensions might be implemented as something like: > > {"type":"record", "name":

Effort towards Avro 2.0?

2013-12-03 Thread Christophe Taton
Hi all, Avro, in its current form, exhibits a number of limitations that are hard to work with or around, and hard to fix within the scope of Avro 1.x : fixing these issues would introduce incompatible changes that warrant a major version bump, ie. Avro 2.0. An Avro 2.0 branch would be an opportun

Re: Effort towards Avro 2.0?

2013-12-03 Thread Doug Cutting
On Mon, Dec 2, 2013 at 1:42 PM, Christophe Taton wrote: > - New extension data type, similar to ProtocolBuffer extensions (incompatible > change). Extensions might be implemented as something like: {"type":"record", "name":"extension", "fields":[ {"name":"fingerprint", "type": {"type":"fi

Re: Effort towards Avro 2.0?

2013-12-02 Thread Philip Zeyliger
It sounds like you're proposing to break language API compatibility. Are you also proposing to break wire compatibility for Avro HTTP RPC, Avro Netty RPC, and/or Avro datafiles? I'd be appreciative of a mechanism by which systems that happen to use Avro currently need not be forced to choose one

Effort towards Avro 2.0?

2013-12-02 Thread Christophe Taton
Hi all, Avro, in its current form, exhibits a number of limitations that are hard to work with or around, and hard to fix within the scope of Avro 1.x : fixing these issues would introduce incompatible changes that warrant a major version bump, ie. Avro 2.0. An Avro 2.0 branch would be an opportun