Re: Avro 1.8.2 release

[S G]

Thanks for merging the patch Ryan. Appreciate it !

On Sun, Oct 30, 2016 at 1:59 PM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> SG,
>
> I'll take a look at it. Thanks!
>
> rb
>
> On Sun, Oct 30, 2016 at 1:58 PM, S G <sg.online.em...@gmail.com> wrote:
>
> > Hey Ryan,
> >
> > Any chance https://github.com/apache/avro/pull/113 can make it to this
> > release?
> > Its been pending review for a couple months now.
> >
> > Thanks
> > SG
> >
> > On Sun, Oct 30, 2016 at 1:46 PM, Ryan Blue <b...@apache.org> wrote:
> >
> > > Hi everyone,
> > >
> > > I'm going to try to get a 1.8.2 release candidate out this week.
> There's
> > > only one issue that I think is a blocker on AVRO-1885, and I think we
> can
> > > get that bug fixed. If anyone would like to get other commits in or
> think
> > > there are more blockers, please link them to AVRO-1885 and we can
> discuss
> > > on that issue.
> > >
> > > Thanks!
> > >
> > > rb
> > >
> > > --
> > > Ryan Blue
> > >
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Avro 1.8.2 release

[S G]

Hey Ryan,

Any chance https://github.com/apache/avro/pull/113 can make it to this
release?
Its been pending review for a couple months now.

Thanks
SG

On Sun, Oct 30, 2016 at 1:46 PM, Ryan Blue  wrote:

> Hi everyone,
>
> I'm going to try to get a 1.8.2 release candidate out this week. There's
> only one issue that I think is a blocker on AVRO-1885, and I think we can
> get that bug fixed. If anyone would like to get other commits in or think
> there are more blockers, please link them to AVRO-1885 and we can discuss
> on that issue.
>
> Thanks!
>
> rb
>
> --
> Ryan Blue
>

Re: Need review/merges for couple of pull requests

[S G]

Hey Ryan,

Any chance you were able to look into this (My email on Jun 6)?

Thanks
SG

On Thu, Jun 16, 2016 at 2:05 PM, S G <sg.online.em...@gmail.com> wrote:

> Doug Cutting, Ryan Blue,
>
> I have worked with both of you on this ticket and would appreciate your
> thoughts regarding circular references.
>
> If you think the current circular references' implementation in Avro is
> indeed easy for clients to use (i.e. they do not have to write all those
> extra classes), then please help me by providing some test-case/code
> reference.
>
> Thanks
> Sachin
>
>
> On Mon, Jun 6, 2016 at 9:18 AM, S G <sg.online.em...@gmail.com> wrote:
>
>>
>> As far as I have understood the attached testcase, I think the solution
>> in https://github.com/apache/avro/pull/23/files (JIRA:
>> https://issues.apache.org/jira/browse/AVRO-695) is more seamless from a
>> user's perspective because the user does not have to write any classes
>> extending LogicalType or Conversion.
>>
>> Writing those classes could become a bit of a challenge:
>> 1) If there are several circular references in the code (something very
>> common in the Hibernate world).
>> 2) It is a pain to change these classes to reflect any changes in the
>> classes being serialized.
>> 3) When the user decides to write a conversion/logical-type class for
>> some record, he has to handle serialization for all fields of the class
>> even though all he wanted to handle was circular references (This is what I
>> have understood but I could be wrong here).
>>
>> In my pull request for AVRO-695, the only thing a user need to do is to
>> call setResolvingCircularRefs(true) on ReflectData. Then every record is
>> converted into a union of the record-type and the newly added class
>> CircularRef. This new class stores the actual reference ID on encountering
>> a circular ref. This is really convenient for the user and there is no
>> penalty on performance. Even if there is a problem, the user can always go
>> back to writing LogicalType and Conversion classes.
>>
>> My request would be to consider this solution for handling circular
>> references instead of asking the user to write a bunch of other classes.
>> If the solution sounds acceptable, then I can bring the patch up-to-date
>> with the latest master branch and reopen the pull request.
>>
>> Cheers,
>> SG
>>
>> On Wed, Jun 1, 2016 at 5:36 PM, S G <sg.online.em...@gmail.com> wrote:
>>
>>> Thanks RB,
>>>
>>> But I am not sure I follow that example (Probably because there is no
>>> actual datum class there?).
>>>
>>> Consider a complex code running in production with lots of circular
>>> references.
>>> Something like:
>>>
>>> class Home {
>>>String address;
>>>Integer zipCode;
>>>List  residents;
>>> }
>>>
>>> class Person {
>>> String name;
>>> Person spouse;
>>> House home;
>>> Map <Car, Dealer> vehiclesOwned;
>>> }
>>>
>>> class Car {
>>>  String make;
>>>  Integer year;
>>>  Dealer soldBy;
>>>  Person ownedBy;
>>> }
>>>
>>> class Dealer {
>>> String name;
>>> String address;
>>> List vehiclesSold;
>>> }
>>>
>>> The above is a made-up example, intentionally using easy-to-understand
>>> references.
>>> But it does show that circular references can become very complex and
>>> across multiple chains.
>>>
>>> If users have to write Conversion classes for all of the above along
>>> with hash-map(s) that finds out IDs for the objects seen already,  writes
>>> them out as separate fields and then reads them back to replace them with
>>> actual objects, then it would be expecting too much from our clients IMO.
>>>
>>> Perhaps it would be more helpful if there was a more seamless way of
>>> getting this done automatically.
>>>
>>> SG
>>>
>>>
>>>
>>> On Wed, Jun 1, 2016 at 5:22 PM, Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> Those aren't the datum classes, those are logical types that are added
>>>> to
>>>> the schema for your datum classes. The "referenceable" logical type is
>>>> applied to the class that gets replaced with an ID reference and points
>>>> to
>>>> the field to use for that ID. The "refe

Re: [GitHub] avro pull request #113: AVRO-1882: ConcurrentHashMap with non-string keys fa...

[S G]

Hi,

Its been about a month since this PR was submitted and there has been no
action on this.
In total, 35 pull requests are open and pending approval/rejection/review:
https://github.com/apache/avro/pulls

Does anyone look at opened PRs regularly?

Thanks
SG

On Thu, Jul 28, 2016 at 10:36 AM, S G <sg.online.em...@gmail.com> wrote:

>
> Hi,
>
> I submitted this pull request and seems to have the approval of a
> committer.
> Please let me know how I can get this merged.
>
> Thanks
> SG
>
> On Mon, Jul 25, 2016 at 2:37 PM, sachingsachin <g...@git.apache.org> wrote:
>
>> GitHub user sachingsachin opened a pull request:
>>
>> https://github.com/apache/avro/pull/113
>>
>> AVRO-1882: ConcurrentHashMap with non-string keys fails in Java 1.8
>>
>> Also updating maven-shade-plugin as per [this thread](
>> http://search-hadoop.com/m/F2svI1oJXGf1htLgb=
>> Re+Unable+to+build+clean+checkout+of+avro+from+master)
>>
>> You can merge this pull request into a Git repository by running:
>>
>> $ git pull https://github.com/sachingsachin/avro AVRO_1882
>>
>> Alternatively you can review and apply these changes as the patch at:
>>
>> https://github.com/apache/avro/pull/113.patch
>>
>> To close this pull request, make a commit to your master/trunk branch
>> with (at least) the following in the commit message:
>>
>> This closes #113
>>
>> 
>> commit faf46e81434adfd4b37844de67c01b39ab478ba2
>> Author: Sachin Goyal <sgoyal6@l-sb8m1heg8w-m.local>
>> Date:   2016-07-25T21:32:46Z
>>
>> AVRO-1882: ConcurrentHashMap with non-string keys fails in Java 1.8
>>
>> 
>>
>>
>> ---
>> If your project is set up for it, you can reply to this email and have
>> your
>> reply appear on GitHub as well. If your project does not have this feature
>> enabled and wishes so, or if the feature is enabled but not working,
>> please
>> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
>> with INFRA.
>> ---
>>
>
>

Re: [GitHub] avro pull request #114: AVRO-1889: Upgrade maven-shade plugin

[S G]

Hi,

Its been about a month since this PR was submitted and there has been no
action on this.
In total, 35 pull requests are open and pending approval/rejection/review:
https://github.com/apache/avro/pulls

Does anyone look at opened PRs regularly?

Thanks
SG

On Thu, Jul 28, 2016 at 10:36 AM, S G <sg.online.em...@gmail.com> wrote:

>
> Hi,
>
> I submitted this pull request sometime back.
> Please let me know how I can get this merged.
>
> Thanks
> SG
>
>
> On Mon, Jul 25, 2016 at 3:15 PM, sachingsachin <g...@git.apache.org> wrote:
>
>> GitHub user sachingsachin opened a pull request:
>>
>> https://github.com/apache/avro/pull/114
>>
>> AVRO-1889: Upgrade maven-shade plugin
>>
>>
>>
>> You can merge this pull request into a Git repository by running:
>>
>> $ git pull https://github.com/sachingsachin/avro AVRO_1889
>>
>> Alternatively you can review and apply these changes as the patch at:
>>
>> https://github.com/apache/avro/pull/114.patch
>>
>> To close this pull request, make a commit to your master/trunk branch
>> with (at least) the following in the commit message:
>>
>> This closes #114
>>
>> 
>> commit c10e6af948d9407ef544a469fa3d73d0facf3a88
>> Author: Sachin Goyal <sgoyal6@l-sb8m1heg8w-m.local>
>> Date:   2016-07-25T22:15:10Z
>>
>> AVRO-1889: Upgrade maven-shade plugin
>>
>> 
>>
>>
>> ---
>> If your project is set up for it, you can reply to this email and have
>> your
>> reply appear on GitHub as well. If your project does not have this feature
>> enabled and wishes so, or if the feature is enabled but not working,
>> please
>> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
>> with INFRA.
>> ---
>>
>
>

Re: [GitHub] avro pull request #114: AVRO-1889: Upgrade maven-shade plugin

[S G]

Hi,

I submitted this pull request sometime back.
Please let me know how I can get this merged.

Thanks
SG


On Mon, Jul 25, 2016 at 3:15 PM, sachingsachin  wrote:

> GitHub user sachingsachin opened a pull request:
>
> https://github.com/apache/avro/pull/114
>
> AVRO-1889: Upgrade maven-shade plugin
>
>
>
> You can merge this pull request into a Git repository by running:
>
> $ git pull https://github.com/sachingsachin/avro AVRO_1889
>
> Alternatively you can review and apply these changes as the patch at:
>
> https://github.com/apache/avro/pull/114.patch
>
> To close this pull request, make a commit to your master/trunk branch
> with (at least) the following in the commit message:
>
> This closes #114
>
> 
> commit c10e6af948d9407ef544a469fa3d73d0facf3a88
> Author: Sachin Goyal 
> Date:   2016-07-25T22:15:10Z
>
> AVRO-1889: Upgrade maven-shade plugin
>
> 
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: [GitHub] avro pull request #113: AVRO-1882: ConcurrentHashMap with non-string keys fa...

[S G]

Hi,

I submitted this pull request and seems to have the approval of a committer.
Please let me know how I can get this merged.

Thanks
SG

On Mon, Jul 25, 2016 at 2:37 PM, sachingsachin  wrote:

> GitHub user sachingsachin opened a pull request:
>
> https://github.com/apache/avro/pull/113
>
> AVRO-1882: ConcurrentHashMap with non-string keys fails in Java 1.8
>
> Also updating maven-shade-plugin as per [this thread](
> http://search-hadoop.com/m/F2svI1oJXGf1htLgb=Re+Unable+to+build+clean+checkout+of+avro+from+master
> )
>
> You can merge this pull request into a Git repository by running:
>
> $ git pull https://github.com/sachingsachin/avro AVRO_1882
>
> Alternatively you can review and apply these changes as the patch at:
>
> https://github.com/apache/avro/pull/113.patch
>
> To close this pull request, make a commit to your master/trunk branch
> with (at least) the following in the commit message:
>
> This closes #113
>
> 
> commit faf46e81434adfd4b37844de67c01b39ab478ba2
> Author: Sachin Goyal 
> Date:   2016-07-25T21:32:46Z
>
> AVRO-1882: ConcurrentHashMap with non-string keys fails in Java 1.8
>
> 
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: Unable to build clean checkout of avro from master

[S G]

The problem was an older version of shade plugin in lang/java/pom.xml
I changed it from 1.7.1 to 2.4.3 and everything built just fine.
We should probably change this and check it in too.

On Fri, Jul 8, 2016 at 5:03 PM, S G <sg.online.em...@gmail.com> wrote:

> strange... I did a clean checkout of the code, removed ~/.m2/repository
> and built from
>
> avro/lang/java  as well as from avro/, but I get the same result.
>
> On Fri, Jul 8, 2016 at 3:38 PM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> hurm.
>>
>> When I do the same steps is finishes fine:
>>
>> https://paste.apache.org/iWLJ
>>
>> What if you try building from the top level rather than lang/java/avro?
>>
>> -Sean
>>
>> On Fri, Jul 8, 2016 at 4:23 PM, S G <sg.online.em...@gmail.com> wrote:
>> > mvn --version is as follows:
>> >
>> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> > 2015-11-10T08:41:47-08:00)
>> > Maven home: ~/work/software/maven
>> > Java version: 1.8.0_71, vendor: Oracle Corporation
>> > Java home:
>> > /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/jre
>> > Default locale: en_US, platform encoding: UTF-8
>> > OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"
>> >
>> >
>> >
>> > On Fri, Jul 8, 2016 at 1:02 PM, Sean Busbey <bus...@cloudera.com>
>> wrote:
>> >
>> >> Could you copy/paste the output of running "mvn --version" ?
>> >>
>> >> On Fri, Jul 8, 2016 at 2:34 PM, S G <sg.online.em...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I am unable to build a clean checkout of avro from master.
>> >> >
>> >> > Is it a known issue or I am doing something wrong?
>> >> >
>> >> >
>> >> > cd lang/java/avro
>> >> >
>> >> > mvn clean install
>> >> >
>> >> > [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @
>> avro
>> >> ---
>> >> > [WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT :
>> Split
>> >> > package, multiple jars provide the same package:org/apache/avro
>> >> > Use Import/Export Package directive
>> >> > -split-package:=(merge-first|merge-last|error|first) to get rid of
>> this
>> >> > warning
>> >> > Package found in   [Jar:., Jar:avro-guava-dependencies]
>> >> > Class path [Jar:., Jar:plexus-utils,
>> >> > Jar:avro-guava-dependencies, Jar:findbugs-annotations,
>> >> > Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
>> >> > Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
>> >> > Jar:slf4j-api, Jar:slf4j-simple]
>> >> > [INFO]
>> >> > [INFO] --- maven-site-plugin:3.0:attach-descriptor
>> >> (attach-descriptor)
>> >> > @ avro ---
>> >> > [INFO]
>> >> > [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
>> >> > [INFO] Building jar:
>> >> >
>> >>
>> ~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
>> >> > [INFO]
>> >> > [INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
>> >> > [WARNING] Error injecting:
>> >> org.apache.maven.plugins.shade.mojo.ShadeMojo
>> >> > java.lang.NoClassDefFoundError:
>> >> > org/codehaus/plexus/util/xml/XmlStreamWriter
>> >> > at java.lang.Class.getDeclaredConstructors0(Native Method)
>> >> > at
>> java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>> >> > at java.lang.Class.getDeclaredConstructors(Class.java:2020)
>> >> > at
>> >> >
>> >>
>> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
>> >> > at
>> >> >
>> >>
>> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
>> >> > at
>> >> >
>> >>
>> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
>> >> > at
>> >> >
>> >>
>> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)

Re: Unable to build clean checkout of avro from master

[S G]

strange... I did a clean checkout of the code, removed ~/.m2/repository and
built from

avro/lang/java  as well as from avro/, but I get the same result.

On Fri, Jul 8, 2016 at 3:38 PM, Sean Busbey <bus...@cloudera.com> wrote:

> hurm.
>
> When I do the same steps is finishes fine:
>
> https://paste.apache.org/iWLJ
>
> What if you try building from the top level rather than lang/java/avro?
>
> -Sean
>
> On Fri, Jul 8, 2016 at 4:23 PM, S G <sg.online.em...@gmail.com> wrote:
> > mvn --version is as follows:
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T08:41:47-08:00)
> > Maven home: ~/work/software/maven
> > Java version: 1.8.0_71, vendor: Oracle Corporation
> > Java home:
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"
> >
> >
> >
> > On Fri, Jul 8, 2016 at 1:02 PM, Sean Busbey <bus...@cloudera.com> wrote:
> >
> >> Could you copy/paste the output of running "mvn --version" ?
> >>
> >> On Fri, Jul 8, 2016 at 2:34 PM, S G <sg.online.em...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I am unable to build a clean checkout of avro from master.
> >> >
> >> > Is it a known issue or I am doing something wrong?
> >> >
> >> >
> >> > cd lang/java/avro
> >> >
> >> > mvn clean install
> >> >
> >> > [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @
> avro
> >> ---
> >> > [WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT :
> Split
> >> > package, multiple jars provide the same package:org/apache/avro
> >> > Use Import/Export Package directive
> >> > -split-package:=(merge-first|merge-last|error|first) to get rid of
> this
> >> > warning
> >> > Package found in   [Jar:., Jar:avro-guava-dependencies]
> >> > Class path [Jar:., Jar:plexus-utils,
> >> > Jar:avro-guava-dependencies, Jar:findbugs-annotations,
> >> > Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
> >> > Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
> >> > Jar:slf4j-api, Jar:slf4j-simple]
> >> > [INFO]
> >> > [INFO] --- maven-site-plugin:3.0:attach-descriptor
> >> (attach-descriptor)
> >> > @ avro ---
> >> > [INFO]
> >> > [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
> >> > [INFO] Building jar:
> >> >
> >>
> ~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
> >> > [INFO]
> >> > [INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
> >> > [WARNING] Error injecting:
> >> org.apache.maven.plugins.shade.mojo.ShadeMojo
> >> > java.lang.NoClassDefFoundError:
> >> > org/codehaus/plexus/util/xml/XmlStreamWriter
> >> > at java.lang.Class.getDeclaredConstructors0(Native Method)
> >> > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> >> > at java.lang.Class.getDeclaredConstructors(Class.java:2020)
> >> > at
> >> >
> >>
> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
> >> > at
> >> >
> >>
> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
> >> > at
> >> >
> >>
> com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
> >> > at
> >> >
> 

Re: Unable to build clean checkout of avro from master

[S G]

mvn --version is as follows:

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T08:41:47-08:00)
Maven home: ~/work/software/maven
Java version: 1.8.0_71, vendor: Oracle Corporation
Java home:
/Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"



On Fri, Jul 8, 2016 at 1:02 PM, Sean Busbey <bus...@cloudera.com> wrote:

> Could you copy/paste the output of running "mvn --version" ?
>
> On Fri, Jul 8, 2016 at 2:34 PM, S G <sg.online.em...@gmail.com> wrote:
> > Hi,
> >
> > I am unable to build a clean checkout of avro from master.
> >
> > Is it a known issue or I am doing something wrong?
> >
> >
> > cd lang/java/avro
> >
> > mvn clean install
> >
> > [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @ avro
> ---
> > [WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT : Split
> > package, multiple jars provide the same package:org/apache/avro
> > Use Import/Export Package directive
> > -split-package:=(merge-first|merge-last|error|first) to get rid of this
> > warning
> > Package found in   [Jar:., Jar:avro-guava-dependencies]
> > Class path [Jar:., Jar:plexus-utils,
> > Jar:avro-guava-dependencies, Jar:findbugs-annotations,
> > Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
> > Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
> > Jar:slf4j-api, Jar:slf4j-simple]
> > [INFO]
> > [INFO] --- maven-site-plugin:3.0:attach-descriptor
> (attach-descriptor)
> > @ avro ---
> > [INFO]
> > [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
> > [INFO] Building jar:
> >
> ~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
> > [INFO]
> > [INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
> > [WARNING] Error injecting:
> org.apache.maven.plugins.shade.mojo.ShadeMojo
> > java.lang.NoClassDefFoundError:
> > org/codehaus/plexus/util/xml/XmlStreamWriter
> > at java.lang.Class.getDeclaredConstructors0(Native Method)
> > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> > at java.lang.Class.getDeclaredConstructors(Class.java:2020)
> > at
> >
> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
> > at
> >
> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
> > at
> >
> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
> > at
> >
> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
> > at
> >
> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
> > at
> >
> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
> > at
> >
> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
> > at
> >
> com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
> > at
> >
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1038)
> > at
> >
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1001)
> > at
> >
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
> > at
> >
> org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
> > at
> >
> com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
> > at
> >
> com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
> > at
> >
> com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65)
> > at
> >
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
> > at
> >
> org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176)
> > at
> >
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
> > at
> >
> com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerSt

Unable to build clean checkout of avro from master

[S G]

Hi,

I am unable to build a clean checkout of avro from master.

Is it a known issue or I am doing something wrong?


cd lang/java/avro

mvn clean install

[INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @ avro ---
[WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT : Split
package, multiple jars provide the same package:org/apache/avro
Use Import/Export Package directive
-split-package:=(merge-first|merge-last|error|first) to get rid of this
warning
Package found in   [Jar:., Jar:avro-guava-dependencies]
Class path [Jar:., Jar:plexus-utils,
Jar:avro-guava-dependencies, Jar:findbugs-annotations,
Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
Jar:slf4j-api, Jar:slf4j-simple]
[INFO]
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor)
@ avro ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
[INFO] Building jar:
~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
[WARNING] Error injecting: org.apache.maven.plugins.shade.mojo.ShadeMojo
java.lang.NoClassDefFoundError:
org/codehaus/plexus/util/xml/XmlStreamWriter
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at
com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
at
com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
at
com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
at
com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
at
com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
at
com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
at
com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
at
com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
at
com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1038)
at
com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1001)
at
com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
at
org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
at
com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at
com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
at
com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65)
at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
at
org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176)
at
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
at
com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
at
com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63)
at
com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
at
com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
at org.eclipse.sisu.inject.Guice4$1.get(Guice4.java:162)
at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81)
at
org.eclipse.sisu.plexus.LazyPlexusBean.getValue(LazyPlexusBean.java:51)
at
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:263)
at
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:255)
at
org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo(DefaultMavenPluginManager.java:517)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:121)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at

Re: Need review/merges for couple of pull requests

[S G]

Doug Cutting, Ryan Blue,

I have worked with both of you on this ticket and would appreciate your
thoughts regarding circular references.

If you think the current circular references' implementation in Avro is
indeed easy for clients to use (i.e. they do not have to write all those
extra classes), then please help me by providing some test-case/code
reference.

Thanks
Sachin


On Mon, Jun 6, 2016 at 9:18 AM, S G <sg.online.em...@gmail.com> wrote:

>
> As far as I have understood the attached testcase, I think the solution in
> https://github.com/apache/avro/pull/23/files (JIRA:
> https://issues.apache.org/jira/browse/AVRO-695) is more seamless from a
> user's perspective because the user does not have to write any classes
> extending LogicalType or Conversion.
>
> Writing those classes could become a bit of a challenge:
> 1) If there are several circular references in the code (something very
> common in the Hibernate world).
> 2) It is a pain to change these classes to reflect any changes in the
> classes being serialized.
> 3) When the user decides to write a conversion/logical-type class for some
> record, he has to handle serialization for all fields of the class even
> though all he wanted to handle was circular references (This is what I have
> understood but I could be wrong here).
>
> In my pull request for AVRO-695, the only thing a user need to do is to
> call setResolvingCircularRefs(true) on ReflectData. Then every record is
> converted into a union of the record-type and the newly added class
> CircularRef. This new class stores the actual reference ID on encountering
> a circular ref. This is really convenient for the user and there is no
> penalty on performance. Even if there is a problem, the user can always go
> back to writing LogicalType and Conversion classes.
>
> My request would be to consider this solution for handling circular
> references instead of asking the user to write a bunch of other classes.
> If the solution sounds acceptable, then I can bring the patch up-to-date
> with the latest master branch and reopen the pull request.
>
> Cheers,
> SG
>
> On Wed, Jun 1, 2016 at 5:36 PM, S G <sg.online.em...@gmail.com> wrote:
>
>> Thanks RB,
>>
>> But I am not sure I follow that example (Probably because there is no
>> actual datum class there?).
>>
>> Consider a complex code running in production with lots of circular
>> references.
>> Something like:
>>
>> class Home {
>>String address;
>>Integer zipCode;
>>List  residents;
>> }
>>
>> class Person {
>> String name;
>> Person spouse;
>> House home;
>> Map <Car, Dealer> vehiclesOwned;
>> }
>>
>> class Car {
>>  String make;
>>  Integer year;
>>  Dealer soldBy;
>>  Person ownedBy;
>> }
>>
>> class Dealer {
>> String name;
>> String address;
>> List vehiclesSold;
>> }
>>
>> The above is a made-up example, intentionally using easy-to-understand
>> references.
>> But it does show that circular references can become very complex and
>> across multiple chains.
>>
>> If users have to write Conversion classes for all of the above along with
>> hash-map(s) that finds out IDs for the objects seen already,  writes them
>> out as separate fields and then reads them back to replace them with actual
>> objects, then it would be expecting too much from our clients IMO.
>>
>> Perhaps it would be more helpful if there was a more seamless way of
>> getting this done automatically.
>>
>> SG
>>
>>
>>
>> On Wed, Jun 1, 2016 at 5:22 PM, Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Those aren't the datum classes, those are logical types that are added to
>>> the schema for your datum classes. The "referenceable" logical type is
>>> applied to the class that gets replaced with an ID reference and points
>>> to
>>> the field to use for that ID. The "reference" logical type is applied to
>>> the class that contains a referencable datum. Then there are conversions
>>> that replace a referenceable with its ID.
>>>
>>> On Wed, Jun 1, 2016 at 4:52 PM, S G <sg.online.em...@gmail.com> wrote:
>>>
>>> > RB,
>>> >
>>> > The test TestCircularReferences shows that the classes having circular
>>> > reference need to extend LogicalType.
>>> > If every circular reference class has to do this, then I think this
>>> would
>>> > be a big limitation fo

Re: Need review/merges for couple of pull requests

[S G]

As far as I have understood the attached testcase, I think the solution in
https://github.com/apache/avro/pull/23/files (JIRA:
https://issues.apache.org/jira/browse/AVRO-695) is more seamless from a
user's perspective because the user does not have to write any classes
extending LogicalType or Conversion.

Writing those classes could become a bit of a challenge:
1) If there are several circular references in the code (something very
common in the Hibernate world).
2) It is a pain to change these classes to reflect any changes in the
classes being serialized.
3) When the user decides to write a conversion/logical-type class for some
record, he has to handle serialization for all fields of the class even
though all he wanted to handle was circular references (This is what I have
understood but I could be wrong here).

In my pull request for AVRO-695, the only thing a user need to do is to
call setResolvingCircularRefs(true) on ReflectData. Then every record is
converted into a union of the record-type and the newly added class
CircularRef. This new class stores the actual reference ID on encountering
a circular ref. This is really convenient for the user and there is no
penalty on performance. Even if there is a problem, the user can always go
back to writing LogicalType and Conversion classes.

My request would be to consider this solution for handling circular
references instead of asking the user to write a bunch of other classes.
If the solution sounds acceptable, then I can bring the patch up-to-date
with the latest master branch and reopen the pull request.

Cheers,
SG

On Wed, Jun 1, 2016 at 5:36 PM, S G <sg.online.em...@gmail.com> wrote:

> Thanks RB,
>
> But I am not sure I follow that example (Probably because there is no
> actual datum class there?).
>
> Consider a complex code running in production with lots of circular
> references.
> Something like:
>
> class Home {
>String address;
>Integer zipCode;
>List  residents;
> }
>
> class Person {
> String name;
> Person spouse;
> House home;
> Map <Car, Dealer> vehiclesOwned;
> }
>
> class Car {
>  String make;
>  Integer year;
>  Dealer soldBy;
>  Person ownedBy;
> }
>
> class Dealer {
> String name;
> String address;
> List vehiclesSold;
> }
>
> The above is a made-up example, intentionally using easy-to-understand
> references.
> But it does show that circular references can become very complex and
> across multiple chains.
>
> If users have to write Conversion classes for all of the above along with
> hash-map(s) that finds out IDs for the objects seen already,  writes them
> out as separate fields and then reads them back to replace them with actual
> objects, then it would be expecting too much from our clients IMO.
>
> Perhaps it would be more helpful if there was a more seamless way of
> getting this done automatically.
>
> SG
>
>
>
> On Wed, Jun 1, 2016 at 5:22 PM, Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> Those aren't the datum classes, those are logical types that are added to
>> the schema for your datum classes. The "referenceable" logical type is
>> applied to the class that gets replaced with an ID reference and points to
>> the field to use for that ID. The "reference" logical type is applied to
>> the class that contains a referencable datum. Then there are conversions
>> that replace a referenceable with its ID.
>>
>> On Wed, Jun 1, 2016 at 4:52 PM, S G <sg.online.em...@gmail.com> wrote:
>>
>> > RB,
>> >
>> > The test TestCircularReferences shows that the classes having circular
>> > reference need to extend LogicalType.
>> > If every circular reference class has to do this, then I think this
>> would
>> > be a big limitation for actual classes used in production code because
>> they
>> > would be already extending other classes plus asking several other
>> teams to
>> > change their base-class would be difficult.
>> > Is there a way to do this without making the classes extend the
>> LogicalType
>> > ?
>> >
>> > SG
>> >
>> >
>> > On Wed, Jun 1, 2016 at 3:29 PM, Ryan Blue <rb...@netflix.com.invalid>
>> > wrote:
>> >
>> > > SG,
>> > >
>> > > The example/test I built uses logical types to remove the circular
>> > > reference when writing and restore it when reading. It doesn't look
>> like
>> > > your test adds logical types, so that's probably the issue.
>> > >
>> > > rb
>> > >
>> > > O

Re: Need review/merges for couple of pull requests

[S G]

Thanks RB,

But I am not sure I follow that example (Probably because there is no
actual datum class there?).

Consider a complex code running in production with lots of circular
references.
Something like:

class Home {
   String address;
   Integer zipCode;
   List  residents;
}

class Person {
String name;
Person spouse;
House home;
Map <Car, Dealer> vehiclesOwned;
}

class Car {
 String make;
 Integer year;
 Dealer soldBy;
 Person ownedBy;
}

class Dealer {
String name;
String address;
List vehiclesSold;
}

The above is a made-up example, intentionally using easy-to-understand
references.
But it does show that circular references can become very complex and
across multiple chains.

If users have to write Conversion classes for all of the above along with
hash-map(s) that finds out IDs for the objects seen already,  writes them
out as separate fields and then reads them back to replace them with actual
objects, then it would be expecting too much from our clients IMO.

Perhaps it would be more helpful if there was a more seamless way of
getting this done automatically.

SG



On Wed, Jun 1, 2016 at 5:22 PM, Ryan Blue <rb...@netflix.com.invalid> wrote:

> Those aren't the datum classes, those are logical types that are added to
> the schema for your datum classes. The "referenceable" logical type is
> applied to the class that gets replaced with an ID reference and points to
> the field to use for that ID. The "reference" logical type is applied to
> the class that contains a referencable datum. Then there are conversions
> that replace a referenceable with its ID.
>
> On Wed, Jun 1, 2016 at 4:52 PM, S G <sg.online.em...@gmail.com> wrote:
>
> > RB,
> >
> > The test TestCircularReferences shows that the classes having circular
> > reference need to extend LogicalType.
> > If every circular reference class has to do this, then I think this would
> > be a big limitation for actual classes used in production code because
> they
> > would be already extending other classes plus asking several other teams
> to
> > change their base-class would be difficult.
> > Is there a way to do this without making the classes extend the
> LogicalType
> > ?
> >
> > SG
> >
> >
> > On Wed, Jun 1, 2016 at 3:29 PM, Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> >
> > > SG,
> > >
> > > The example/test I built uses logical types to remove the circular
> > > reference when writing and restore it when reading. It doesn't look
> like
> > > your test adds logical types, so that's probably the issue.
> > >
> > > rb
> > >
> > > On Wed, Jun 1, 2016 at 1:51 PM, S G <sg.online.em...@gmail.com> wrote:
> > >
> > > > Hey RB,
> > > >
> > > > I was trying out the circular refs with latest 1.8.1 version of Avro
> > and
> > > it
> > > > doesn't seem to be working out of the box for me.
> > > > Perhaps I am missing something and would appreciate your help.
> > > >
> > > > Thanks,
> > > > SG
> > > >
> > > >
> > > > Here is my test code:
> > > >
> > > > public class CircularRefsTest
> > > > {
> > > > @Test
> > > > public void testSerialization() throws Exception
> > > > {
> > > > // Create a circular linked-list
> > > > ListNode first = new ListNode("first");
> > > > ListNode second = new ListNode("second");
> > > > second.next = first;
> > > > first.next = second;
> > > >
> > > > ReflectData rdata = ReflectData.AllowNull.get();
> > > >
> > > > // Print Schema
> > > > Schema schema = rdata.getSchema(ListNode.class);
> > > > System.out.println(schema);
> > > >
> > > > // Serialize
> > > > DatumWriter datumWriter = new
> > > > ReflectDatumWriter(ListNode.class);
> > > > ByteArrayOutputStream byteArrayOutputStream = new
> > > > ByteArrayOutputStream();
> > > > Encoder encoder =
> > > > EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null);
> > > > datumWriter.write(first, encoder);
> > > > encoder.flush();
> > > > byte[] bytes = byteArrayOutputStream.toByteArray();
> > > >
> > > > assertTrue( bytes != null );
> > > > }
> > > > 

Re: Need review/merges for couple of pull requests

[S G]

RB,

The test TestCircularReferences shows that the classes having circular
reference need to extend LogicalType.
If every circular reference class has to do this, then I think this would
be a big limitation for actual classes used in production code because they
would be already extending other classes plus asking several other teams to
change their base-class would be difficult.
Is there a way to do this without making the classes extend the LogicalType
?

SG


On Wed, Jun 1, 2016 at 3:29 PM, Ryan Blue <rb...@netflix.com.invalid> wrote:

> SG,
>
> The example/test I built uses logical types to remove the circular
> reference when writing and restore it when reading. It doesn't look like
> your test adds logical types, so that's probably the issue.
>
> rb
>
> On Wed, Jun 1, 2016 at 1:51 PM, S G <sg.online.em...@gmail.com> wrote:
>
> > Hey RB,
> >
> > I was trying out the circular refs with latest 1.8.1 version of Avro and
> it
> > doesn't seem to be working out of the box for me.
> > Perhaps I am missing something and would appreciate your help.
> >
> > Thanks,
> > SG
> >
> >
> > Here is my test code:
> >
> > public class CircularRefsTest
> > {
> > @Test
> > public void testSerialization() throws Exception
> > {
> > // Create a circular linked-list
> > ListNode first = new ListNode("first");
> > ListNode second = new ListNode("second");
> > second.next = first;
> > first.next = second;
> >
> > ReflectData rdata = ReflectData.AllowNull.get();
> >
> > // Print Schema
> > Schema schema = rdata.getSchema(ListNode.class);
> > System.out.println(schema);
> >
> > // Serialize
> > DatumWriter datumWriter = new
> > ReflectDatumWriter(ListNode.class);
> > ByteArrayOutputStream byteArrayOutputStream = new
> > ByteArrayOutputStream();
> > Encoder encoder =
> > EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null);
> > datumWriter.write(first, encoder);
> > encoder.flush();
> > byte[] bytes = byteArrayOutputStream.toByteArray();
> >
> > assertTrue( bytes != null );
> > }
> > }
> >
> > class ListNode
> > {
> > String data;
> > ListNode next;
> >
> > public ListNode() {
> > }
> > public ListNode(String data) {
> > this.data = data;
> > }
> > }
> >
> >
> >
> > On Thu, May 28, 2015 at 11:00 AM, Ryan Blue <b...@cloudera.com> wrote:
> >
> > > SG,
> > >
> > > The data ends up looking like this:
> > >
> > > {"id":1,"p":"parent data!","child":{"c":"child
> > data!","parent":{"long":1}}}
> > >
> > > I produced that with avro-tools 1.7.6 and tojson.
> > >
> > > Here's the schema: {
> > >   "type" : "record",
> > >   "name" : "Parent",
> > >   "fields" : [ {
> > >     "name" : "id",
> > > "type" : "long"
> > >   }, {
> > > "name" : "p",
> > > "type" : "string"
> > >   }, {
> > > "name" : "child",
> > > "type" : {
> > >   "type" : "record",
> > >   "name" : "Child",
> > >   "fields" : [ {
> > > "name" : "c",
> > > "type" : "string"
> > >   }, {
> > > "name" : "parent",
> > > "type" : [ "null", "long", "Parent" ]
> > >   } ],
> > >   "logicalType" : "reference",
> > >   "ref-field-name" : "parent"
> > > }
> > >   } ],
> > >   "logicalType" : "referenceable",
> > >   "id-field-name" : "id"
> > > }
> > >
> > > rb
> > >
> > >
> > > On 05/28/2015 09:50 AM, S G wrote:
> > >
> > >> RB,
> > >>
> > >> Could you please attach the schema and the JSON serialized output from
> > >> your
> > >> test-code as well?
> > >> 

Re: Need review/merges for couple of pull requests

[S G]

Hey RB,

I was trying out the circular refs with latest 1.8.1 version of Avro and it
doesn't seem to be working out of the box for me.
Perhaps I am missing something and would appreciate your help.

Thanks,
SG


Here is my test code:

public class CircularRefsTest
{
@Test
public void testSerialization() throws Exception
{
// Create a circular linked-list
ListNode first = new ListNode("first");
ListNode second = new ListNode("second");
second.next = first;
first.next = second;

ReflectData rdata = ReflectData.AllowNull.get();

// Print Schema
Schema schema = rdata.getSchema(ListNode.class);
System.out.println(schema);

// Serialize
DatumWriter datumWriter = new
ReflectDatumWriter(ListNode.class);
ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
Encoder encoder =
EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null);
datumWriter.write(first, encoder);
encoder.flush();
byte[] bytes = byteArrayOutputStream.toByteArray();

assertTrue( bytes != null );
}
}

class ListNode
{
String data;
ListNode next;

public ListNode() {
}
public ListNode(String data) {
this.data = data;
}
}



On Thu, May 28, 2015 at 11:00 AM, Ryan Blue <b...@cloudera.com> wrote:

> SG,
>
> The data ends up looking like this:
>
> {"id":1,"p":"parent data!","child":{"c":"child data!","parent":{"long":1}}}
>
> I produced that with avro-tools 1.7.6 and tojson.
>
> Here's the schema: {
>   "type" : "record",
>   "name" : "Parent",
>   "fields" : [ {
> "name" : "id",
> "type" : "long"
>   }, {
> "name" : "p",
> "type" : "string"
>   }, {
> "name" : "child",
> "type" : {
>   "type" : "record",
>   "name" : "Child",
>   "fields" : [ {
> "name" : "c",
> "type" : "string"
>   }, {
> "name" : "parent",
> "type" : [ "null", "long", "Parent" ]
>   } ],
>   "logicalType" : "reference",
>   "ref-field-name" : "parent"
> }
>   } ],
>   "logicalType" : "referenceable",
>   "id-field-name" : "id"
> }
>
> rb
>
>
> On 05/28/2015 09:50 AM, S G wrote:
>
>> RB,
>>
>> Could you please attach the schema and the JSON serialized output from
>> your
>> test-code as well?
>> My build environment is currently broken as I am grappling with some Java
>> 8
>> update issues.
>>
>> Thanks
>> SG
>>
>>
>> On Wed, May 27, 2015 at 3:28 PM, Ryan Blue <b...@cloudera.com> wrote:
>>
>> SG,
>>>
>>> Now that logical types are in, I had some time to look at this issue.
>>> Thanks for your patience on this.
>>>
>>> When I started looking at the use case, this began to look very much like
>>> a logical type issue. (I know, I've been saying that a lot.) When you
>>> write, you replace any referenced object with its id. When you read, you
>>> replace those ids with the correct object. I went ahead and implemented
>>> this using 2 logical types: Referenceable for an object with an id, and
>>> Reference for a record with a field that references another object by id.
>>>
>>> There were some trade-offs to this approach. For example, I had to use a
>>> logical type for the object containing the reference rather than for the
>>> reference itself because I used a callback to set the object once it is
>>> found. That happens because children with references to a parent are
>>> deserialized completely first. The parent is the last object to be
>>> assembled and passed to the logical type conversion.
>>>
>>> A bigger issue was that logical types are currently conservative and
>>> don't
>>> overlap with reflect types or anything that sets java-class. That means
>>> that this currently only works with generic types. But, I'd rather make
>>> logical types work for reflect than add more custom code to support this.
>>> Does that sound reasonable?
>>>
>>> I'm attaching a diff with the working test code so you can take a look at
>>> the approach.

Re: [GitHub] avro pull request: AVRO-1568: Allow Java polymorphism in Avro for ...

[S G]

Thanks RB,

I cannot find any example or documentation for the same.
If you can point me to some, then I will evaluate the same for my use case
and let you know how it goes.

Thanks
SG

On Mon, May 9, 2016 at 9:16 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:

> SG,
>
> I think this is addressed by logical types because you can register methods
> that handle conversion between any object and how you want to store it. I
> agree that we don't want to introduce inheritance, so I think this is a
> good direction to go. Are you trying to use logical types for this and
> running into trouble?
>
> rb
>
> On Thu, May 5, 2016 at 6:08 PM, S G <sg.online.em...@gmail.com> wrote:
>
> > Hey Ryan,
> >
> > I think the issue mentioned in
> > https://issues.apache.org/jira/browse/AVRO-1568 is still present.
> >
> > Do you know if logical types fixed it somehow?
> >
> > If not, we can work on merging the pull request in latest code-base.
> >
> > Thanks
> > SG
> >
> >
> > On Thu, Feb 4, 2016 at 2:53 PM, asfgit <g...@git.apache.org> wrote:
> >
> > > Github user asfgit closed the pull request at:
> > >
> > > https://github.com/apache/avro/pull/20
> > >
> > >
> > > ---
> > > If your project is set up for it, you can reply to this email and have
> > your
> > > reply appear on GitHub as well. If your project does not have this
> > feature
> > > enabled and wishes so, or if the feature is enabled but not working,
> > please
> > > contact infrastructure at infrastruct...@apache.org or file a JIRA
> > ticket
> > > with INFRA.
> > > ---
> > >
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [GitHub] avro pull request: AVRO-1568: Allow Java polymorphism in Avro for ...

[S G]

Hey Ryan,

I think the issue mentioned in
https://issues.apache.org/jira/browse/AVRO-1568 is still present.

Do you know if logical types fixed it somehow?

If not, we can work on merging the pull request in latest code-base.

Thanks
SG


On Thu, Feb 4, 2016 at 2:53 PM, asfgit  wrote:

> Github user asfgit closed the pull request at:
>
> https://github.com/apache/avro/pull/20
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: [DISCUSS] Quarterly release goal

[S G]

+1 for quarterly releases !

On Sat, Apr 16, 2016 at 7:03 PM, Thiruvalluvan MG <
thiru...@yahoo.com.invalid> wrote:

> +1.
> Thanks
> Thiru
>
> On Sunday, 17 April 2016 5:45 AM, Ryan Blue  wrote:
>
>
>  Hi everyone,
>
> It's been about 3 months since we released Avro 1.8.0 and we've already
> accumulated several fixes that we should get out in a release. Sean
> suggested it a few days ago, but I'm not sure if everyone saw that thread.
> Anyone interested in being the release manager for 1.8.1?
>
> I think we should set a goal of a release about once each quarter. If we
> let ourselves go 18 months between releases, then contributors can't use
> their work soon enough to continue caring and contributing.
>
> I think a quarterly release goal is a good first step toward making the
> project more contributor-friendly. For example, we usually have good
> participation getting ready for releases so it's a good excuse to get some
> reviews done in addition to getting committed fixes out the door.
>
> Thoughts and comments?
>
> rb
>
>
> --
> Ryan Blue
>
>
>
>

Re: Need review/merges for couple of pull requests

[S G]

RB,

Could you please attach the schema and the JSON serialized output from your
test-code as well?
My build environment is currently broken as I am grappling with some Java 8
update issues.

Thanks
SG


On Wed, May 27, 2015 at 3:28 PM, Ryan Blue b...@cloudera.com wrote:

 SG,

 Now that logical types are in, I had some time to look at this issue.
 Thanks for your patience on this.

 When I started looking at the use case, this began to look very much like
 a logical type issue. (I know, I've been saying that a lot.) When you
 write, you replace any referenced object with its id. When you read, you
 replace those ids with the correct object. I went ahead and implemented
 this using 2 logical types: Referenceable for an object with an id, and
 Reference for a record with a field that references another object by id.

 There were some trade-offs to this approach. For example, I had to use a
 logical type for the object containing the reference rather than for the
 reference itself because I used a callback to set the object once it is
 found. That happens because children with references to a parent are
 deserialized completely first. The parent is the last object to be
 assembled and passed to the logical type conversion.

 A bigger issue was that logical types are currently conservative and don't
 overlap with reflect types or anything that sets java-class. That means
 that this currently only works with generic types. But, I'd rather make
 logical types work for reflect than add more custom code to support this.
 Does that sound reasonable?

 I'm attaching a diff with the working test code so you can take a look at
 the approach. Let me know what you are thinking.

 rb

 On 05/20/2015 12:05 PM, S G wrote:

 I am requesting some help with AVRO-695.
 Here are some pieces from the last conversation.


 Doug Cutting
 https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cutting
 added
 a comment - 02/Oct/14 21:19

 Here's a modified version of the patch. It moves all significant changes
 to
 reflect. Since reflect is the only implementation that can generate an
 appropriate schema, changes should be confined to there.

 The tests need to be updated, as they still assume generic.
 https://issues.apache.org/jira/browse/AVRO-695#
 
 https://issues.apache.org/jira/browse/AVRO-695?focusedCommentId=14286370page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14286370
 
 Sachin Goyal
 https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sachingoyal
 added
 a comment - 21/Jan/15 21:56

 Here is a patch with the updated test-cases.
 I also confirm that all my changes are there in the patch.


 The patch was submitted in June 2014 and was very hot till October 2014.
 Since then, there has been no action on this even though I have sent many
 reminders in this group.

 I understand that everyone is very busy with their own stuff but I would
 really appreciate if someone could help a fellow engineer in getting his
 patch accepted.
 It would encourage more participation as well as help people wanting to
 use
 Avro for circular references.

 Regards
 SG



 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Re: Need review/merges for couple of pull requests

[S G]

I am requesting some help with AVRO-695.
Here are some pieces from the last conversation.


Doug Cutting
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cutting added
a comment - 02/Oct/14 21:19

Here's a modified version of the patch. It moves all significant changes to
reflect. Since reflect is the only implementation that can generate an
appropriate schema, changes should be confined to there.

The tests need to be updated, as they still assume generic.
https://issues.apache.org/jira/browse/AVRO-695#
https://issues.apache.org/jira/browse/AVRO-695?focusedCommentId=14286370page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14286370
Sachin Goyal
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sachingoyal added
a comment - 21/Jan/15 21:56

Here is a patch with the updated test-cases.
I also confirm that all my changes are there in the patch.


The patch was submitted in June 2014 and was very hot till October 2014.
Since then, there has been no action on this even though I have sent many
reminders in this group.

I understand that everyone is very busy with their own stuff but I would
really appreciate if someone could help a fellow engineer in getting his
patch accepted.
It would encourage more participation as well as help people wanting to use
Avro for circular references.

Regards
SG



On Thu, Apr 30, 2015 at 11:12 AM, S G sg.online.em...@gmail.com wrote:

 Hey Ryan,

 Any chances we could atleast get the circular references patch in?
 (https://issues.apache.org/jira/browse/AVRO-695)
 It's been reviewed extensively by Doug Cutting already and my patch for
 circular references in Hive Avro SerDe has also been accepted in Hive.
 It would be good to have the Avro side of patch also accepted so that the
 next versions of Avro and Hive can handle these without any issues.

 Thank you so much in advance for helping,
 Regards
 Sachin


 On Wed, Mar 25, 2015 at 9:42 AM, Ryan Blue b...@cloudera.com wrote:

 Hi S.G.,

 The UUID issue is next on my list to review and I'm also planning on
 looking at the schema resolution problem in detail this week. Thanks for
 being patient on these,

 rb


 On 03/23/2015 10:15 AM, S G wrote:

 Hi,

 I have submitted patches/pull-requests for a couple of issues.
 Can someone please review/commit these contributions?


 1) https://issues.apache.org/jira/browse/AVRO-695
 Cycle Reference Support


 2) https://issues.apache.org/jira/browse/AVRO-1569  (Bug fix)
 ReflectData.AllowNull fails with polymorphism while ReflectData works


 3) https://issues.apache.org/jira/browse/AVRO-1554
 UUID support in Avro


 4) https://issues.apache.org/jira/browse/AVRO-1568
 Allow Java polymorphism in Avro for third-party code


 5) https://issues.apache.org/jira/browse/AVRO-1562
 Add support for types extending Maps/Collections



 Thanks
 Sachin



 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Re: Working with github pull requests

[S G]

Thanks for this summary Ryan.
I have also found github to be more useful than submitting patch-files.

Hope to see more PRs in future.

sg

On Tue, Mar 31, 2015 at 2:22 PM, Ryan Blue b...@cloudera.com wrote:

 Hi everyone,

 There have been more github pull requests lately, partly because I like
 that for reviewing contributions. I like to be able to comment on specific
 lines and have a bit more detailed discussion.

 I just wanted to share a couple of good things to know:

 * The github JIRA comments have a link to a PR patch file, if you prefer
 patches.
 * The patch link gets the latest changes when the PR is updated.
 * To close a PR when you commit a change, just add a comment like Closes
 #18. Otherwise we can't close them.

 Lately I've been using github as an alternate remote so I can fetch all of
 the PRs at once and cherry pick into my git-svn branch. That works pretty
 well and removes the problem of remembering to add files that are in a
 patch but not added to git or svn.

 Hopefully that's helpful to others. Thanks,

 rb


 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Need review/merges for couple of pull requests

[S G]

Hi,

I have submitted patches/pull-requests for a couple of issues.
Can someone please review/commit these contributions?


1) https://issues.apache.org/jira/browse/AVRO-695
Cycle Reference Support


2) https://issues.apache.org/jira/browse/AVRO-1569  (Bug fix)
ReflectData.AllowNull fails with polymorphism while ReflectData works


3) https://issues.apache.org/jira/browse/AVRO-1554
UUID support in Avro


4) https://issues.apache.org/jira/browse/AVRO-1568
Allow Java polymorphism in Avro for third-party code


5) https://issues.apache.org/jira/browse/AVRO-1562
Add support for types extending Maps/Collections



Thanks
Sachin

Need review/merges for couple of pull requests

[S G]

Hi,

I have submitted patches/pull-requests for couple of issues.
Most of the review comments for these have also been fixed.
Can someone please do the final reviews and help me commit the same?


1) https://issues.apache.org/jira/browse/AVRO-1568
Allow Java polymorphism in Avro for third-party code


2) https://issues.apache.org/jira/browse/AVRO-1569
ReflectData.AllowNull fails with polymorphism while ReflectData works


3) https://issues.apache.org/jira/browse/AVRO-680
Allow non-string keys in maps


4) https://issues.apache.org/jira/browse/AVRO-695
Cycle Reference Support


5) https://issues.apache.org/jira/browse/AVRO-1554
UUID support in Avro


6) https://issues.apache.org/jira/browse/AVRO-1562
Add support for types extending Maps/Collections



Thanks
Sachin

Merging AVRO-695: Cyclical references

[S G]

Hi,

Can someone please review/merge the following patch?
https://issues.apache.org/jira/browse/AVRO-695

It has been reviewed and performance tested already and an issue similar to
it has been already accepted in Hive.

Thanks
Sachin

Re: Patches requiring review for submission

[S G]

Hey Ryan,

I would appreciate your help on getting these patches in.

Thanks
Sachin


On Wed, Feb 4, 2015 at 11:25 AM, S G sg.online.em...@gmail.com wrote:

 I have submitted a modified patch for #3 as well.

 Please help me review and get these committed.

 Thanks
 Sachin


 On Tue, Feb 3, 2015 at 12:42 PM, S G sg.online.em...@gmail.com wrote:

 Thanks a lot for the useful comments Ryan.

 1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)
 is just ready to be patched in.
 It has all the suggestions from Doug Cutting and an exhaustive testcase.
 Doug Cutting, if you can respond to this JIRA ticket, your help would be
 much appreciated.

 2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)
 is also completely ready for review.
 Please go ahead for the review.

 3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in
 Avro)
 Yes, I will remove the date-part and re-submit the patch.

 Thanks
 Sachin





 On Mon, Feb 2, 2015 at 3:36 PM, Ryan Blue b...@cloudera.com wrote:

 On 01/26/2015 11:03 AM, S G wrote:

 Hi,

 I have submitted a couple of patches that need some review.
 1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)


 This looks complicated and has a long history, so I'll try to help out
 on the other two.

  2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)


 Looks like the latest on this is some discussion about whether or not it
 would be useful. In my opinion, it would be a good addition. Other formats
 and processing engines don't have the restriction that keys are strings.
 For example, Parquet allows any object to be a key and Hive allows any
 primitive type.

 Is there anything else blocking this, or is the current patch ready for
 review?

  3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in
 Avro)


 This notes some updates that need to be done as of September last year
 (remove Date), but the last patch is from June. Do you want to update the
 patch?

 rb


 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Re: Patches requiring review for submission

[S G]

I have submitted a modified patch for #3 as well.

Please help me review and get these committed.

Thanks
Sachin


On Tue, Feb 3, 2015 at 12:42 PM, S G sg.online.em...@gmail.com wrote:

 Thanks a lot for the useful comments Ryan.

 1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)
 is just ready to be patched in.
 It has all the suggestions from Doug Cutting and an exhaustive testcase.
 Doug Cutting, if you can respond to this JIRA ticket, your help would be
 much appreciated.

 2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)
 is also completely ready for review.
 Please go ahead for the review.

 3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in
 Avro)
 Yes, I will remove the date-part and re-submit the patch.

 Thanks
 Sachin





 On Mon, Feb 2, 2015 at 3:36 PM, Ryan Blue b...@cloudera.com wrote:

 On 01/26/2015 11:03 AM, S G wrote:

 Hi,

 I have submitted a couple of patches that need some review.
 1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)


 This looks complicated and has a long history, so I'll try to help out on
 the other two.

  2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)


 Looks like the latest on this is some discussion about whether or not it
 would be useful. In my opinion, it would be a good addition. Other formats
 and processing engines don't have the restriction that keys are strings.
 For example, Parquet allows any object to be a key and Hive allows any
 primitive type.

 Is there anything else blocking this, or is the current patch ready for
 review?

  3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in
 Avro)


 This notes some updates that need to be done as of September last year
 (remove Date), but the last patch is from June. Do you want to update the
 patch?

 rb


 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Re: Patches requiring review for submission

[S G]

Thanks a lot for the useful comments Ryan.

1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references) is
just ready to be patched in.
It has all the suggestions from Doug Cutting and an exhaustive testcase.
Doug Cutting, if you can respond to this JIRA ticket, your help would be
much appreciated.

2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys) is
also completely ready for review.
Please go ahead for the review.

3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in Avro)
Yes, I will remove the date-part and re-submit the patch.

Thanks
Sachin





On Mon, Feb 2, 2015 at 3:36 PM, Ryan Blue b...@cloudera.com wrote:

 On 01/26/2015 11:03 AM, S G wrote:

 Hi,

 I have submitted a couple of patches that need some review.
 1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)


 This looks complicated and has a long history, so I'll try to help out on
 the other two.

  2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)


 Looks like the latest on this is some discussion about whether or not it
 would be useful. In my opinion, it would be a good addition. Other formats
 and processing engines don't have the restriction that keys are strings.
 For example, Parquet allows any object to be a key and Hive allows any
 primitive type.

 Is there anything else blocking this, or is the current patch ready for
 review?

  3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in
 Avro)


 This notes some updates that need to be done as of September last year
 (remove Date), but the last patch is from June. Do you want to update the
 patch?

 rb


 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

Patches requiring review for submission

[S G]

Hi,

I have submitted a couple of patches that need some review.
1) https://issues.apache.org/jira/browse/AVRO-695 (Cyclical references)
2) https://issues.apache.org/jira/browse/AVRO-680 (Non-string map keys)
3) https://issues.apache.org/jira/browse/AVRO-1554 (Supporting UUID in Avro)

Would really appreciate some help from the Avro community to get them
accepted.
We need to have them available in Avro to fully use Avro for our projects.

Thanks
Sachin

Need review on couple of patch submissions

[S G]

Hi,

I would appreciate if someone could review the following submissions:


1) https://issues.apache.org/jira/browse/AVRO-695 (Performance impact for
circular refs patch)

2) https://issues.apache.org/jira/browse/AVRO-680 (Performance impact for
non-string map-keys patch)

3) https://issues.apache.org/jira/browse/AVRO-1554 (Discussion required for
partial patch)

4) https://issues.apache.org/jira/browse/AVRO-1571 (Discussion required for
patch-approach)


Thanks
Sachin

Need reviews on couple of patch submissions

[S G]

Hi,

I would appreciate if someone could review the following submissions:


1) https://issues.apache.org/jira/browse/AVRO-695 (Performance impact for
circular refs patch)

2) https://issues.apache.org/jira/browse/AVRO-680 (Performance impact for
non-string map-keys patch)

3) https://issues.apache.org/jira/browse/AVRO-1554 (Discussion required for
partial patch)

4) https://issues.apache.org/jira/browse/AVRO-1571 (Discussion required for
patch-approach)


Thanks
Sachin

Patches ready for review

[S G]

Hi,

I have incorporated the changes suggested for the following issues:
1) AVRO-680
2) AVRO-695
3) AVRO-1562
Would appreciate if someone can review these.


Also, I would like to get some more views/thoughts on AVRO-1571 as well
before submitting a patch.

Thanks,
Sachin

Re: Supporting Polymorphic fields in Avro

[S G]

That would be a good solution.
I have created https://issues.apache.org/jira/browse/AVRO-1568 to discuss
more on this.


On Thu, Aug 21, 2014 at 11:51 AM, Doug Cutting cutt...@apache.org wrote:

 One way to handle polymorphism in Avro is with unions.  For example, if
 you're using reflect, you can annotate your abstract base class with the
 set of concrete classes that implement it, e.g.:

 {code}
 @Union(Concrete1.class, Concrete2.class) class AbstractBase { ... }
 {code}

 Then Avro reflect will use this union schema for all references to the
 AbstractBase type.

 Doug



 On Wed, Aug 20, 2014 at 4:40 PM, S G sg.online.em...@gmail.com wrote:

  Hi,
 
  We want to use Java polymorphic fields with Avro.
  Here is an example of the same.
 
  
  class Base
  {
 Integer a = 5;
  }
 
  class Derived extends Base
  {
  String b = Foo;
  }
 
  class PolymorphicDO
  {
 Base b = new Derived();
  }
  
 
 
  Jackson handles this kind of field by using annotations such as:
  @JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include =
  JsonTypeInfo.As.PROPERTY, property = @class”)
 
 
  If such a thing can be added to Avro, it can really handle a large number
  of Java designs without changes and same could be immediately useful in
  Hadoop using AvroSerde as well.
 
  I can work on a patch if the feature seems useful.
 
  Thanks
  Sachin
 

Supporting Polymorphic fields in Avro

[S G]

Hi,

We want to use Java polymorphic fields with Avro.
Here is an example of the same.


class Base
{
   Integer a = 5;
}

class Derived extends Base
{
String b = Foo;
}

class PolymorphicDO
{
   Base b = new Derived();
}



Jackson handles this kind of field by using annotations such as:
@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include =
JsonTypeInfo.As.PROPERTY, property = @class”)


If such a thing can be added to Avro, it can really handle a large number
of Java designs without changes and same could be immediately useful in
Hadoop using AvroSerde as well.

I can work on a patch if the feature seems useful.

Thanks
Sachin

Re: Release Avro 1.8.0 soon?

[S G]

I have put up a question for AVRO-680 (non-string map keys) and
AVRO-1554 (support
for UUID and Date-types).
Will appreciate if someone could answer those.

Thanks
Sachin


On Tue, Aug 5, 2014 at 7:13 AM, Christophe Taton ta...@wibidata.com wrote:

 I'd be happy to see AVRO-1334 and AVRO-1550 checked in.

 C.


 On Tue, Aug 5, 2014 at 6:40 AM, Willem van Bergen wil...@vanbergen.org
 wrote:

  We also previously discussed dropping support for Ruby 1.8, as it is EOL.
  I created a patch that removes some of the hacks needed to support string
  encodings in Ruby 1.8 and 1.9+ simultaneously:
  https://issues.apache.org/jira/browse/AVRO-1559
 
 
  Willem
 
  On Aug 4, 2014, at 6:31 PM, Doug Cutting cutt...@apache.org wrote:
 
   This introduces a minor incompatibility, so needs to go into 1.8.0, not
  1.7.8.
  
   Perhaps we should identify other changes that also introduce minor
   incompatibilities and push out a 1.8.0 release soon?  I don't want to
   open the gates to major incompatibilities, but a few whose
   incompatibilities that are high value and are relatively easy to
   diagnose might be reasonable.
  
   Other obvious candidates might be:
- AVRO-1334 (update java dependencies)
- AVRO-1550 (update protobuf dependency)
- AVRO-1514 (update perl dependencies)
  
   What do others think?
  
   Doug
  
   On Mon, Aug 4, 2014 at 2:50 PM, Sean Busbey (JIRA) j...@apache.org
  wrote:
  
   [
 
 https://issues.apache.org/jira/browse/AVRO-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
  ]
  
   Sean Busbey updated AVRO-997:
   -
  
  Status: Patch Available  (was: In Progress)
  
   Union of enum and null cannot be serialized
   ---
  
  Key: AVRO-997
  URL: https://issues.apache.org/jira/browse/AVRO-997
  Project: Avro
   Issue Type: Bug
 Affects Versions: 1.5.1
 Reporter: Aaron Kimball
 Assignee: Sean Busbey
  Fix For: 1.8.0
  
  Attachments: AVRO-997.patch, AVRO-997.patch, AVRO-997.patch,
  AVRO-997.permissive-generic-api.patch
  
  
   I have a schema like:
   {code}
   [
   {
type: enum,
name: Gender,
symbols: [M, F]
   },
   {
type : record,
name : Foo,
fields : [
  { type : [Gender, null], name : gender },
  ...
]
   }
   ]
   {code}
   I build a record like {{Foo foo = new Foo(); foo.gender = Gender.M;}}
   When I go to serialize this, I get:
   {code}Not in union
  [{type:enum,name:Gender,symbols:[M,F]},null]: M
at
  org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:482)
at
 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:70)
at
 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
   {code}
  
  
  
   --
   This message was sent by Atlassian JIRA
   (v6.2#6252)
 
 

Why are java.* classes excluded from Avro serialization?

[S G]

Hi,

I see that all the java classes in java.util package are completely ignored
by Avro.
This includes some of the very commonly used classes like:
1) java.util.Date,
2) java.util.UUID,
3) java.util.Currency and
4) java.util.Locale etc.

The code that does that is in ReflectData#getFields(Class? recordClass,
boolean excludeJava)
It is called from ReflectData#getCachedFields(Class? recordClass) as
getFields(recordClass, true)


Thinking aloud, would it be a good idea to have a white-list of classes
(like those mentioned above) such that the internals of those classes would
be always serialized, thus adding support for these commonly-used types in
Avro?
Another solution could be to have a custom-encoding for all such classes.

Please suggest.
Thanks
Sachin

Review request for non-string map-keys

[S G]

Hi,

Could someone please review the patch non_string_map_keys4.patch for
AVRO-680?
(https://issues.apache.org/jira/browse/AVRO-680)

There have already been some reviews and I think its very close to being
frozen.
The patch is complete with unit-tests and I also added one for JsonEncoder.

Thanks
Sachin

Review for AVRO-680 patch

[S G]

Hi,

Could someone please review the patch for AVRO-680?
(https://issues.apache.org/jira/browse/AVRO-680)

I have incorporated all the suggestions and also tested it extensively.
The patch is complete with unit-tests.

Thanks
Sachin

Circular references and non-string map-keys patch

[S G]

Hi Avro Developers,

I have submitted a patch for Circular references and non-string map-keys
support in Avro at
https://issues.apache.org/jira/browse/AVRO-695

Can someone guide me how I can get it reviewed and commit this patch?

Thanks for helping me out,
Sachin