Always getting a value...

[Niels Basjes]

Hi,

I have a Java project where I'm using Avro as the serialization technology.
I have a deep nested structure and many fields are optional.

Because of the 'random' nature of my application I want to be able to
simply 'set' a value in that tree easily.

So I want to have a setter in my class that looks like this (Assume
myStruct is an instance of the 'root' class generated by Avro):

public void setSomething(String value) {
myStruct
.getFoo()
.getBar()
.getOne()
.getOther()
.setSomething(value);
}

The 'problem' I ran into is that any of the 4 get methods can return a null
value so the code I have to write is really huge.
For every step in this method I have to build null checks and create the
underlying instance if it is null.
I already started writing helper methods to do this for parts of my tree.

To solve this in a way that makes this code readable I came up with the
following which I want to propose to you guys (before I start working on a
patch).

My idea is to generate a new 'get' method in addition to the existing
normal get method for the regular instance of the class.

So in addition to the

public void getFoo() {
return foo;
}

I propose to generate something like this as well

public void getAlwaysFoo() {
if (foo == null) {
setFoo(Foo.newBuilder().build());
}
return foo;
}

This way the automatically created instance immediately has all the
defaults I have defined.

Assuming this naming my code will be readable because it will look like
this:
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}

I can create this same effect in my own project code.
But having this automatically generated by the Avro engine would make life
a lot easier.

Is this an idea you guys would consider to have in the main (Java)
generated code base?
If so, is the 'getAlways' prefix sensible?

-- 
Best regards

Niels Basjes

Re: Always getting a value...

[Niels Basjes]

Oops,
That should be
 public Foo getFoo()
and
 public Foo getAlwaysFoo()
ofcourse

On Wed, Nov 26, 2014 at 6:28 PM, Niels Basjes ni...@basjes.nl wrote:

 Hi,

 I have a Java project where I'm using Avro as the serialization technology.
 I have a deep nested structure and many fields are optional.

 Because of the 'random' nature of my application I want to be able to
 simply 'set' a value in that tree easily.

 So I want to have a setter in my class that looks like this (Assume
 myStruct is an instance of the 'root' class generated by Avro):

 public void setSomething(String value) {
 myStruct
 .getFoo()
 .getBar()
 .getOne()
 .getOther()
 .setSomething(value);
 }

 The 'problem' I ran into is that any of the 4 get methods can return a
 null value so the code I have to write is really huge.
 For every step in this method I have to build null checks and create the
 underlying instance if it is null.
 I already started writing helper methods to do this for parts of my tree.

 To solve this in a way that makes this code readable I came up with the
 following which I want to propose to you guys (before I start working on a
 patch).

 My idea is to generate a new 'get' method in addition to the existing
 normal get method for the regular instance of the class.

 So in addition to the

 public void getFoo() {
 return foo;
 }

 I propose to generate something like this as well

 public void getAlwaysFoo() {
 if (foo == null) {
 setFoo(Foo.newBuilder().build());
 }
 return foo;
 }

 This way the automatically created instance immediately has all the
 defaults I have defined.

 Assuming this naming my code will be readable because it will look like
 this:
 public void setSomething(String value) {
 myStruct
 .getAlwaysFoo()
 .getAlwaysBar()
 .getAlwaysOne()
 .getAlwaysOther()
 .setSomething(value);
 }

 I can create this same effect in my own project code.
 But having this automatically generated by the Avro engine would make life
 a lot easier.

 Is this an idea you guys would consider to have in the main (Java)
 generated code base?
 If so, is the 'getAlways' prefix sensible?

 --
 Best regards

 Niels Basjes




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Always getting a value...

[Niels Basjes]

Yes, the builders would do that for mandatory (= non-null) fields.

The schemas I'm using have a lot of this pattern

record Thing {
  union { null, Foo } optionalFoo;

  union { null, Bar } optionalBar;

}

record Bar {
  union { null, Blabla } optionalBlabla;
}

etc.

So creating a record with a simple Thing.newBuilder().build() would still
yield null values for both optionalFoo and optionalBar.



On Wed, Nov 26, 2014 at 8:58 PM, Sean Busbey bus...@cloudera.com wrote:

 What does an example schema look like in this case?

 Shouldn't the extant builders give you this behavior if you've made the
 default for the optional fields non-null?

 --
 Sean
 On Nov 26, 2014 11:34 AM, Niels Basjes ni...@basjes.nl wrote:

  Oops,
  That should be
   public Foo getFoo()
  and
   public Foo getAlwaysFoo()
  ofcourse
 
  On Wed, Nov 26, 2014 at 6:28 PM, Niels Basjes ni...@basjes.nl wrote:
 
   Hi,
  
   I have a Java project where I'm using Avro as the serialization
  technology.
   I have a deep nested structure and many fields are optional.
  
   Because of the 'random' nature of my application I want to be able to
   simply 'set' a value in that tree easily.
  
   So I want to have a setter in my class that looks like this (Assume
   myStruct is an instance of the 'root' class generated by Avro):
  
   public void setSomething(String value) {
   myStruct
   .getFoo()
   .getBar()
   .getOne()
   .getOther()
   .setSomething(value);
   }
  
   The 'problem' I ran into is that any of the 4 get methods can return a
   null value so the code I have to write is really huge.
   For every step in this method I have to build null checks and create
 the
   underlying instance if it is null.
   I already started writing helper methods to do this for parts of my
 tree.
  
   To solve this in a way that makes this code readable I came up with the
   following which I want to propose to you guys (before I start working
 on
  a
   patch).
  
   My idea is to generate a new 'get' method in addition to the existing
   normal get method for the regular instance of the class.
  
   So in addition to the
  
   public void getFoo() {
   return foo;
   }
  
   I propose to generate something like this as well
  
   public void getAlwaysFoo() {
   if (foo == null) {
   setFoo(Foo.newBuilder().build());
   }
   return foo;
   }
  
   This way the automatically created instance immediately has all the
   defaults I have defined.
  
   Assuming this naming my code will be readable because it will look like
   this:
   public void setSomething(String value) {
   myStruct
   .getAlwaysFoo()
   .getAlwaysBar()
   .getAlwaysOne()
   .getAlwaysOther()
   .setSomething(value);
   }
  
   I can create this same effect in my own project code.
   But having this automatically generated by the Avro engine would make
  life
   a lot easier.
  
   Is this an idea you guys would consider to have in the main (Java)
   generated code base?
   If so, is the 'getAlways' prefix sensible?
  
   --
   Best regards
  
   Niels Basjes
  
 
 
 
  --
  Best regards / Met vriendelijke groeten,
 
  Niels Basjes
 




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Always getting a value...

[Niels Basjes]

Hi,

To see how it would work I simply wrote the patch and played in my
environment with the effects on the generated Java code.
[ I created a Jira issue and attached this draft patch to it:
https://issues.apache.org/jira/browse/AVRO-1614 ]

The idea works but I also found that for my usecase it is not very pleasant
to work with.

Assume this example again:

public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}

The main problem is that in order to do .getAlwaysOne() I MUST define ALL
fields of that type with a default value.
What I don;t like about that is that I want the schema definition to
enforce the fact that some fields are mandatory.
By adding a default value to 'everything' I lose that capability of AVRO
... which I don't want.

At this point in time the only workaround this (for me major) issue is by
introducing something where I can do something like having a 'tree of
incomplete Builders' and when I say 'build()' to the top one it will build
the entire tree.

Any thoughts?
Do you guy think there is a need/usecase for this idea (so I leave the
issue open) or shall I simply close AVRO-1614 with a 'Won't fix'?

Niels Basjes



On Thu, Nov 27, 2014 at 1:37 AM, Doug Cutting cutt...@apache.org wrote:

 On Wed, Nov 26, 2014 at 2:33 PM, svante karlsson s...@csi.se wrote:
  I'm not sure that works for avro where null is used for an optional
 field.

 It should work.  If it doesn't, it's a bug.  For example:

 record Foo {
   union {string, null} name = default;
 }

 record Bar {
   union {Foo, null} foo = {name = non-default};
 }

 Default values in IDL are JSON format.

 Doug




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Always getting a value...

[Niels Basjes]

I uploaded a patch for AVRO-1614 and AVRO-1616 (just an update to the
.gitignore).

It seems that there is no automatic pickup by Jenkins configured (like with
Hadoop and HBase).

How do I proceed in getting your feedback on these patches and getting them
reviewed and if possible comitted?

Thanks.

Niels Basjes

On Mon, Dec 1, 2014 at 6:05 PM, Niels Basjes ni...@basjes.nl wrote:

 Hi,

 I realized over the weekend that the Builder instances do support an
 'incomplete' schema and this is validated at the time the build() is called.
 Based upon that I redid the patch today so that in a Builder in addition
 to the actual value of a field there is now also a Builder field for that
 field possible.
 If that is used then you can have the incomplete form of the sub-schema in
 a Builder.
 So for any Builder instance there is a getFooBuilder() that either returns
 the existing or creates a new Builder instance for the Foo field if such a
 builder is supported.

 As a consequence:
 - schema validation is postponed until the actual build() is called.
 - for the fields where this Builder is used the actual build() call
 becomes recursive.

 So in my testing code I can now do this:


 *Measurement.Builder measurementBuilder = Measurement.newBuilder();*

 *measurementBuilder*
 *.getTransportBuilder()*
 *  .getConnectionBuilder()*
 *.getNetworkConnectionBuilder()*
 *  .setNetworkAddress(127.0.0.1)*
 *  .setNetworkType(NetworkType.IPv4);*

 *Measurement measurement = measurementBuilder.build();*

 Open question: I have not seen unit tests that validate the generated Java
 code yet. How to approach this?

 Niels Basjes


 On Thu, Nov 27, 2014 at 5:37 PM, Niels Basjes ni...@basjes.nl wrote:

 Hi,

 To see how it would work I simply wrote the patch and played in my
 environment with the effects on the generated Java code.
 [ I created a Jira issue and attached this draft patch to it:
 https://issues.apache.org/jira/browse/AVRO-1614 ]

 The idea works but I also found that for my usecase it is not very
 pleasant to work with.

 Assume this example again:

 public void setSomething(String value) {
 myStruct
 .getAlwaysFoo()
 .getAlwaysBar()
 .getAlwaysOne()
 .getAlwaysOther()
 .setSomething(value);
 }

 The main problem is that in order to do .getAlwaysOne() I MUST define
 ALL fields of that type with a default value.
 What I don;t like about that is that I want the schema definition to
 enforce the fact that some fields are mandatory.
 By adding a default value to 'everything' I lose that capability of AVRO
 ... which I don't want.

 At this point in time the only workaround this (for me major) issue is by
 introducing something where I can do something like having a 'tree of
 incomplete Builders' and when I say 'build()' to the top one it will build
 the entire tree.

 Any thoughts?
 Do you guy think there is a need/usecase for this idea (so I leave the
 issue open) or shall I simply close AVRO-1614 with a 'Won't fix'?

 Niels Basjes



 On Thu, Nov 27, 2014 at 1:37 AM, Doug Cutting cutt...@apache.org wrote:

 On Wed, Nov 26, 2014 at 2:33 PM, svante karlsson s...@csi.se wrote:
  I'm not sure that works for avro where null is used for an optional
 field.

 It should work.  If it doesn't, it's a bug.  For example:

 record Foo {
   union {string, null} name = default;
 }

 record Bar {
   union {Foo, null} foo = {name = non-default};
 }

 Default values in IDL are JSON format.

 Doug




 --
 Best regards / Met vriendelijke groeten,

 Niels Basjes




 --
 Best regards / Met vriendelijke groeten,

 Niels Basjes




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

What are the next steps?

[Niels Basjes]

Hi,

I filed a few issues in JIRA to improve AVRO and provided patches to the
best of how I understand you want them.


*Java: adding getSomethingBuilder() to the generated Builder code*
*https://issues.apache.org/jira/browse/AVRO-1614
https://issues.apache.org/jira/browse/AVRO-1614 *


*Minor fix in the .gitignore*

*https://issues.apache.org/jira/browse/AVRO-1616
https://issues.apache.org/jira/browse/AVRO-1616*


*Java: Generate better JavaDoc*

*https://issues.apache.org/jira/browse/AVRO-1619
https://issues.apache.org/jira/browse/AVRO-1619*

On the first one ( AVRO-1614 ) I already got some great feedback from Doug
Cutting which I already implemented.

So what is the next step in the process?

What do I need to do to get this reviewed and committed ?
(or rejected if you guys really don't want the change I proposed)

-- 
Best regards

Niels Basjes

Re: What are the next steps?

[Niels Basjes]

Thanks for explaining.
About a day after I sent this email I saw a news item that you were in
somewhere in Asia.
So I already understood that I simply had to be a bit more patient.

Niels

On Mon, Dec 15, 2014 at 8:30 PM, Doug Cutting cutt...@apache.org wrote:

 You're doing everything right, including sending this friendly reminder
 message.

 The Avro community tries to review patches promptly and commit them
 when ready but we sometimes fall behind and/or lose track of things.
 I was travelling last week and didn't have much time to look at Avro.
 This week I hope to catch up.  I'll try to take a look at these issues
 soon.

 We welcome reviews from non-committers and tend to readily promote
 folks to committer status once they've made more than a handful of
 contributions so they can then help reduce the backlog.

 Cheers,

 Doug

 On Thu, Dec 11, 2014 at 8:50 AM, Niels Basjes ni...@basjes.nl wrote:
  Hi,
 
  I filed a few issues in JIRA to improve AVRO and provided patches to the
  best of how I understand you want them.
 
 
  *Java: adding getSomethingBuilder() to the generated Builder code*
  *https://issues.apache.org/jira/browse/AVRO-1614
  https://issues.apache.org/jira/browse/AVRO-1614 *
 
 
  *Minor fix in the .gitignore*
 
  *https://issues.apache.org/jira/browse/AVRO-1616
  https://issues.apache.org/jira/browse/AVRO-1616*
 
 
  *Java: Generate better JavaDoc*
 
  *https://issues.apache.org/jira/browse/AVRO-1619
  https://issues.apache.org/jira/browse/AVRO-1619*
 
  On the first one ( AVRO-1614 ) I already got some great feedback from
 Doug
  Cutting which I already implemented.
 
  So what is the next step in the process?
 
  What do I need to do to get this reviewed and committed ?
  (or rejected if you guys really don't want the change I proposed)
 
  --
  Best regards
 
  Niels Basjes



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Contributing to C# implementation

[Niels Basjes]

Hi,

I would simply make it a New Feature as it is a completely new language
and leave the priority to the default value.

Niels Basjes

On Wed, Dec 24, 2014 at 6:17 PM, Alexey Orlov alex...@microsoft.com wrote:

 Hello again to everybody.

 To save reading the entire thread here is a quick summary:

 I'm a Microsoft employee. Inside Microsoft we have created a C#
 implementation of Avro which we believe could be a valuable contribution to
 the project. The library is currently available at
 https://hadoopsdk.codeplex.com/wikipage?title=Avro%20Library

 Doug below suggested opening an issue in Jira to start the discussion on
 this. Now I need to confess that I'm quite new to the ASF Jira setup, so I
 would like to open it in the right way.

 Thus two questions:

 1) Issue type for such discussion? (Improvement/New feature/Wish)
 2) Suggested priority?

 And Merry Christmas to all!

 Thank you,
 Alexey

 -Original Message-
 From: Alexey Orlov [mailto:alex...@microsoft.com]
 Sent: Monday, June 2, 2014 2:25 PM
 To: dev@avro.apache.org
 Subject: RE: Contributing to C# implementation

 Doug, thank you, I will make the final check with my legal people and open
 the case.

 Now about dependency on Windows. The main purpose of the library was to
 enable people with .NET applications to effectively use Avro. Thus it was
 natural that we were relying on Microsoft implementation of .NET Framework
 while creating the library. However we did NOT utilize anything exotic,
 that would prevent the library to be compiled vs, e.g., Mono. Note though
 we never tested such cases (see the main motivation of creating the
 library).

 If anybody from the community would like to try this out, the complete
 code is available (see my initial post).

 Thank you,
 Alexey

 -Original Message-
 From: Doug Cutting [mailto:cutt...@apache.org]
 Sent: Wednesday, May 28, 2014 7:51 PM
 To: dev@avro.apache.org
 Subject: Re: Contributing to C# implementation

 Alexey,

 This would be a great contribution to Apache Avro.  I suggest opening an
 issue in Jira to discuss this.

 The Microsoft implementation of Avro appears only to build on Windows,
 unlike other implementations which mostly build on multiple platforms.
  Is the dependency on Windows deep, or is it only in the build?

 Thanks,

 Doug

 On Wed, May 28, 2014 at 3:10 AM, Alexey Orlov alex...@microsoft.com
 wrote:
  Hello,
 
  I am a Program Manager working in Microsoft and responsible for C#
  implementation of Avro, that we have recently made public as Microsoft
  Avro Library. The code, binaries and docs are available here:
  https://hadoopsdk.codeplex.com/wikipage?title=Avro%20Libraryreferring
  Title=Home
 
  Now we would like to start a conversation related to contributing the
 library to the project. Would be thankful for help.
 
  Thank you,
  Alexey




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Eclipse not building ?

[Niels Basjes]

You must make sure you have Maven support in your Eclipse.
The pom.xml file contains all Java dependencies for the project.
The error you showed seems to indicate that these dependencies have not
been imported.

Niels

On Thu, Jan 22, 2015 at 11:08 PM, Matan Shukry matanshu...@gmail.com
wrote:

 Hi, thought i'll try and contribute some code to avro.
 I checked out the code

 svn checkout https://svn.apache.org/repos/asf/avro/trunk/ avro-trunk


 then added the code to eclipse. However, I am getting compile errors in
 eclipse, seems as if dependencies aren't working. an example error:

 The import org.apache.maven cannot be resolved

 if it matters, in file

 /avro-trunk/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/AbstractAvroMojo.java

 What am I missing here?




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: New Avro committer - Ryan Blue

[Niels Basjes]

Congrats!

On Tue, Jan 6, 2015 at 11:10 AM, Tom White t...@cloudera.com wrote:

 On behalf of the Apache Avro PMC I'm pleased to announce that Ryan
 Blue has been elected as a committer on the Avro project.

 Welcome Ryan!

 Cheers,
 Tom




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Next Avro release

[Niels Basjes]

Hi,

I would say go for the 1.8.0

Note that there are a few issues that seem appropriate to include for
this one because they relate to a change in 'backwards compatibility:

- AVRO-1586 Build against Hadoop 2
  ( which should also fix AVRO-1453 Release version of avro-tools
compiled against hadoop2 )
- AVRO-1559 Drop support for Ruby 1.8

On a personal note I would like a 'Yes, commit' / 'No, won't fix'
choice from you guys regarding this proposal (Patch included):
   AVRO-1633 Add additional setXxx(Builder) method to make user code
more readable.

Niels Basjes

On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com wrote:
 Hi everyone,

 It would be good to do another Avro release soon. I'm happy to create
 a release candidate so we can vote on it.

 What do folks think about releasing 1.7.8 vs. 1.8.0? There are very
 few changes in the 1.7 branch, so it might be best to release 1.8.0
 from trunk. There have been enough changes to justify a new minor
 release I think. Are there any drawbacks to doing that?

 The unresolved 1.8.0 issues are here:

 http://s.apache.org/Cdt

 And the unresolved 1.7.8 issues are here:

 http://s.apache.org/5ShO

 Thanks,
 Tom



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Encrypting only some of the fields.

[Niels Basjes]

Hi,

I'm working on a project where I plan to put clickstream data into Kafka
serialized using AVRO. In a later stage I want these records persisted into
AVRO files so they can be used by people using PIG.

So far this is no problem at all.

Now some of those fields (not all) are privacy sensitive so I do not want
them to be 'plain text' in the data. I want them to be encrypted so that
they can only be read by the people who need access to these fields.

The only thing I have found so far about encrypting data in AVRO is
https://issues.apache.org/jira/browse/AVRO-1371 which states
Quote
*  Similar to compression and decompression, encryption and decryption *

*  can be implemented with Codecs, a concept that already exists in
Avro.*

I had a look at that Codecs API and it simply takes the 'entire thing' as a
ByteBuffer and compresses it. So this means the entire record is encrypted
(which is not what I want).

I want without storing the data twice (it is too big for that):
- All consumers to be able to read 'most' fields.
- Some consumers to be able to read 'all' fields.

I was contemplating to simply put the keyid and the encrypted bytes into a
field of the type 'bytes'. That way there is no need to change the
underlying file format.

To keep it simple I would simply have the application code generate the
'encrypted value' and store it in the record. Then at the PIG side I would
simply create a UDF that does the decryption again.

To make using this easier I even thought about extending the IDL language
(keyword 'encrypted') and then generate extra/different utility methods
that wrap/encrypt that field via the setters/builders and put that in a
normal AVRO file as bytes.

But before I start coding;
Has anyone ever thought about what the 'right' approach is to do this in
AVRO?
Has anyone build something I can have a look at?

-- 
Best regards

Niels Basjes

Re: Next Avro release

[Niels Basjes]

I did a similar fight with forrest a few weeks ago trying to get pig to
build in docker.
See https://issues.apache.org/jira/browse/PIG-4526 for the current draft of
that.
In this case it turned out to be a case where Forrest 0.8 would fail with
strange errors when ran under a new jvm.
Where new is java 6 or newer...

Niels

On Sat, 18 Jul 2015 11:14 Tom White t...@cloudera.com wrote:

 Quick update: after spending way too much time fighting Forrest I've
 managed to get the full build working on Docker to the point where all
 tests pass and all artifacts are created and signed. (If you're
 interested you can try it out by typing './build.sh docker', then
 './build.sh clean dist' in the Docker container.)

 I had hoped to roll a release this week, and I'm travelling for the
 next two weeks so it will probably have to wait until I'm back.

 Cheers,
 Tom

 On Thu, Jul 9, 2015 at 4:13 PM, Tom White t...@cloudera.com wrote:
  I committed the Ruby fixes that Sean reviewed, and I've looked at
  Marcin's patch. Is anyone familiar with the C implementation able to
  take a look at Thomas' patch?
 
  Thanks,
  Tom
 
  On Thu, Jul 9, 2015 at 9:30 AM, Thomas Sanchez thomas.san...@gmail.com
 wrote:
  Hi,
  Sorry to bust in, but if one could take a final look at AVRO-1663, I'd
  appreciate.
  It is a minor fix.
 
  Thanks,
 
  2015-07-07 17:16 GMT+02:00 Sean Busbey bus...@cloudera.com:
  I have done non-binding reviews on the following, if a committer could
 take
  look:
 
  * AVRO-1645
  * AVRO-1693
 
  On Tue, Jul 7, 2015 at 8:41 AM, Tom White t...@cloudera.com wrote:
 
  Committers - please take a look at the outstanding patch available
  JIRAs for 1.7.8/1.8.0 and review and commit any that you'd like to see
  in 1.8.0:
 
  http://s.apache.org/xXz
 
  Thanks!
  Tom
 
  On Tue, Jul 7, 2015 at 2:34 PM, Tom White t...@cloudera.com wrote:
   Thanks Niels. Comments inline.
  
   On Wed, Jul 1, 2015 at 3:50 PM, Niels Basjes ni...@basjes.nl
 wrote:
   Hi,
  
   I would say go for the 1.8.0
  
   Note that there are a few issues that seem appropriate to include
 for
   this one because they relate to a change in 'backwards
 compatibility:
  
   - AVRO-1586 Build against Hadoop 2
 ( which should also fix AVRO-1453 Release version of avro-tools
   compiled against hadoop2 )
   - AVRO-1559 Drop support for Ruby 1.8
  
   I've committed both of these now.
  
  
   On a personal note I would like a 'Yes, commit' / 'No, won't fix'
   choice from you guys regarding this proposal (Patch included):
  AVRO-1633 Add additional setXxx(Builder) method to make user
 code
   more readable.
  
   I think this is a won't fix as it doesn't seem to produce any real
   increase in readability and could actually be confusing.
  
   Thanks,
   Tom
  
  
   Niels Basjes
  
   On Wed, Jul 1, 2015 at 3:25 PM, Tom White t...@cloudera.com
 wrote:
   Hi everyone,
  
   It would be good to do another Avro release soon. I'm happy to
 create
   a release candidate so we can vote on it.
  
   What do folks think about releasing 1.7.8 vs. 1.8.0? There are
 very
   few changes in the 1.7 branch, so it might be best to release
 1.8.0
   from trunk. There have been enough changes to justify a new minor
   release I think. Are there any drawbacks to doing that?
  
   The unresolved 1.8.0 issues are here:
  
   http://s.apache.org/Cdt
  
   And the unresolved 1.7.8 issues are here:
  
   http://s.apache.org/5ShO
  
   Thanks,
   Tom
  
  
  
   --
   Best regards / Met vriendelijke groeten,
  
   Niels Basjes
 
 
 
 
  --
  Sean
 
 
 
  --
  Thomas Sanchez

Re: [VOTE] Avro release 1.8.0 (rc1)

[Niels Basjes]

+1
* Set the Maven staging as a repo on my system.
* Updated several of my projects that rely heavily on AVRO to use the 1.8.0
version.
* Rebuild and ran all the unit tests on those projects; all passed without
incident



On Wed, Dec 16, 2015 at 6:55 PM, Tom White <t...@cloudera.com> wrote:

> I have created a new candidate build for Avro release 1.8.0 following the
> rc0 vote in August that didn't pass due to licensing/notice issues. (Thanks
> to Ryan and Sean for fixing them!)
>
> The changes are listed at:
> http://s.apache.org/avro180
>
> The release artifacts can be found here:
> *https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc1/
> <https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc1/>*
>
> The tag corresponding to this release candidate is
> *http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc1/
> <http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc1/>*
>
> You can find the KEYS file here:
> https://dist.apache.org/repos/dist/release/avro/KEYS
>
> The Maven staging repository is at:
> *https://repository.apache.org/content/repositories/orgapacheavro-1003
> <https://repository.apache.org/content/repositories/orgapacheavro-1003>*
>
> Please download, verify, and test. This is the first release that has been
> built using Docker, so please pay extra attention to the languages you are
> interested in. Thanks in advance for voting!
>
> Cheers,
> Tom
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Migrate to Java 7

[Niels Basjes]

+1
On 18 Dec 2015 12:33, "Tom White"  wrote:

> +1
>
> Tom
>
> On Tue, Dec 15, 2015 at 1:08 AM, Ryan Blue  wrote:
>
> > I just noticed that our tests are still compiling and running with Java
> 6.
> > Java 7 is already end-of-life (public patches at least), so I think it is
> > reasonable to start migrating. Is everyone okay with updating the builds
> > and dropping support for Java 6?
> >
> > rb
> >
> > --
> > Ryan Blue
> >
>

[DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Niels Basjes]

Hi,

I'm working on a project where I'm putting Avro records into Kafka and at
the other end pull them out again.
For that purpose I wrote two methods 'toBytes' and 'fromBytes' in a
separate class (see below).

I see this as the type of problem many developers run into.
Would it be a good idea to generate methods like these into the generated
Java code?

This would make it possible to serialize and deserialize singles records
like this:

byte [] someBytes = measurement.toBytes();
Measurement m = Measurement.fromBytes(someBytes);

Niels Basjes

P.S. possibly not name it toBytes but getBytes (similar to what the String
class has)

public final class MeasurementSerializer {
private MeasurementSerializer() {
}

public static Measurement fromBytes(byte[] bytes) throws IOException {
try {
DatumReader reader = new
SpecificDatumReader<>(Measurement.getClassSchema());
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
return reader.read(null, decoder);
} catch (RuntimeException rex) {
throw new IOException(rex.getMessage());
}
}

public static byte[] toBytes(Measurement measurement) throws IOException {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
SpecificDatumWriter writer = new
SpecificDatumWriter<>(Measurement.class);
writer.write(measurement, encoder);
encoder.flush();
out.close();
return out.toByteArray();
} catch (RuntimeException rex) {
throw new IOException(rex.getMessage());
}
}
}



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Avro release 1.8.0 (rc1)

[Niels Basjes]

@Martin: Isn't there a unit test in the ruby part that validates this?

On Sat, Dec 19, 2015 at 7:53 PM, Martin Kleppmann <mar...@kleppmann.com>
wrote:

> There is a problem with the Ruby gem artifact. I tried installing it in a
> Ruby project using Avro; when the gem is loaded, it fails with the error:
>
> kernel_require.rb:54:in `require': cannot load such file --
> avro/schema_normalization (LoadError)
>
> It seems that the following two files, which are present in the source,
> are missing from the gem file:
>
> lang/ruby/lib/avro/schema_normalization.rb
> lang/ruby/test/case_finder.rb
>
> I think those file paths need to be added to lang/ruby/Manifest so that
> they get included in the gem build process.
>
> Martin
>
> > On 18 Dec 2015, at 11:36, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > +1
> > * Set the Maven staging as a repo on my system.
> > * Updated several of my projects that rely heavily on AVRO to use the
> 1.8.0
> > version.
> > * Rebuild and ran all the unit tests on those projects; all passed
> without
> > incident
> >
> >
> >
> > On Wed, Dec 16, 2015 at 6:55 PM, Tom White <t...@cloudera.com> wrote:
> >
> >> I have created a new candidate build for Avro release 1.8.0 following
> the
> >> rc0 vote in August that didn't pass due to licensing/notice issues.
> (Thanks
> >> to Ryan and Sean for fixing them!)
> >>
> >> The changes are listed at:
> >> http://s.apache.org/avro180
> >>
> >> The release artifacts can be found here:
> >> *https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc1/
> >> <https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc1/>*
> >>
> >> The tag corresponding to this release candidate is
> >> *http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc1/
> >> <http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc1/>*
> >>
> >> You can find the KEYS file here:
> >> https://dist.apache.org/repos/dist/release/avro/KEYS
> >>
> >> The Maven staging repository is at:
> >> *https://repository.apache.org/content/repositories/orgapacheavro-1003
> >> <https://repository.apache.org/content/repositories/orgapacheavro-1003
> >*
> >>
> >> Please download, verify, and test. This is the first release that has
> been
> >> built using Docker, so please pay extra attention to the languages you
> are
> >> interested in. Thanks in advance for voting!
> >>
> >> Cheers,
> >> Tom
> >>
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Niels Basjes]

Thanks for pointing this out.
This is exactly what I was working on.

The way I solved the 'does the schema match' question at work is by
requiring that all schema's start with a single text field "schema
classname" being the full class name of the class that was used to generate
it.
That way we can have newer versions of the schema and still be able to
unpack them. In this form the classname is essentially an indicator if
schema migration is possible; even though the schemas are different.

What do you think of this direction?

Niels


On Mon, Dec 21, 2015 at 11:30 PM, Ryan Blue <b...@cloudera.com> wrote:

> Niels,
>
> This sounds like a good idea to me to have methods like this. I've had to
> write those methods several times!
>
> The idea is also related to AVRO-1704 [1], which is a suggestion to
> standardize the encoding that is used for single records. Some projects
> have been embedding the schema fingerprint at the start of each record, for
> example, which would be a helpful thing to do.
>
> It may also be a good idea to create a helper object rather than attaching
> new methods to the datum classes themselves. In your example below, you
> have to create a new encoder or decoder for each method call. We could
> instead keep a backing buffer and encoder/decoder on a class that the
> caller instantiates so that they can be reused. At the same time, that
> would make it possible to reuse the class with any data model and manage
> the available schemas (if embedding the fingerprint).
>
> I'm thinking something like this:
>
>   ReflectClass datum = new ReflectClass();
>   ReflectData model = ReflectData.get();
>   DatumCodec codec = new DatumCodec(model, schema);
>
>   # convert datum to bytes using data model
>   byte[] asBytes = codec.toBytes(datum);
>
>   # convert bytes to datum using data model
>   ReflectClass copy = codec.fromBytes(asBytes);
>
> What do you think?
>
> rb
>
>
> [1]: https://issues.apache.org/jira/browse/AVRO-1704
>
>
> On 12/18/2015 05:01 AM, Niels Basjes wrote:
>
>> Hi,
>>
>> I'm working on a project where I'm putting Avro records into Kafka and at
>> the other end pull them out again.
>> For that purpose I wrote two methods 'toBytes' and 'fromBytes' in a
>> separate class (see below).
>>
>> I see this as the type of problem many developers run into.
>> Would it be a good idea to generate methods like these into the generated
>> Java code?
>>
>> This would make it possible to serialize and deserialize singles records
>> like this:
>>
>> byte [] someBytes = measurement.toBytes();
>> Measurement m = Measurement.fromBytes(someBytes);
>>
>> Niels Basjes
>>
>> P.S. possibly not name it toBytes but getBytes (similar to what the String
>> class has)
>>
>> public final class MeasurementSerializer {
>>  private MeasurementSerializer() {
>>  }
>>
>>  public static Measurement fromBytes(byte[] bytes) throws IOException
>> {
>>  try {
>>  DatumReader reader = new
>> SpecificDatumReader<>(Measurement.getClassSchema());
>>  Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,
>> null);
>>  return reader.read(null, decoder);
>>  } catch (RuntimeException rex) {
>>  throw new IOException(rex.getMessage());
>>  }
>>  }
>>
>>  public static byte[] toBytes(Measurement measurement) throws
>> IOException {
>>  try {
>>  ByteArrayOutputStream out = new ByteArrayOutputStream();
>>  Encoder encoder = EncoderFactory.get().binaryEncoder(out,
>> null);
>>  SpecificDatumWriter writer = new
>> SpecificDatumWriter<>(Measurement.class);
>>  writer.write(measurement, encoder);
>>  encoder.flush();
>>  out.close();
>>  return out.toByteArray();
>>  } catch (RuntimeException rex) {
>>  throw new IOException(rex.getMessage());
>>  }
>>  }
>> }
>>
>>
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Niels Basjes]

I was not clear enough in my previous email.
What I meant is to 'wrap' the application schema in a serialization wrapper
schema that has a field indicating the "schema classname".
That (generic setup) combined with some generated code in the schema
classes should yield a solution that supports schema migration.

Niels

On Tue, Dec 22, 2015 at 11:55 AM, Niels Basjes <ni...@basjes.nl> wrote:

> Thanks for pointing this out.
> This is exactly what I was working on.
>
> The way I solved the 'does the schema match' question at work is by
> requiring that all schema's start with a single text field "schema
> classname" being the full class name of the class that was used to generate
> it.
> That way we can have newer versions of the schema and still be able to
> unpack them. In this form the classname is essentially an indicator if
> schema migration is possible; even though the schemas are different.
>
> What do you think of this direction?
>
> Niels
>
>
> On Mon, Dec 21, 2015 at 11:30 PM, Ryan Blue <b...@cloudera.com> wrote:
>
>> Niels,
>>
>> This sounds like a good idea to me to have methods like this. I've had to
>> write those methods several times!
>>
>> The idea is also related to AVRO-1704 [1], which is a suggestion to
>> standardize the encoding that is used for single records. Some projects
>> have been embedding the schema fingerprint at the start of each record, for
>> example, which would be a helpful thing to do.
>>
>> It may also be a good idea to create a helper object rather than
>> attaching new methods to the datum classes themselves. In your example
>> below, you have to create a new encoder or decoder for each method call. We
>> could instead keep a backing buffer and encoder/decoder on a class that the
>> caller instantiates so that they can be reused. At the same time, that
>> would make it possible to reuse the class with any data model and manage
>> the available schemas (if embedding the fingerprint).
>>
>> I'm thinking something like this:
>>
>>   ReflectClass datum = new ReflectClass();
>>   ReflectData model = ReflectData.get();
>>   DatumCodec codec = new DatumCodec(model, schema);
>>
>>   # convert datum to bytes using data model
>>   byte[] asBytes = codec.toBytes(datum);
>>
>>   # convert bytes to datum using data model
>>   ReflectClass copy = codec.fromBytes(asBytes);
>>
>> What do you think?
>>
>> rb
>>
>>
>> [1]: https://issues.apache.org/jira/browse/AVRO-1704
>>
>>
>> On 12/18/2015 05:01 AM, Niels Basjes wrote:
>>
>>> Hi,
>>>
>>> I'm working on a project where I'm putting Avro records into Kafka and at
>>> the other end pull them out again.
>>> For that purpose I wrote two methods 'toBytes' and 'fromBytes' in a
>>> separate class (see below).
>>>
>>> I see this as the type of problem many developers run into.
>>> Would it be a good idea to generate methods like these into the generated
>>> Java code?
>>>
>>> This would make it possible to serialize and deserialize singles records
>>> like this:
>>>
>>> byte [] someBytes = measurement.toBytes();
>>> Measurement m = Measurement.fromBytes(someBytes);
>>>
>>> Niels Basjes
>>>
>>> P.S. possibly not name it toBytes but getBytes (similar to what the
>>> String
>>> class has)
>>>
>>> public final class MeasurementSerializer {
>>>  private MeasurementSerializer() {
>>>  }
>>>
>>>  public static Measurement fromBytes(byte[] bytes) throws
>>> IOException {
>>>  try {
>>>  DatumReader reader = new
>>> SpecificDatumReader<>(Measurement.getClassSchema());
>>>  Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,
>>> null);
>>>  return reader.read(null, decoder);
>>>  } catch (RuntimeException rex) {
>>>  throw new IOException(rex.getMessage());
>>>  }
>>>  }
>>>
>>>  public static byte[] toBytes(Measurement measurement) throws
>>> IOException {
>>>  try {
>>>  ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>  Encoder encoder = EncoderFactory.get().binaryEncoder(out,
>>> null);
>>>  SpecificDatumWriter writer = new
>>> SpecificDatumWriter<>(Measurement.class);
>>>  writer.write(measurement, encoder);
>>>  encoder.flush();
>>>  out.close();
>>>  return out.toByteArray();
>>>  } catch (RuntimeException rex) {
>>>  throw new IOException(rex.getMessage());
>>>  }
>>>  }
>>> }
>>>
>>>
>>>
>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Cloudera, Inc.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Niels Basjes]

Yes, as a generic crossplatform solution this makes a lot of sense. It s
easy to build and stops consuming the messages as soon as the fingerprint
changes.
In my corporate reality I see that a source of messages puts them into
Kafka, then several consumers read and deserialize them in near-realtime.
In the static situation of "the schema doesn't change" having a fingerprint
of the schema would suffice.
But now the schema changes. I would like to deploy the new version of the
source system, and consume these new messages using the schema migration
idea. I.e. read the records as long as they are backwards compatible.
As a general assumption: it is impossible to deploy a new version of all
consuming applications at the same time.
I want as little down time as possible and I do not want to lose any
messages.

Is this possible with a simple fingerprint?
The only direction I can think of right now is by having a central registry
that contains all schemas ever used, indexed by their fingerprint.

What alternative solutions can we think of?

Niels

On Wed, 23 Dec 2015, 01:42 Sean Busbey <bus...@cloudera.com> wrote:

> Including a schema fingerprint at the start
>
> 1) reuses stuff we have
> 2) gives a language independent notion of compatibility
> 3) doesn't bind how folks get stuff in/out of the single record form.
>
> --
> Sean Busbey
> On Dec 22, 2015 06:52, "Niels Basjes" <ni...@basjes.nl> wrote:
>
> > I was not clear enough in my previous email.
> > What I meant is to 'wrap' the application schema in a serialization
> wrapper
> > schema that has a field indicating the "schema classname".
> > That (generic setup) combined with some generated code in the schema
> > classes should yield a solution that supports schema migration.
> >
> > Niels
> >
> > On Tue, Dec 22, 2015 at 11:55 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > > Thanks for pointing this out.
> > > This is exactly what I was working on.
> > >
> > > The way I solved the 'does the schema match' question at work is by
> > > requiring that all schema's start with a single text field "schema
> > > classname" being the full class name of the class that was used to
> > generate
> > > it.
> > > That way we can have newer versions of the schema and still be able to
> > > unpack them. In this form the classname is essentially an indicator if
> > > schema migration is possible; even though the schemas are different.
> > >
> > > What do you think of this direction?
> > >
> > > Niels
> > >
> > >
> > > On Mon, Dec 21, 2015 at 11:30 PM, Ryan Blue <b...@cloudera.com> wrote:
> > >
> > >> Niels,
> > >>
> > >> This sounds like a good idea to me to have methods like this. I've had
> > to
> > >> write those methods several times!
> > >>
> > >> The idea is also related to AVRO-1704 [1], which is a suggestion to
> > >> standardize the encoding that is used for single records. Some
> projects
> > >> have been embedding the schema fingerprint at the start of each
> record,
> > for
> > >> example, which would be a helpful thing to do.
> > >>
> > >> It may also be a good idea to create a helper object rather than
> > >> attaching new methods to the datum classes themselves. In your example
> > >> below, you have to create a new encoder or decoder for each method
> > call. We
> > >> could instead keep a backing buffer and encoder/decoder on a class
> that
> > the
> > >> caller instantiates so that they can be reused. At the same time, that
> > >> would make it possible to reuse the class with any data model and
> manage
> > >> the available schemas (if embedding the fingerprint).
> > >>
> > >> I'm thinking something like this:
> > >>
> > >>   ReflectClass datum = new ReflectClass();
> > >>   ReflectData model = ReflectData.get();
> > >>   DatumCodec codec = new DatumCodec(model, schema);
> > >>
> > >>   # convert datum to bytes using data model
> > >>   byte[] asBytes = codec.toBytes(datum);
> > >>
> > >>   # convert bytes to datum using data model
> > >>   ReflectClass copy = codec.fromBytes(asBytes);
> > >>
> > >> What do you think?
> > >>
> > >> rb
> > >>
> > >>
> > >> [1]: https://issues.apache.org/jira/browse/AVRO-1704
> > >>
> > >>
> > >> On 

Re: [DISCUSS][JAVA] Generating toBytes/fromBytes methods?

[Niels Basjes]

Guys? Any opinions on this idea?
On 18 Dec 2015 14:01, "Niels Basjes" <ni...@basjes.nl> wrote:

> Hi,
>
> I'm working on a project where I'm putting Avro records into Kafka and at
> the other end pull them out again.
> For that purpose I wrote two methods 'toBytes' and 'fromBytes' in a
> separate class (see below).
>
> I see this as the type of problem many developers run into.
> Would it be a good idea to generate methods like these into the generated
> Java code?
>
> This would make it possible to serialize and deserialize singles records
> like this:
>
> byte [] someBytes = measurement.toBytes();
> Measurement m = Measurement.fromBytes(someBytes);
>
> Niels Basjes
>
> P.S. possibly not name it toBytes but getBytes (similar to what the String
> class has)
>
> public final class MeasurementSerializer {
> private MeasurementSerializer() {
> }
>
> public static Measurement fromBytes(byte[] bytes) throws IOException {
> try {
> DatumReader reader = new 
> SpecificDatumReader<>(Measurement.getClassSchema());
> Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
> return reader.read(null, decoder);
> } catch (RuntimeException rex) {
> throw new IOException(rex.getMessage());
> }
> }
>
> public static byte[] toBytes(Measurement measurement) throws IOException {
> try {
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
> SpecificDatumWriter writer = new 
> SpecificDatumWriter<>(Measurement.class);
> writer.write(measurement, encoder);
> encoder.flush();
> out.close();
> return out.toByteArray();
>     } catch (RuntimeException rex) {
> throw new IOException(rex.getMessage());
> }
> }
> }
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

Re: [DISCUSS] Migrate to Java 7

[Niels Basjes]

Ah, yes I understand what you mean.
I mean to say something slightly different.
1) Keep/Make the source Java 1.7 compliant.
2) Compile using the newest compiler (i.e. 1.8)
3) Compile towards the binary 1.7 compliant.

That way we're taking advantage of the best compiler yet with enough
backward compatibility.
This would also avoid the 1.8 bytecode problems.

Niels Basjes



On Fri, Dec 18, 2015 at 5:59 PM, Ryan Blue <b...@cloudera.com> wrote:

> I would agree, but unfortunately Java 8 features required bytecode changes
> and Java 8 can't be compiled to target Java 7. There is a good summary of
> it a few answers down on this SO question:
>
>
>
> https://stackoverflow.com/questions/16143684/can-java-8-code-be-compiled-to-run-on-java-7-jvm
>
> I think that means we should stay on Java 7 as long as possible. I propose
> taking a look at this is 6 months or so to see how many people are still
> running Java 7 or have moved to 8.
>
> rb
>
>
> On 12/18/2015 05:03 AM, Niels Basjes wrote:
>
>> Ryan,
>>
>> Perhaps we should even make the step to build using Java 8, yet generate
>> bytecode at the Java 7 level.
>>
>> Niels
>>
>> On Fri, Dec 18, 2015 at 1:25 PM, Niels Basjes <ni...@basj.es> wrote:
>>
>> +1
>>> On 18 Dec 2015 12:33, "Tom White" <t...@cloudera.com> wrote:
>>>
>>> +1
>>>>
>>>> Tom
>>>>
>>>> On Tue, Dec 15, 2015 at 1:08 AM, Ryan Blue <b...@apache.org> wrote:
>>>>
>>>> I just noticed that our tests are still compiling and running with Java
>>>>>
>>>> 6.
>>>>
>>>>> Java 7 is already end-of-life (public patches at least), so I think it
>>>>>
>>>> is
>>>>
>>>>> reasonable to start migrating. Is everyone okay with updating the
>>>>> builds
>>>>> and dropping support for Java 6?
>>>>>
>>>>> rb
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Avro release 1.8.0 (rc3)

[Niels Basjes]

+1
* Set the Maven staging as a repo on my system.
* Updated several of my projects that rely heavily on AVRO to use the 1.8.0
version.
* Rebuild and ran all the unit tests on those projects; all passed without
incident

On Wed, Jan 27, 2016 at 7:36 PM, Douglas Creager <dcrea...@dcreager.net>
wrote:

> +1
>
> Verified signatures and checksums.  C tests all pass, shared library
> version looks good.
>
> On Mon, Jan 25, 2016, at 07:05 AM, Tom White wrote:
> > Hi everyone,
> >
> > I've created a new release candidate for Avro 1.8.0.
> >
> > The changes are listed at:
> > http://s.apache.org/avro180
> >
> > The release artifacts can be found here:
> > https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc3/
> >
> > The tag corresponding to this release candidate is:
> > http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc3/
> >
> > You can find the KEYS file here:
> > https://dist.apache.org/repos/dist/release/avro/KEYS
> >
> > The Maven staging repository is at:
> > https://repository.apache.org/content/repositories/orgapacheavro-1005/
> >
> > Please download, verify, and test. The vote will remain open for at least
> > 72 hours. Again, this is the first release that has been built using
> > Docker, so please pay extra attention to the languages you are interested
> > in. Thanks in advance for voting!
> >
> > Cheers,
> > Tom
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Draft board report

[Niels Basjes]

Looks good to me
On 13 Apr 2016 06:21, "Sean Busbey"  wrote:

> +1 thanks Ryan!
>
> On Tue, Apr 12, 2016 at 6:20 PM, Ryan Blue 
> wrote:
> > Hi everyone,
> >
> > Board reports are due tomorrow, here's a draft of what I'm planning to
> > submit. If you want to add or edit something, please reply.
> >
> > I think it's been a good quarter; I'm really excited that we got the
> 1.8.0
> > release out and added the new JS implementation. I'm also highlighting
> that
> > something we should work increasing engagement with potential
> contributors.
> > We've been letting some bug reports slip through without attention, some
> of
> > which even have fixes attached. I have a few ideas about how to work on
> > this that I'll write up, but please throw out your ideas for increasing
> our
> > engagement!
> >
> > Thanks,
> >
> > rb
> >
> > ## Description:
> >
> >  - Avro is a cross-language data serialization system.
> >
> > ## Issues:
> >
> >  - There are no issues requiring board attention at this time.
> >
> > ## Activity:
> >
> >  - Released 1.8.0 on 29 January 2016.
> >  - The latest release brings the project's licensing documentation
> >up-to-date, which was a major blocker.
> >  - 1.8.0 includes a new JavaScript implementation
> >
> > ## Health report:
> >
> >  - Avro's last release, 1.7.7, was 23 July 2014. More than a year
> >between releases is not a healthy cadence.
> >  - Avro needs to increase engagement with potential contributors.
> > - Some bug reports, reviews are not getting enough attention.
> > - Some language implementations are unmaintained.
> >
> > ## PMC changes:
> >
> >  - Currently 16 PMC members.
> >  - New PMC members:
> > - Sean Busbey was added to the PMC on Sun Feb 14 2016
> > - Martin Kleppmann was added to the PMC on Sun Feb 14 2016
> >
> > ## Committer base changes:
> >
> >  - Currently 23 committers.
> >  - Matthieu Monsch was added as a committer on Mon Jan 18 2016
> >
> > ## Releases:
> >
> >  - Last release: 1.8.0 on 29 January 2016
> >
> > ## Mailing list activity:
> >
> >  - dev@avro.apache.org:
> > - 273 subscribers (up 2 in the last 3 months):
> > - 635 emails sent to list (541 in previous quarter)
> >
> >  - u...@avro.apache.org:
> > - 605 subscribers (up 7 in the last 3 months):
> > - 107 emails sent to list (70 in previous quarter)
> >
> > ## JIRA activity:
> >
> >  - 45 JIRA tickets created in the last 3 months
> >  - 26 JIRA tickets closed/resolved in the last 3 months
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
>
>
>
> --
> busbey
>

Re: [DISCUSS] Quarterly release goal

[Niels Basjes]

+1 for quarterly releases
On 17 Apr 2016 02:15, "Ryan Blue"  wrote:

> Hi everyone,
>
> It's been about 3 months since we released Avro 1.8.0 and we've already
> accumulated several fixes that we should get out in a release. Sean
> suggested it a few days ago, but I'm not sure if everyone saw that thread.
> Anyone interested in being the release manager for 1.8.1?
>
> I think we should set a goal of a release about once each quarter. If we
> let ourselves go 18 months between releases, then contributors can't use
> their work soon enough to continue caring and contributing.
>
> I think a quarterly release goal is a good first step toward making the
> project more contributor-friendly. For example, we usually have good
> participation getting ready for releases so it's a good excuse to get some
> reviews done in addition to getting committed fixes out the door.
>
> Thoughts and comments?
>
> rb
>
>
> --
> Ryan Blue
>

Re: 1.8.1 release

[Niels Basjes]

Ok, I'll investigate this.
On 13 May 2016 17:44, "Ryan Blue" <rb...@netflix.com.invalid> wrote:

> I think that toString problem has already been fixed, the original issue
> was AVRO-1799.
>
> rb
>
> On Fri, May 13, 2016 at 7:04 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > I added an extra issue that I'll work on this weekend.
> > https://issues.apache.org/jira/browse/AVRO-1845
> >
> > It is a pull request on github that involves data corruption.
> >
> > Niels
> >
> > On Mon, May 9, 2016 at 5:36 PM, Ryan Blue <b...@apache.org> wrote:
> >
> > > Hi everyone,
> > >
> > > Just wanted to post an update on the 1.8.1 release. The release issue,
> > > AVRO-1833 [1], accumulated 5 issues to complete before the release. 2
> of
> > > those are done, and two more are pending review. One last issue,
> > AVRO-1723,
> > > is a nice-to-have because I'm not sure it will be done in time.
> > >
> > > I'm planning to get a release vote out this week, so please commit any
> > > remaining items you'd like to get in.
> > >
> > > If we can get the release out this week, then that puts the next
> release
> > > around mid-August. If anyone would like to be RM for that release,
> please
> > > speak up!
> > >
> > >
> > > rb
> > >
> > > [1]: https://issues.apache.org/jira/browse/AVRO-1833
> > >
> > >
> > > --
> > > Ryan Blue
> > >
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Adding myself to the "Credits" page ?

[Niels Basjes]

Thanks, that worked perfectly.

On Tue, May 3, 2016 at 5:47 PM, Ryan Blue <rb...@netflix.com.invalid> wrote:

> The site is stored in SVN here: https://svn.apache.org/repos/asf/avro/site
>
> You should be able to add yourself by editing files in author/. The site is
> built with ant and you need to point ant to a download of Apache Forrest (I
> used 0.9):
>
> ```
> ant -Dforrest.home=/path/to/apache-forrest-0.9/
> ```
>
> Then just svn commit to publish. I think all committers have access, but
> let me know if you run into trouble.
>
> rb
>
> On Tue, May 3, 2016 at 7:15 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > How do I go about adding myself as one of the active committers to this
> > page?
> > https://avro.apache.org/credits.html
> >
> > This is what I would like to see added:
> >
> > username: nielsbasjes
> > name:   Niels Basjes
> > organization:  Bol.com
> > roles: java, docker
> > timezone: +1  I'm in TZ Europe/Amsterdam ~
> > CET/CEST which is +1/+2 depending on summertime
> >
> > I also noticed that for example Ryan is not on that list also.
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Adding myself to the "Credits" page ?

[Niels Basjes]

Hi,

How do I go about adding myself as one of the active committers to this
page?
https://avro.apache.org/credits.html

This is what I would like to see added:

username: nielsbasjes
name:   Niels Basjes
organization:  Bol.com
roles: java, docker
timezone: +1  I'm in TZ Europe/Amsterdam ~
CET/CEST which is +1/+2 depending on summertime

I also noticed that for example Ryan is not on that list also.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Moved to git!

[Niels Basjes]

Hi,

I just noticed both these pages
 http://avro.apache.org/version_control.html
 https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
have not been updated to reflect the change to git.

Can one of you guys pick this up please?

Niels Basjes


On Wed, Feb 10, 2016 at 5:13 PM, Sean Busbey <bus...@cloudera.com> wrote:

> On Tue, Feb 9, 2016 at 8:56 AM, Tom White <t...@cloudera.com> wrote:
>
> > On Thu, Feb 4, 2016 at 11:23 PM, Ryan Blue <b...@apache.org> wrote:
> > > The new git repository is live! You can clone it from here:
> > >
> > >   https://git-wip-us.apache.org/repos/asf/avro.git
> > >
> > > It looks like the commit hashes are identical to the ones in the github
> > > mirror, so it should just appear like trunk has been renamed to master
> if
> > > you've already cloned the github mirror. In that case, just run this:
> > >
> > >   git remote add apache
> https://git-wip-us.apache.org/repos/asf/avro.git
> > >
> > > The old SVN repository is still RW so we can change the site/ folders,
> so
> > > please remember to push to the git repo's master instead of committing
> > code
> > > changes to SVN. Does anyone know if we still use those folders? If not,
> > then
> > > we can probably switch it over to read-only.
> >
> > The site folder (https://svn.apache.org/repos/asf/avro/site) contains
> > the website, so we do still need to able to write to it. As far as I
> > know, sites still use svnpubsub at the ASF so we need to keep these
> > pages in svn (see http://www.apache.org/dev/project-site.html).
> >
> > Tom
> >
> >
> There's a gitpubsub now, so we could move the site over to git if we so
> desired. It can either be a branch (asf-site) or it can be in its own git
> repo (useful if we want to automate site generation and publication at some
> point).
>
> --
> Sean
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[IDEA] Making schema evolution for enums slightly easier.

[Niels Basjes]

Hi,

I'm working on a project where we are putting message serialized avro
records into Kafka. The schemas are made available via a schema registry of
some sorts.
Because Kafka stores the messages for a longer period 'weeks' we have two
common scenarios that occur when a new version of the schema is introduced
(i.e. from V1 to V2).

1) A V2 producer is released and a V1 consumer must be able to read the
records.
2) A 'new' V2 consumer is released a few days after the V2 producer started
creating records. The V2 consumer starts reading Kafka "from the beginning"
and as a consequence first has to go through a set of V1 records.

So in this usecase we need schema evolution in two directions.

To make sure it all works as expected I did some experiments and found that
these requirements are all doable except when you are in need of an enum.

This 'two directions' turns out to have a problem with changing the values
of an enum.

You cannot write an enum { 'A', 'B', 'C' } and then read it with the schema
enum { 'A', 'B' }


So I was thinking about a possible way to make this easier for the
developer.

The current idea that I want your opinion on:
1) In the IDL we add a way of directing that we want the enum to be stored
in a different way in the schema. I was thinking about something like
either defining a new type like 'string enum' or perhaps use an annotation
of some sorts.
2) The 'string enum' is mapped into the actual schema as a string (which
can contain ANY value). So anyone using the json schema can simply read it
because it is a string.
3) The generated code that is used to set/change the value enforces that
only the allowed values can be set.

This way a 'reader' can read any value, the schema is compatible in all
directions.

What do you guys think?
Is this an idea worth trying out?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [IDEA] Making schema evolution for enums slightly easier.

[Niels Basjes]

Thanks for the idea.
I'm gonna play around with that to see if it could work.

Niels

On Tue, Jan 31, 2017 at 5:57 PM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> If you want to solve this problem by using a String to encode the value,
> then you can do that by defining a logical type that is an enum-as-string.
> But I'm not sure you want to do that. The nice thing about an enum is that
> you use what you know about the schema ahead of time to get a much more
> compact representation -- usually a byte rather than encoding the entire
> string. So I'd much rather find a way of handling this case that keeps the
> compact representation, while allowing for applications to gracefully
> handling these.
>
> For generic, enum symbols are translated to GenericEnumSymbol, which can
> hold any symbol. Adding an option to return the symbol from the writer's
> schema even if it isn't in the reader's schema is one way around the
> problem. That wouldn't work for reflect or specific, though.
>
> Another option that was suggested last year is to designate a catch-all
> enum symbol. So your enum would be { 'A', 'B', 'UNKNOWN' } and { 'A', 'B',
> 'C', 'UNKNOWN' }. When a v1 consumer reads v2 records, C gets turned into
> UNKNOWN.
>
> I like the designated catch-all symbol because it is a reasonable way to
> opt-in for forward-compatibility.
>
> rb
>
> On Tue, Jan 31, 2017 at 2:04 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > I'm working on a project where we are putting message serialized avro
> > records into Kafka. The schemas are made available via a schema registry
> of
> > some sorts.
> > Because Kafka stores the messages for a longer period 'weeks' we have two
> > common scenarios that occur when a new version of the schema is
> introduced
> > (i.e. from V1 to V2).
> >
> > 1) A V2 producer is released and a V1 consumer must be able to read the
> > records.
> > 2) A 'new' V2 consumer is released a few days after the V2 producer
> started
> > creating records. The V2 consumer starts reading Kafka "from the
> beginning"
> > and as a consequence first has to go through a set of V1 records.
> >
> > So in this usecase we need schema evolution in two directions.
> >
> > To make sure it all works as expected I did some experiments and found
> that
> > these requirements are all doable except when you are in need of an enum.
> >
> > This 'two directions' turns out to have a problem with changing the
> values
> > of an enum.
> >
> > You cannot write an enum { 'A', 'B', 'C' } and then read it with the
> schema
> > enum { 'A', 'B' }
> >
> >
> > So I was thinking about a possible way to make this easier for the
> > developer.
> >
> > The current idea that I want your opinion on:
> > 1) In the IDL we add a way of directing that we want the enum to be
> stored
> > in a different way in the schema. I was thinking about something like
> > either defining a new type like 'string enum' or perhaps use an
> annotation
> > of some sorts.
> > 2) The 'string enum' is mapped into the actual schema as a string (which
> > can contain ANY value). So anyone using the json schema can simply read
> it
> > because it is a string.
> > 3) The generated code that is used to set/change the value enforces that
> > only the allowed values can be set.
> >
> > This way a 'reader' can read any value, the schema is compatible in all
> > directions.
> >
> > What do you guys think?
> > Is this an idea worth trying out?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Release Apache Avro 1.8.2 RC1

[Niels Basjes]

+1

I ran clean, test, dist and rat from within docker for all languages and
all passed.

Niels Basjes

On Tue, Nov 8, 2016 at 12:03 PM, Tom White <t...@cloudera.com> wrote:

> +1
>
> I verified checksums & signature, source tag, license/notice, and ran Java
> unit tests.
>
> Cheers,
> Tom
>
> On Sun, Nov 6, 2016 at 9:59 PM, Ryan Blue <b...@apache.org> wrote:
>
> > Hi everyone,
> >
> > I propose the following RC to be released as official Apache Avro 1.8.2
> > release.
> >
> > The commit id is 7ec35ea24ff0270586a26afbc6f6f530d272d1f7
> > * This corresponds to the tag: release-1.8.2-rc1
> > * https://github.com/apache/avro/tree/7ec35ea2
> > *
> > https://git-wip-us.apache.org/repos/asf/projects/repo?p=
> > avro.git=commit=7ec35ea2
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc1/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/release/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > * https://repository.apache.org/content/groups/staging/org/apache/avro/
> >
> > This release includes:
> > * A spec for single-message Avro encoding and a Java implementation
> > * Java: Bug fixes for Java logical types
> > * Java: Support for Decimal with specific classes
> > * Ruby: A fix for compatibility when using snappy
> > * Python 3: Updated to use the standard module-level logging pattern
> > * C++: Support for Boost >= 1.59
> > * And more bug fixes...
> >
> > Please download, verify, and test, then please vote by the end of
> > Wednesday, 9 November.
> >
> > [ ] +1 Release this as Apache Avro 1.8.2
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> >
> > --
> > Ryan Blue
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Release Apache Avro 1.8.2 RC1

[Niels Basjes]

@Simon I find it strange you say that the docker is 'leaking' something so
I did some testing.
I ran the tests on my system using docker and everything worked.
Then I remove all docker containers and images related to the avro build
environment and ran it again.
The second time indeed it failed

My best guess right now is that
1) during the creation of the docker image we do and update of the base
system and install all the tools.
2) Upstream something relevant to us changed.

People (like me) who have and existing image on their system the 'update'
of the OS is not done because the local image was built with the
instructions that are in the dockerfile.
So "New" docker images are different because the base OS has the updates
installed from a later point in time.

I created https://issues.apache.org/jira/browse/AVRO-1956
Working on a fix.

Niels



On Tue, Nov 8, 2016 at 10:52 PM, Simon Woodford <simonwoodf...@gmail.com>
wrote:

> -1 from me. C# tests fail to run. The source of the problem is as follows:
>
> Prior to this tag, the lang/csharp/build.sh file contained the test line:
> nunit-console --framework=4.0 Avro.nunit
>
> From discussion with @jcustenborder, it seems like the C# tests won't run
> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
> been done for 1.8.2) makes the tests run. But removing it makes the tests
> fail on my system (CentOS 7). This suggests that something is missing from
> the docker setup and is being picked up from the host computer, but I
> haven't had a chance to investigate.
>
> Given that it's a choice between it passing for me or passing on all Macs
> everywhere, go ahead and release. I'll chase this up for the 1.8.3 release.
> All tests other than the C# pass when running build.sh test from docker.
>
> Simon
>
>
> On Sun, Nov 6, 2016 at 9:59 PM, Ryan Blue <b...@apache.org> wrote:
>
> > Hi everyone,
> >
> > I propose the following RC to be released as official Apache Avro 1.8.2
> > release.
> >
> > The commit id is 7ec35ea24ff0270586a26afbc6f6f530d272d1f7
> > * This corresponds to the tag: release-1.8.2-rc1
> > * https://github.com/apache/avro/tree/7ec35ea2
> > *
> > https://git-wip-us.apache.org/repos/asf/projects/repo?p=
> > avro.git=commit=7ec35ea2
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc1/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/release/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > * https://repository.apache.org/content/groups/staging/org/apache/avro/
> >
> > This release includes:
> > * A spec for single-message Avro encoding and a Java implementation
> > * Java: Bug fixes for Java logical types
> > * Java: Support for Decimal with specific classes
> > * Ruby: A fix for compatibility when using snappy
> > * Python 3: Updated to use the standard module-level logging pattern
> > * C++: Support for Boost >= 1.59
> > * And more bug fixes...
> >
> > Please download, verify, and test, then please vote by the end of
> > Wednesday, 9 November.
> >
> > [ ] +1 Release this as Apache Avro 1.8.2
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> >
> > --
> > Ryan Blue
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Release Apache Avro 1.8.2 RC1

[Niels Basjes]

Ryan,

When you fix this build problem I vote +1 again.
Thanks.

Niels

On Thu, Nov 10, 2016 at 5:04 PM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> The C# issue was fixed by AVRO-1626 [1], which had to be reverted for the
> 1.8.2 RC because it caused build failures in docker. I reopened the issue
> so we can fix it for next release. Either way, the tests pass in the common
> environment and I would rather not fail an RC for something like this.
>
> [1]: https://issues.apache.org/jira/browse/AVRO-1626
>
> On Thu, Nov 10, 2016 at 7:17 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Guys,
> >
> > Please have a look at this idea: https://github.com/apache/avro/pull/153
> > If you agree/disagree please go ahead and change/fix/commit it.
> > I'll be at the Apache Con BigData in sevilla next week so I wil probably
> be
> > rather busy.
> >
> > Oh and yes, this also means I'm changing my stance on Avro 1.8.2 RC1 to :
> > -1
> > This RC1 is not good enough due to this issue.
> >
> > Niels Basjes
> >
> > On Thu, Nov 10, 2016 at 1:59 PM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > > @Simon I find it strange you say that the docker is 'leaking' something
> > so
> > > I did some testing.
> > > I ran the tests on my system using docker and everything worked.
> > > Then I remove all docker containers and images related to the avro
> build
> > > environment and ran it again.
> > > The second time indeed it failed
> > >
> > > My best guess right now is that
> > > 1) during the creation of the docker image we do and update of the base
> > > system and install all the tools.
> > > 2) Upstream something relevant to us changed.
> > >
> > > People (like me) who have and existing image on their system the
> 'update'
> > > of the OS is not done because the local image was built with the
> > > instructions that are in the dockerfile.
> > > So "New" docker images are different because the base OS has the
> updates
> > > installed from a later point in time.
> > >
> > > I created https://issues.apache.org/jira/browse/AVRO-1956
> > > Working on a fix.
> > >
> > > Niels
> > >
> > >
> > >
> > > On Tue, Nov 8, 2016 at 10:52 PM, Simon Woodford <
> simonwoodf...@gmail.com
> > >
> > > wrote:
> > >
> > >> -1 from me. C# tests fail to run. The source of the problem is as
> > follows:
> > >>
> > >> Prior to this tag, the lang/csharp/build.sh file contained the test
> > line:
> > >> nunit-console --framework=4.0 Avro.nunit
> > >>
> > >> From discussion with @jcustenborder, it seems like the C# tests won't
> > run
> > >> on a Mac if we keep the --framework=4.0 in this line; removing it (as
> > has
> > >> been done for 1.8.2) makes the tests run. But removing it makes the
> > tests
> > >> fail on my system (CentOS 7). This suggests that something is missing
> > from
> > >> the docker setup and is being picked up from the host computer, but I
> > >> haven't had a chance to investigate.
> > >>
> > >> Given that it's a choice between it passing for me or passing on all
> > Macs
> > >> everywhere, go ahead and release. I'll chase this up for the 1.8.3
> > >> release.
> > >> All tests other than the C# pass when running build.sh test from
> docker.
> > >>
> > >> Simon
> > >>
> > >>
> > >> On Sun, Nov 6, 2016 at 9:59 PM, Ryan Blue <b...@apache.org> wrote:
> > >>
> > >> > Hi everyone,
> > >> >
> > >> > I propose the following RC to be released as official Apache Avro
> > 1.8.2
> > >> > release.
> > >> >
> > >> > The commit id is 7ec35ea24ff0270586a26afbc6f6f530d272d1f7
> > >> > * This corresponds to the tag: release-1.8.2-rc1
> > >> > * https://github.com/apache/avro/tree/7ec35ea2
> > >> > *
> > >> > https://git-wip-us.apache.org/repos/asf/projects/repo?p=
> > >> > avro.git=commit=7ec35ea2
> > >> >
> > >> > The release tarball, signature, and checksums are here:
> > >> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc1/
> > >> >
> > >> > You can find the KEYS file here:
> > >> > * https://dist.apache.org/repos/dist/release/

Re: [VOTE] Release Apache Avro 1.8.2 RC1

[Niels Basjes]

Guys,

Please have a look at this idea: https://github.com/apache/avro/pull/153
If you agree/disagree please go ahead and change/fix/commit it.
I'll be at the Apache Con BigData in sevilla next week so I wil probably be
rather busy.

Oh and yes, this also means I'm changing my stance on Avro 1.8.2 RC1 to :
-1
This RC1 is not good enough due to this issue.

Niels Basjes

On Thu, Nov 10, 2016 at 1:59 PM, Niels Basjes <ni...@basjes.nl> wrote:

> @Simon I find it strange you say that the docker is 'leaking' something so
> I did some testing.
> I ran the tests on my system using docker and everything worked.
> Then I remove all docker containers and images related to the avro build
> environment and ran it again.
> The second time indeed it failed
>
> My best guess right now is that
> 1) during the creation of the docker image we do and update of the base
> system and install all the tools.
> 2) Upstream something relevant to us changed.
>
> People (like me) who have and existing image on their system the 'update'
> of the OS is not done because the local image was built with the
> instructions that are in the dockerfile.
> So "New" docker images are different because the base OS has the updates
> installed from a later point in time.
>
> I created https://issues.apache.org/jira/browse/AVRO-1956
> Working on a fix.
>
> Niels
>
>
>
> On Tue, Nov 8, 2016 at 10:52 PM, Simon Woodford <simonwoodf...@gmail.com>
> wrote:
>
>> -1 from me. C# tests fail to run. The source of the problem is as follows:
>>
>> Prior to this tag, the lang/csharp/build.sh file contained the test line:
>> nunit-console --framework=4.0 Avro.nunit
>>
>> From discussion with @jcustenborder, it seems like the C# tests won't run
>> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
>> been done for 1.8.2) makes the tests run. But removing it makes the tests
>> fail on my system (CentOS 7). This suggests that something is missing from
>> the docker setup and is being picked up from the host computer, but I
>> haven't had a chance to investigate.
>>
>> Given that it's a choice between it passing for me or passing on all Macs
>> everywhere, go ahead and release. I'll chase this up for the 1.8.3
>> release.
>> All tests other than the C# pass when running build.sh test from docker.
>>
>> Simon
>>
>>
>> On Sun, Nov 6, 2016 at 9:59 PM, Ryan Blue <b...@apache.org> wrote:
>>
>> > Hi everyone,
>> >
>> > I propose the following RC to be released as official Apache Avro 1.8.2
>> > release.
>> >
>> > The commit id is 7ec35ea24ff0270586a26afbc6f6f530d272d1f7
>> > * This corresponds to the tag: release-1.8.2-rc1
>> > * https://github.com/apache/avro/tree/7ec35ea2
>> > *
>> > https://git-wip-us.apache.org/repos/asf/projects/repo?p=
>> > avro.git=commit=7ec35ea2
>> >
>> > The release tarball, signature, and checksums are here:
>> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc1/
>> >
>> > You can find the KEYS file here:
>> > * https://dist.apache.org/repos/dist/release/avro/KEYS
>> >
>> > Binary artifacts for Java are staged in Nexus here:
>> > * https://repository.apache.org/content/groups/staging/org/apache/avro/
>> >
>> > This release includes:
>> > * A spec for single-message Avro encoding and a Java implementation
>> > * Java: Bug fixes for Java logical types
>> > * Java: Support for Decimal with specific classes
>> > * Ruby: A fix for compatibility when using snappy
>> > * Python 3: Updated to use the standard module-level logging pattern
>> > * C++: Support for Boost >= 1.59
>> > * And more bug fixes...
>> >
>> > Please download, verify, and test, then please vote by the end of
>> > Wednesday, 9 November.
>> >
>> > [ ] +1 Release this as Apache Avro 1.8.2
>> > [ ] +0
>> > [ ] -1 Do not release this because...
>> >
>> >
>> > --
>> > Ryan Blue
>> >
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Community engagement

[Niels Basjes]

Hi,

Regarding the CI integration (like Travis) I think this is something we
should do anyway. So +1 on that one.
I would like to see it set up that
- any (github) merge request is built automatically.
- any commit to master is built automatically AND (if successful) pushed to
a maven repo as SNAPSHOT.

About splitting up the project per language: I share the concern that some
languages will start to fall behind.
Also:
Assume have a separate "project" that manages the specification and the
test cases that all languages must be able to handle/read/write in order to
classify as "Follows Avro specification version X". With that we can
release a new version of the  libraries independently
from the others.

A slightly alternative route would be do make Java the 'reference
implementation' by combining Spec
That way the specification and java would be a single project and the
others would be separate.
The effect is then that with a new release of the Java the version of the
'spec' would be updated too and we will have testing libraries available to
creating test cases.
Al in all ... not too different from what we now have.

I think for now we should implement the CI stuff (preferable like I
indicated above).
My instinct tells me that with that in place any 'build breaking change' in
'any language' will be caught very quickly and should not be a problem.

Niels Basjes

P.S. The CI build should do as many of the things currently in the build.sh
scripts. This includes things like running rat.


On Thu, Nov 24, 2016 at 12:10 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> Simon brought up compatibility as well. This is an existing problem and
> right now we have some interop tests where some languages generate data and
> then read all of the data from other languages. I don't think this problem
> changes much if we move to separate releases.
>
> We can still run these tests, but could change them so that we update just
> one version of a particular language to pull in an RC and use released
> versions of other implementations. That could make it easier to spot
> problems because only one language is changing at a time.
>
> Of course, just having these tests doesn't mean they are kept up to date.
> Ruby's snappy implementation was incompatible with the others until
> recently. I think that overall, splitting the projects won't result in a
> noticeable change in interoperability.
>
> rb
>
> On Mon, Nov 21, 2016 at 10:20 AM, Zoltan Ivanfi <z...@cloudera.com> wrote:
>
> > Hi,
> >
> > A question just came to my mind: How would inter-language compatibility
> > work after the split? How will we now which versions of the different
> > language bindings support the same features and allow bidirectional
> > exchange of data?
> >
> > Or is it an already existing problem? I don't know too much about the
> > language bindings you mentioned; if some of them have not been worked on
> > recently, does this mean that they are lacking in support for some
> features
> > of the specification? If I use libraries from the same release of Avro in
> > software components of different languages, are they not guaranteed to be
> > compatible with each other?
> >
> > Thanks,
> >
> > Zoltan
> >
> > On Sat, Nov 19, 2016 at 10:03 PM Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> >
> > > This is in response to Tom's concerns:
> > >
> > > > When I did the 1.8.0 release I had to fix a few failing tests for a
> > > couple of languages, but it was mainly just doing library updates.
> > >
> > > I think there will be less trouble with builds if we have separate
> > > releases, but my main motivation is to avoid languages blocking one
> > another
> > > and make it possible to get more releases out. I think we could have
> had
> > > several JavaScript releases this year if JS could have been released
> > > independently, but the rest of the community isn't moving fast enough
> for
> > > that. And that's fine; we need to fix bugs in C before we put out a C
> > > release, but we don't want C to prevent other language contributors
> from
> > > moving forward.
> > >
> > > > I'm not at all convinced that splitting the project into per-language
> > > pieces
> > > will help. It will be a lot of work and I think we'll just get lots of
> > > orphaned languages that never get released.
> > >
> > > Splitting does two positive things: it allows releases to happen
> > > independently and it makes maintenance and contribution easier.
> > >
> > > Independent releases are a net positive, in my opinion. It is impo

Checking if schema evolution will work.

[Niels Basjes]

Hi,

I have been looking but I have not been able to find a tool that simply
tells me if a schema and evolve into another schema.
I want to create a test in my application that simply checks (unit test) if
the current schema can be used in conjuction with an older schema or if we
will have problems in production.

Do we have such a check somewhere (in the code base)?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [REPORT] January 2017 board report

[Niels Basjes]

LGTM

On 13 Jan 2017 19:53, "Ryan Blue" <rb...@netflix.com.invalid> wrote:

> Everyone,
>
> Here's the January 2017 board report. Please reply if you'd like to make a
> correction. Notably, my summary of recent threads on community engagement
> may need to be updated. My intent was a neutral summary.
>
> Thanks,
>
> rb
>
>
> ## Description:
>
>  - Avro is a cross-language data serialization system.
>
> ## Issues:
>
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>
>  - The 1.8.2 release candidate failed with too few PMC votes.
>
> ## Health report:
>
>  - The 1.8.2 release candidate failed because only 2 PMC members
>voted. When discussed on the dev list, there were two
>suggestions to avoid this in the future. First, use open-ended
>votes to give PMC members more time to notice there is a vote.
>Second, the RM should reach out to PMC members to make sure
>they see that a vote is happening.
>
>  - After last report IDF asked, "What steps precisely do you have
>planned to take to increase contributor engagement?" The step
>that the community agrees on is continuous integration to
>provide automated feedback to both contributors and committers.
>Not much has been done to set up automated testing.
>
>Part of the community is also in favor of splitting out
>implementations into separate repositories to make the project
>easier to work with. This idea is controversial and discussion
>has stalled without a resolution.
>
>  - Last report stated, "Contributor engagement is still a goal we
>are working on. Many committers do not have time to review
>patches." IDF asked, "Where does this time pressure come from?"
>To clarify, time pressure was a generous assumption. A more
>correct statement is simply that reviews are not keeping pace
>with contributions. Part of the community considers this a
>problem becuase without reviews we can't grow the community.
>
> ## PMC changes:
>
>  - Currently 17 PMC members.
>  - Niels Basjes was added to the PMC on Fri Dec 16 2016
>
> ## Committer base changes:
>
>  - Currently 23 committers.
>  - No new committers added in the last 3 months
>  - Last committer addition was Matthieu Monsch at Mon Jan 18 2016
>
> ## Releases:
>
>  - 1.8.1 was released on Fri May 20 2016
>
> ## Mailing list activity:
>
>  - dev@avro.apache.org:
> - 278 subscribers (down -1 in the last 3 months):
> - 462 emails sent to list (757 in previous quarter)
>
>  - u...@avro.apache.org:
> - 633 subscribers (up 4 in the last 3 months):
> - 68 emails sent to list (45 in previous quarter)
>
> ## JIRA activity:
>
>  - 51 JIRA tickets created in the last 3 months
>  - 18 JIRA tickets closed/resolved in the last 3 months
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Fix for avro compiler error

[Niels Basjes]

Hi,

I had a quick look.
As far as I can see this issue has already been fixed.
https://issues.apache.org/jira/browse/AVRO-1901

So why another pull request?

Niels Basjes


On Fri, Jan 13, 2017 at 12:57 PM, Ben McCann <bmcc...@linkedin.com.invalid>
wrote:

> Just a friendly reminder to please take a look at this PR.
>
> Thanks,
> Ben
>
>
> On Wed, Jan 11, 2017 at 11:47 AM, Ben McCann <bmcc...@linkedin.com> wrote:
>
> > Hi,
> >
> > Would someone be able to merge this pending PR?
> >
> > https://github.com/apache/avro/pull/140
> >
> > Thanks,
> > Ben
> >
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Avro 1.8.2 release

[Niels Basjes]

This jira and pull request for this NPE:

https://issues.apache.org/jira/browse/AVRO-1966
https://github.com/apache/avro/pull/166

On Wed, Nov 30, 2016 at 9:59 PM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi Guys,
>
> Today a colleague of mine ran into a nasty NPE in the generated code.
> I found and fixed it.
> Can you please review it.
> I would really like to see this one go into the 1.8.2 is it is possible.
>
> Thanks
>
> Niels Basjes
>
> On Mon, Oct 31, 2016 at 6:22 PM, S G <sg.online.em...@gmail.com> wrote:
>
>> Thanks for merging the patch Ryan. Appreciate it !
>>
>> On Sun, Oct 30, 2016 at 1:59 PM, Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> > SG,
>> >
>> > I'll take a look at it. Thanks!
>> >
>> > rb
>> >
>> > On Sun, Oct 30, 2016 at 1:58 PM, S G <sg.online.em...@gmail.com> wrote:
>> >
>> > > Hey Ryan,
>> > >
>> > > Any chance https://github.com/apache/avro/pull/113 can make it to
>> this
>> > > release?
>> > > Its been pending review for a couple months now.
>> > >
>> > > Thanks
>> > > SG
>> > >
>> > > On Sun, Oct 30, 2016 at 1:46 PM, Ryan Blue <b...@apache.org> wrote:
>> > >
>> > > > Hi everyone,
>> > > >
>> > > > I'm going to try to get a 1.8.2 release candidate out this week.
>> > There's
>> > > > only one issue that I think is a blocker on AVRO-1885, and I think
>> we
>> > can
>> > > > get that bug fixed. If anyone would like to get other commits in or
>> > think
>> > > > there are more blockers, please link them to AVRO-1885 and we can
>> > discuss
>> > > > on that issue.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > rb
>> > > >
>> > > > --
>> > > > Ryan Blue
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Ryan Blue
>> > Software Engineer
>> > Netflix
>> >
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Avro 1.8.2 release

[Niels Basjes]

Hi Guys,

Today a colleague of mine ran into a nasty NPE in the generated code.
I found and fixed it.
Can you please review it.
I would really like to see this one go into the 1.8.2 is it is possible.

Thanks

Niels Basjes

On Mon, Oct 31, 2016 at 6:22 PM, S G <sg.online.em...@gmail.com> wrote:

> Thanks for merging the patch Ryan. Appreciate it !
>
> On Sun, Oct 30, 2016 at 1:59 PM, Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
> > SG,
> >
> > I'll take a look at it. Thanks!
> >
> > rb
> >
> > On Sun, Oct 30, 2016 at 1:58 PM, S G <sg.online.em...@gmail.com> wrote:
> >
> > > Hey Ryan,
> > >
> > > Any chance https://github.com/apache/avro/pull/113 can make it to this
> > > release?
> > > Its been pending review for a couple months now.
> > >
> > > Thanks
> > > SG
> > >
> > > On Sun, Oct 30, 2016 at 1:46 PM, Ryan Blue <b...@apache.org> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm going to try to get a 1.8.2 release candidate out this week.
> > There's
> > > > only one issue that I think is a blocker on AVRO-1885, and I think we
> > can
> > > > get that bug fixed. If anyone would like to get other commits in or
> > think
> > > > there are more blockers, please link them to AVRO-1885 and we can
> > discuss
> > > > on that issue.
> > > >
> > > > Thanks!
> > > >
> > > > rb
> > > >
> > > > --
> > > > Ryan Blue
> > > >
> > >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Status of release 1.8.2??

[Niels Basjes]

What do you think of this very simple attempt?
https://github.com/apache/avro/pull/153

Niels

On 1 Dec 2016 18:31, "Ryan Blue" <rb...@netflix.com.invalid> wrote:

> I'd like to get a release out as well. I'm happy to roll another RC soon.
>
> Could someone look at the C# issue that was a problem for some people
> during the last vote? It would be nice to have something that works both
> inside and outside docker, without detecting whether or not it is running
> in docker.
>
> rb
>
> On Wed, Nov 30, 2016 at 1:42 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > What is the current status of the new release?
> > The first RC failed yet I would really like to see it 'out there'.
> > So what is the status at the moment?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Status of release 1.8.2??

[Niels Basjes]

Hi,

What is the current status of the new release?
The first RC failed yet I would really like to see it 'out there'.
So what is the status at the moment?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Status of release 1.8.2??

[Niels Basjes]

Yes, we should avoid this kind of diversion.
My knowledge of C# is however too limited to know of a better solution. I
sure hope someone can tell what the right solution is.

Niels Basjes

On 5 Dec 2016 01:27, "Ryan Blue" <rb...@netflix.com.invalid> wrote:

Thanks for taking the time to put that together. I don't think we should
let the docker image diverge from what is generally used outside. If we
think that we need to update scripts, then we should update docker to
reflect that change, too, so we don't get into a situation where what we
test with (at least some of us) doesn't match a real environment used by
people more familiar with that language.

Does that sound reasonable?

rb

On Thu, Dec 1, 2016 at 10:53 AM, Niels Basjes <ni...@basj.es> wrote:

> What do you think of this very simple attempt?
> https://github.com/apache/avro/pull/153
>
> Niels
>
> On 1 Dec 2016 18:31, "Ryan Blue" <rb...@netflix.com.invalid> wrote:
>
> > I'd like to get a release out as well. I'm happy to roll another RC
soon.
> >
> > Could someone look at the C# issue that was a problem for some people
> > during the last vote? It would be nice to have something that works both
> > inside and outside docker, without detecting whether or not it is
running
> > in docker.
> >
> > rb
> >
> > On Wed, Nov 30, 2016 at 1:42 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > > Hi,
> > >
> > > What is the current status of the new release?
> > > The first RC failed yet I would really like to see it 'out there'.
> > > So what is the status at the moment?
> > >
> > > --
> > > Best regards / Met vriendelijke groeten,
> > >
> > > Niels Basjes
> > >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>



--
Ryan Blue
Software Engineer
Netflix

Re: [VOTE] Release Apache Avro 1.8.2 RC2

[Niels Basjes]

Hi,

NOTE: I'm at the Hadoop Summit in Munchen Germany with limited tools (and
limited power for my laptop) here.

I did a git clone of the avro code based and checked out the project at
commit
6bbcf7658e31cd7e92c459300fd70263b9b89de6

I found a simple problem that needs to be fixed:
Running ./build.sh rat reveals a build failure caused by a missing
copyright comment in
lang/java/avro/src/test/java/org/apache/avro/TestFixed.java

Niels Basjes




On Thu, Apr 6, 2017 at 12:16 AM, Harsha <a...@harsha.io> wrote:

> Sean,
>  Agree on most of points made here. I am new to the avro project but
> also active committer/PMC in other apache projects.
> I understand that contributors can help getting the reviews up to the
> speed but at the end of the day patches need to be merged
> in to make it into the release.
> There are quite a few patches with reviews available but not much activity
> in-terms of merging or any further feedback. I am happy to help
> wherever I can. I’ll open a new thread for further discussion on this.
>
> To get back on track of voting.
>
> 1. Ran unit tests 1.8.2 rc tag
> 2. Verified signature of artifacts
> 3. Tested avro-java module
>
>
> +1 (non-binding).
>
>
> Thanks,
> Harsha
>
> On Apr 5, 2017, 2:29 PM -0700, Sean Busbey <bus...@cloudera.com>, wrote:
> > Personally, I think it's more important that we get back on our
> > footing for release cadence. There are already a ton of things in the
> > 1.8.2 RC. We can parallelize trying to get the other noted items
> > reviewed and then include them in a 1.8.3 release later (say with a
> > goal of next month).
> >
> > I don't want to hijack Suraj's VOTE thread, but the two things we most
> > need right now are 1) a release cadence we can use to set expectations
> > about when something that's been merged will see the light of day and
> > 2) more active reviewers. We're a volunteer run project, so our finite
> > supply of folks times has to be applied to both of these if we want to
> > succeed. If we hold up getting releases out for the work of recovering
> > our review bandwidth we'll perpetually be stuck with no releases.
> >
> > Due to the project's structure, only committers can help out with the
> > release machinery (and thanks a ton for taking this on Suraj!).
> > However, any contributor who has a spare hour here or there can help
> > out with reviews. For example, it takes much less effort for me as a
> > committer to go through patches once I see someone else has already
> > provided feedback, especially if it's someone who's been making a
> > habit of it.
> >
> > On Wed, Apr 5, 2017 at 2:09 PM, Harsha <a...@harsha.io> wrote:
> > > Suraj,
> > > There are further comments on this JIRA https://issues.apache.org/
> jira/browse/AVRO-1885 to include additional patches in the release.
> > > It looks like the patch-merge process is really slow in Avro. But It
> would good to pick the patches raised by other users in the JIRA.
> > > Can we do a cut with those patches in.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Apr 5, 2017, 1:50 PM -0700, suraj acharya <suraj@gmail.com>,
> wrote:
> > > > +1 Non binding
> > > >
> > > >
> > > > -Suraj Acharya
> > > >
> > > > On Wed, Apr 5, 2017 at 3:49 PM, Suraj Acharya <su...@apache.org>
> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I propose the following RC to be released as official Apache Avro
> 1.8.2
> > > > > release.
> > > > >
> > > > > The commit id is 6bbcf7658e31cd7e92c459300fd70263b9b89de6
> > > > > * This corresponds to the tag: release-1.8.2-rc2
> > > > > * https://s.apache.org/avro-1.8.2-rc2
> > > > >
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc2/
> > > > >
> > > > > You can find the KEYS file here:
> > > > > * https://dist.apache.org/repos/dist/release/avro/KEYS
> > > > >
> > > > > Binary artifacts for Java are staged in Nexus here:
> > > > > * https://repository.apache.org/content/repositories/
> orgapacheavro-1010/
> > > > >
> > > > > This release includes:
> > > > > * A spec for single-message Avro encoding and a Java implementation
> > > > > * Java: Bug fixes for Java logical types
> > > > > * Java: Support for Decimal with specific classes
> > > > > * Ruby: A fix for compatibility when using snappy
> > > > > * Python 3: Updated to use the standard module-level logging
> pattern
> > > > > * C++: Support for Boost >= 1.59
> > > > > * And more bug fixes...
> > > > >
> > > > > Please download, verify, and test. This vote will remain open for
> at least
> > > > > 72 hours. Given sufficient votes, I would like to close it on or
> about 9AM
> > > > > PDT on Wednesday, 12 April 2017.
> > > > >
> > > > > [ ] +1 Release this as Apache Avro 1.8.2
> > > > > [ ] +0
> > > > > [ ] -1 Do not release this because...
> > > > >
> > > > >
> > > > > --
> > > > > Suraj Acharya
> > > > >
> >
> >
> >
> > --
> > busbey
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Welcome to our newest PMC member, Suraj Acharya

[Niels Basjes]

Welcome!

On Fri, Apr 14, 2017 at 9:02 PM, Ryan Blue <b...@apache.org> wrote:

> Everyone,
>
> In recognition of the great contributions Suraj has been making lately,
> like stepping up to get the 1.8.2 release out, he's been invited to join
> the Avro project management committee (and accepted).
>
> Congratulations, Suraj! Thanks for your contributions, we look forward to
> more.
>
> rb
>
> --
> Ryan Blue
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Drop hadoop1 support in Avro 1.9

[Niels Basjes]

Yes. Good idea.
+1

On Jul 27, 2017 09:52, "Gabor Szadovszky"  wrote:

> Hi All,
>
> It has already appeared in some discussions/code reviews that we might not
> want to support hadoop1 anymore. As it would be a breaking change I would
> suggest removing the hadoop1 support in the next major release 1.9.0.
> What do you think?
>
> Cheers,
> Gabor
>

Re: [VOTE] Release Apache Avro 1.8.2 RC4

[Niels Basjes]

Downloaded source, ran build test and rat checks.
Everything passed

+1

On Wed, May 10, 2017 at 12:18 AM, Doug Cutting <cutt...@gmail.com> wrote:

> +1
>
> Downloaded source, verified checksum.  All tests passed on Ubuntu 16.04.
>
> Doug
>
> On Mon, May 8, 2017 at 7:23 PM, Suraj Acharya <su...@apache.org> wrote:
> > Hi everyone,
> >
> > I propose the following RC to be released as official Apache Avro 1.8.2
> > release.
> >
> > The commit id is 7f2b8dda4be515a3d1f0b60d5175ee500dbe16e2
> >
> > * This corresponds to the tag: release-1.8.2-rc4
> > * https://s.apache.org/avro-1.8.2-rc4
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.8.2-rc4/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/release/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > * https://repository.apache.org/content/repositories/orgapacheavro-1013
> > * https://repository.apache.org/content/repositories/orgapacheavro-1014
> >
> > This release includes:
> > * A spec for single-message Avro encoding and a Java implementation
> > * Java: Bug fixes for Java logical types
> > * Java: Support for Decimal with specific classes
> > * Ruby: A fix for compatibility when using snappy
> > * Python 3: Updated to use the standard module-level logging pattern
> > * C++: Support for Boost >= 1.59
> > * And more bug fixes...
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close it on or about
> 9AM
> > CDT on Sunday, 14 May 2017.
> >
> > [ ] +1 Release this as Apache Avro 1.8.2
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > --
> > Suraj Acharya
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Why are there so many fixes missing in the 1.8.2 release ?!?

[Niels Basjes]

Hi,

Just now I ran into an obscure problem in the 1.8.2 release I thought I had
already fixed.
Researching the issue I found that some of the fixes that have been
committed to the master over the last few months have not been included in
this release.

I myself made some of those fixes and I would really like to know:  What
went wrong? What did I do wrong?

To get some insight I simply did a diff of the CHANGELOG and found these to
be missing:

AVRO-1993: C++ Byte ordering macro does not work on FreeBSD (thiru)
AVRO-1975: Upgrade java dependencies (gabor)
AVRO-1960: Add log4j properties for avro-tools
AVRO-1748. Add Snappy Compression to C++ DataFile (J. Langley via thiru)
AVRO-1626: C#: Fix Avro.pref build error. (Naruto Takahashi via blue)
AVRO-1966: Java: Fix NPE When copying builder with nullable record.
(Niels Basjes)
AVRO-1967: Java: Fix NPE when calling getXyzBuilder on instance where
the xyz is null (Niels Basjes)
AVRO-1970: Java: Flaky test: TestInputBytes. (Gabor Szadovszky via tomwhite)
AVRO-1881: Java: Avro (Java) Memory Leak when reusing JsonDecoder
instance. (Nandor Kollar via tomwhite)
AVRO-1954: Java: Schema.Field.defaultVal() generates: Unknown datum
type (Nandor Kollar via tomwhite)
AVRO-1930: JsonParser doesn't handle integer scientific notation
(Pietro Cerutti via thiru)
AVRO-1912: C++ Resolving Decoding doesn't work if element removed from
record in array. (via thiru)
AVRO-1866. JsonNullFormatter fwd-declared as class, defined as struct
( Pietro Cerutti via thiru)
AVRO-1750. GenericDatum API behavior breaking change (thiru)
AVRO-1995: JSON Parser does not properly check current state (Victor
Mota via thiru)
AVRO-1216. Setting precision for the output stream (John McClean via thiru)
AVRO-1937: C++ generator for recursive structure crashes (thiru)
AVRO-1892. C++ library cannot parse unions with default values (Hua
Zhang via thiru)
AVRO-1994. C++ Code Generator Generates Invalid Code if Field is of
type Null (Darryl Green via thiru)
AVRO-1997. Avro Field.defaultVal broken for Fixed fields. (Zoltan
Farkasi via thiru)
AVRO-1838: Java: Update checkstyle to catch trailing whitespace.
(nielsbasjes via blue)


Since several of these are NPE/crashes area I think we should consider
getting 1.8.3 out quicker than what we would normally do.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [Announce] Release of Avro-1.8.2

[Niels Basjes]

Hi,

I know now that while fixing the bugs I should have committed them to both
the master and the 1.8 branch instead of 'just the master'.
Just diffing the change log of master and the 1.8 branch shows that this
mistake is made quite a large number of times and in both directions.

As a consequence the release notes (obviously based on the Jira tags) shows
many bugs/changes that are not present in the release.

How do we fix this properly?

Niels Basjes


On Thu, Jun 1, 2017 at 3:55 AM, Suraj Acharya <su...@apache.org> wrote:

> It gives me great pleasure to announce the release of Avro 1.8.2!
>
>
> Changes are listed at: https://s.apache.org/avro-release-note-1.8.2
>
>
> This release can be downloaded from:
> https://www.apache.org/dyn/closer.cgi/avro/
>
>
> Java artifacts are available from Maven Central, Ruby artifacts are at
> RubyGems, Python is at PyPI, JS at NPM.
>
>
> Thanks to everyone for contributing and helping out.
>
>
> Suraj Acharya
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Convert to gitbox

[Niels Basjes]

+1

On Aug 29, 2017 18:50, "Gabor Szadovszky"  wrote:

> +1
>
> On Tue, Aug 29, 2017 at 6:43 PM, Ryan Blue 
> wrote:
>
> > +!
> >
> > On Tue, Aug 29, 2017 at 9:29 AM, Sean Busbey  wrote:
> >
> > > Way back in May we had consensus[1] on using gitbox so that we could
> > start
> > > making use of pull requests.
> > >
> > > In the subsequent JIRAs to implement, it became clear that the entry
> bar
> > > to start is convert our repo to use gitbox. this means the writeable
> > > repository will be on github and some committer set up will be
> > needed[2]. I
> > > don't think this got adequately conveyed originally, so I wanted to
> call
> > it
> > > out here.
> > >
> > > Additionally, the infra process for converting to gitbox requires a
> link
> > > to a community consensus thread. While I think we already had that back
> > in
> > > [1], May was a long time ago and some folks might not be watching for
> > > [DISCUSS] threads.
> > >
> > > Please vote:
> > >
> > > +1: we should convert to gitbox and enable pull request merging
> > > -1: we should not convert to gitbox because...
> > >
> > > Vote will be open for at least 72 hours and will be subject to Majority
> > > Approval (3 or more +1s and more +1s than -1s).
> > >
> > > [1]: https://s.apache.org/ieB0
> > > [2]: Committers will need to flag a github account as theirs via the
> > > account linking utility:
> > > https://gitbox.apache.org/setup/
> > >
> > > The git-wip repository will also cease, so folks who currently use it
> > will
> > > need to update their remote url for existing git clones. We can call
> this
> > > out in our contribution docs.
> > >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>

Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

I was going through the codebase and I found that several files are not
Apache licensed.

https://github.com/apache/avro/tree/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/schema

Some of these files do not have a copyright block (fixable), yet some have
this:

/*

* Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
*/

And according to https://www.apache.org/legal/resolved.html#category-x the
LGPL is not allowed to be included.

How do we fix this problem?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Zoltan,

Because the copyright notice now says you own it I guess the best way to
approach this is is when you put up a pull request with all those files
files having a new license header.
That way it is clear that you made the license switch. I think this should
be a separate jira to document this clearly.

What do you guys think about this approach?

@Nandor / Gabor: I'll put up a ticket that we should run rat much more
often (for both 1.8 and master). (i.e. no longer only in separate profile
of maven)

Niels Basjes


On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas <zolyfar...@yahoo.com.invalid
> wrote:

> Hi Niels, the license is a mistake made by me.
> Those files were based from my work on spf4j-avro which is currently dual
> licensed with LGPL and Apache .
>
> We should just replace the license headers with the appropriate Apache
> header.
> Let me know if you need me to do anything.
>
> Thank you
>
> --z
>
> > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > Hi all,
> >
> > I was going through the codebase and I found that several files are not
> > Apache licensed.
> >
> > https://github.com/apache/avro/tree/master/lang/java/
> compiler/src/main/java/org/apache/avro/compiler/schema
> >
> > Some of these files do not have a copyright block (fixable), yet some
> have
> > this:
> >
> > /*
> >
> > * Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
> > *
> > * This library is free software; you can redistribute it and/or
> > * modify it under the terms of the GNU Lesser General Public
> > * License as published by the Free Software Foundation; either
> > * version 2.1 of the License, or (at your option) any later version.
> > *
> > * This library is distributed in the hope that it will be useful,
> > * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > * GNU General Public License for more details.
> > *
> > * You should have received a copy of the GNU Lesser General Public
> > * License along with this program; if not, write to the Free Software
> > * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307,
> USA.
> > */
> >
> > And according to https://www.apache.org/legal/resolved.html#category-x
> the
> > LGPL is not allowed to be included.
> >
> > How do we fix this problem?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

I had a closer look at the code base.

Most important:
1) I have found these files only in the master branch.
2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
present in any of those releases. (
So we're ok on this part.

I have found exactly 2 files with this problem:
./lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitorAction.java
./lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitor.java

I have found 1 additional commit that touches these two files:

https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9

commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
> CommitDate: Sun Apr 30 23:31:29 2017 +0530
> Added more tests and fixed a couple of bugs. Also formatted the code


In both these files the only changes are:
- Removing the author tag
- Whitespace changes.

See:
https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92

@Thiru: To be 100% sure: You agree with changing these two files to the
Apache license?

Niels Basjes


On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org> wrote:

> In addition to Zoltan we'll need to confirm anyone else who has modified
> the files.
>
> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
>
> > Have these files made it into a release?
> >
> > On Dec 13, 2017 10:18, "Niels Basjes" <ni...@basjes.nl> wrote:
> >
> >> Zoltan,
> >>
> >> Because the copyright notice now says you own it I guess the best way to
> >> approach this is is when you put up a pull request with all those files
> >> files having a new license header.
> >> That way it is clear that you made the license switch. I think this
> should
> >> be a separate jira to document this clearly.
> >>
> >> What do you guys think about this approach?
> >>
> >> @Nandor / Gabor: I'll put up a ticket that we should run rat much more
> >> often (for both 1.8 and master). (i.e. no longer only in separate
> profile
> >> of maven)
> >>
> >> Niels Basjes
> >>
> >>
> >> On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas
> >> <zolyfar...@yahoo.com.invalid
> >> > wrote:
> >>
> >> > Hi Niels, the license is a mistake made by me.
> >> > Those files were based from my work on spf4j-avro which is currently
> >> dual
> >> > licensed with LGPL and Apache .
> >> >
> >> > We should just replace the license headers with the appropriate Apache
> >> > header.
> >> > Let me know if you need me to do anything.
> >> >
> >> > Thank you
> >> >
> >> > --z
> >> >
> >> > > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >> > >
> >> > > Hi all,
> >> > >
> >> > > I was going through the codebase and I found that several files are
> >> not
> >> > > Apache licensed.
> >> > >
> >> > > https://github.com/apache/avro/tree/master/lang/java/
> >> > compiler/src/main/java/org/apache/avro/compiler/schema
> >> > >
> >> > > Some of these files do not have a copyright block (fixable), yet
> some
> >> > have
> >> > > this:
> >> > >
> >> > > /*
> >> > >
> >> > > * Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
> >> > > *
> >> > > * This library is free software; you can redistribute it and/or
> >> > > * modify it under the terms of the GNU Lesser General Public
> >> > > * License as published by the Free Software Foundation; either
> >> > > * version 2.1 of the License, or (at your option) any later version.
> >> > > *
> >> > > * This library is distributed in the hope that it will be useful,
> >> > > * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >> > > * GNU General Public License for more details.
> >> > > *
> >> > > * You should have received a copy of the GNU Lesser General Public
> >> > > * License along with this program; if not, write to the Free
> Software
> >> > > * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
> >> 02111-1307,
> >> > USA.
> >> > > */
> >> > >
> >> > > And according to https://www.apache.org/legal/r
> >> esolved.html#category-x
> >> > the
> >> > > LGPL is not allowed to be included.
> >> > >
> >> > > How do we fix this problem?
> >> > >
> >> > > --
> >> > > Best regards / Met vriendelijke groeten,
> >> > >
> >> > > Niels Basjes
> >> >
> >> >
> >>
> >>
> >> --
> >> Best regards / Met vriendelijke groeten,
> >>
> >> Niels Basjes
> >>
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

After we hear back from Thiru I would like Zoltan to fix these 4 files.

lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitorAction.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitor.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/Schemas.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/CloningVisitor.java

See: https://issues.apache.org/jira/browse/AVRO-2118

After those have been fixed we can commit this change (guys, please review
this. Thanks.)
https://issues.apache.org/jira/browse/AVRO-2119

Niels Basjes


On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> I had a closer look at the code base.
>
> Most important:
> 1) I have found these files only in the master branch.
> 2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
> present in any of those releases. (
> So we're ok on this part.
>
> I have found exactly 2 files with this problem:
> ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> schema/SchemaVisitorAction.java
> ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> schema/SchemaVisitor.java
>
> I have found 1 additional commit that touches these two files:
>
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9
>
> commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
>> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
>> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
>> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
>> CommitDate: Sun Apr 30 23:31:29 2017 +0530
>> Added more tests and fixed a couple of bugs. Also formatted the code
>
>
> In both these files the only changes are:
> - Removing the author tag
> - Whitespace changes.
>
> See:
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92
>
> @Thiru: To be 100% sure: You agree with changing these two files to the
> Apache license?
>
> Niels Basjes
>
>
> On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org> wrote:
>
>> In addition to Zoltan we'll need to confirm anyone else who has modified
>> the files.
>>
>> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
>>
>> > Have these files made it into a release?
>> >
>> > On Dec 13, 2017 10:18, "Niels Basjes" <ni...@basjes.nl> wrote:
>> >
>> >> Zoltan,
>> >>
>> >> Because the copyright notice now says you own it I guess the best way
>> to
>> >> approach this is is when you put up a pull request with all those files
>> >> files having a new license header.
>> >> That way it is clear that you made the license switch. I think this
>> should
>> >> be a separate jira to document this clearly.
>> >>
>> >> What do you guys think about this approach?
>> >>
>> >> @Nandor / Gabor: I'll put up a ticket that we should run rat much more
>> >> often (for both 1.8 and master). (i.e. no longer only in separate
>> profile
>> >> of maven)
>> >>
>> >> Niels Basjes
>> >>
>> >>
>> >> On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas
>> >> <zolyfar...@yahoo.com.invalid
>> >> > wrote:
>> >>
>> >> > Hi Niels, the license is a mistake made by me.
>> >> > Those files were based from my work on spf4j-avro which is currently
>> >> dual
>> >> > licensed with LGPL and Apache .
>> >> >
>> >> > We should just replace the license headers with the appropriate
>> Apache
>> >> > header.
>> >> > Let me know if you need me to do anything.
>> >> >
>> >> > Thank you
>> >> >
>> >> > --z
>> >> >
>> >> > > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
>> >> > >
>> >> > > Hi all,
>> >> > >
>> >> > > I was going through the codebase and I found that several files are
>> >> not
>> >> > > Apache licensed.
>> >> > >
>> >> > > https://github.com/apache/avro/tree/master/lang/java/
>> >> > compiler/src/main/java/org/apache/avro/compiler/schema
>> >> &g

Re: [ANNOUNCE] new committer Nándor Kollár

[Niels Basjes]

Welcome !

On Nov 6, 2017 17:04, "Sean Busbey"  wrote:

> Hi Folks!
>
> I'm very pleased to announce that Nándor Kollár has accepted the PMC's
> invitation to become a committer on the Apache Avro project.
>
> Please join me in congratulating Nándor on this recognition of his great
> work thus far in our community!
>
> -busbey
>

Re: [VOTE] Convert to gitbox

[Niels Basjes]

Hi,

I see this transition was started but at this point it does not seem to
work yet.
What is the current status?

Niels

On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:

> +1
>
> On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org> wrote:
>
> > +1
> >
> > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
> > > Way back in May we had consensus[1] on using gitbox so that we could
> > start making use of pull requests.
> > >
> > > In the subsequent JIRAs to implement, it became clear that the entry
> bar
> > to start is convert our repo to use gitbox. this means the writeable
> > repository will be on github and some committer set up will be
> needed[2]. I
> > don't think this got adequately conveyed originally, so I wanted to call
> it
> > out here.
> > >
> > > Additionally, the infra process for converting to gitbox requires a
> link
> > to a community consensus thread. While I think we already had that back
> in
> > [1], May was a long time ago and some folks might not be watching for
> > [DISCUSS] threads.
> > >
> > > Please vote:
> > >
> > > +1: we should convert to gitbox and enable pull request merging
> > > -1: we should not convert to gitbox because...
> > >
> > > Vote will be open for at least 72 hours and will be subject to Majority
> > Approval (3 or more +1s and more +1s than -1s).
> > >
> > > [1]: https://s.apache.org/ieB0
> > > [2]: Committers will need to flag a github account as theirs via the
> > account linking utility:
> > > https://gitbox.apache.org/setup/
> > >
> > > The git-wip repository will also cease, so folks who currently use it
> > will need to update their remote url for existing git clones. We can call
> > this out in our contribution docs.
> > >
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Convert to gitbox

[Niels Basjes]

I have the same problem; I cannot see the ticket so I have no clue what the
current status is.

Niels

On Thu, Dec 7, 2017 at 3:29 AM, Suraj Acharya <su...@apache.org> wrote:

> Ryan had filed an Infra ticket to turn it on.
> I am not sure what the status of that ticket is.
> I don't have view privileges on that ticket.
>
> Suraj
>
> On Wed, Dec 6, 2017 at 3:09 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > I see this transition was started but at this point it does not seem to
> > work yet.
> > What is the current status?
> >
> > Niels
> >
> > On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:
> >
> > > +1
> > >
> > > On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
> > > > > Way back in May we had consensus[1] on using gitbox so that we
> could
> > > > start making use of pull requests.
> > > > >
> > > > > In the subsequent JIRAs to implement, it became clear that the
> entry
> > > bar
> > > > to start is convert our repo to use gitbox. this means the writeable
> > > > repository will be on github and some committer set up will be
> > > needed[2]. I
> > > > don't think this got adequately conveyed originally, so I wanted to
> > call
> > > it
> > > > out here.
> > > > >
> > > > > Additionally, the infra process for converting to gitbox requires a
> > > link
> > > > to a community consensus thread. While I think we already had that
> back
> > > in
> > > > [1], May was a long time ago and some folks might not be watching for
> > > > [DISCUSS] threads.
> > > > >
> > > > > Please vote:
> > > > >
> > > > > +1: we should convert to gitbox and enable pull request merging
> > > > > -1: we should not convert to gitbox because...
> > > > >
> > > > > Vote will be open for at least 72 hours and will be subject to
> > Majority
> > > > Approval (3 or more +1s and more +1s than -1s).
> > > > >
> > > > > [1]: https://s.apache.org/ieB0
> > > > > [2]: Committers will need to flag a github account as theirs via
> the
> > > > account linking utility:
> > > > > https://gitbox.apache.org/setup/
> > > > >
> > > > > The git-wip repository will also cease, so folks who currently use
> it
> > > > will need to update their remote url for existing git clones. We can
> > call
> > > > this out in our contribution docs.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Request for review: AVRO-2195

[Niels Basjes]

Thanks for putting up this contribution.

I put some minor comments in the Jira.

There is one thing I want to mention here also

An important feature of Avro is being crossplatform/crosslanguage so I am
very curious regarding the portability of this compressor across languages.

Simply put: does it exist outside the Java world?

Niels Basjes



On Mon, Jul 2, 2018 at 8:48 AM, Benson Qiu  wrote:

> Hi,
>
> Can a committer please assign AVRO-2195
> <https://issues.apache.org/jira/browse/AVRO-2195> to me and possibly leave
> comments on my patch?
>
> Thank you.
> - Benson
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

AVRO-2117: Code cleanup

[Niels Basjes]

Hi,

Merry X-Mas to you all.


When there are a lot of warnings that are 'trivial' they tend to hide the
warnings that are important.
So in most projects I aim for a 'zero warning' situation so you can spot
any problem in de code base much faster.


I recently put some effort into cleaning up some of the simple and needless
warnings that various tools give on the current codebase. This is not 'all
warnings' but it is a big step in the right direction.

https://issues.apache.org/jira/browse/AVRO-2117
https://github.com/apache/avro/pull/266
To keep an overview of what was changed I used a single commit per "type of
fix".

I would really appreciate feedback from you guys if and how to proceed with
this one.

Thanks.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [RESULT] Convert to gitbox

[Niels Basjes]

FYI:

Just now I put in the request with the Apache Infra team via ticket
https://issues.apache.org/jira/browse/INFRA-15913

Niels

On Thu, Jan 25, 2018 at 9:05 AM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> First of all apologies for the delay.
>
> For the vote to migrate Apache Avro to use gitbox we have seen
> 5   "+1" votes
> 0   "0" votes
> 0   "-1" votes
>
> This means that the vote has PASSED.
>
> Because of the extreme delay since we started the vote I'm waiting an
> additional 24 hours before actually submitting a ticket with INFRA to start
> the migration to gitbox.
>
> So unless we see good reasons not to migrate I'm starting the migration
> tomorrow.
>
> Best regards,
>
> Niels Basjes
> Apache Avro PMC
>
>
> On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:
>
>> +1
>>
>> On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org> wrote:
>>
>> > +1
>> >
>> > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
>> > > Way back in May we had consensus[1] on using gitbox so that we could
>> > start making use of pull requests.
>> > >
>> > > In the subsequent JIRAs to implement, it became clear that the entry
>> bar
>> > to start is convert our repo to use gitbox. this means the writeable
>> > repository will be on github and some committer set up will be
>> needed[2]. I
>> > don't think this got adequately conveyed originally, so I wanted to
>> call it
>> > out here.
>> > >
>> > > Additionally, the infra process for converting to gitbox requires a
>> link
>> > to a community consensus thread. While I think we already had that back
>> in
>> > [1], May was a long time ago and some folks might not be watching for
>> > [DISCUSS] threads.
>> > >
>> > > Please vote:
>> > >
>> > > +1: we should convert to gitbox and enable pull request merging
>> > > -1: we should not convert to gitbox because...
>> > >
>> > > Vote will be open for at least 72 hours and will be subject to
>> Majority
>> > Approval (3 or more +1s and more +1s than -1s).
>> > >
>> > > [1]: https://s.apache.org/ieB0
>> > > [2]: Committers will need to flag a github account as theirs via the
>> > account linking utility:
>> > > https://gitbox.apache.org/setup/
>> > >
>> > > The git-wip repository will also cease, so folks who currently use it
>> > will need to update their remote url for existing git clones. We can
>> call
>> > this out in our contribution docs.
>> > >
>> >
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[RESULT] Convert to gitbox

[Niels Basjes]

Hi all,

First of all apologies for the delay.

For the vote to migrate Apache Avro to use gitbox we have seen
5   "+1" votes
0   "0" votes
0   "-1" votes

This means that the vote has PASSED.

Because of the extreme delay since we started the vote I'm waiting an
additional 24 hours before actually submitting a ticket with INFRA to start
the migration to gitbox.

So unless we see good reasons not to migrate I'm starting the migration
tomorrow.

Best regards,

Niels Basjes
Apache Avro PMC


On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:

> +1
>
> On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org> wrote:
>
> > +1
> >
> > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
> > > Way back in May we had consensus[1] on using gitbox so that we could
> > start making use of pull requests.
> > >
> > > In the subsequent JIRAs to implement, it became clear that the entry
> bar
> > to start is convert our repo to use gitbox. this means the writeable
> > repository will be on github and some committer set up will be
> needed[2]. I
> > don't think this got adequately conveyed originally, so I wanted to call
> it
> > out here.
> > >
> > > Additionally, the infra process for converting to gitbox requires a
> link
> > to a community consensus thread. While I think we already had that back
> in
> > [1], May was a long time ago and some folks might not be watching for
> > [DISCUSS] threads.
> > >
> > > Please vote:
> > >
> > > +1: we should convert to gitbox and enable pull request merging
> > > -1: we should not convert to gitbox because...
> > >
> > > Vote will be open for at least 72 hours and will be subject to Majority
> > Approval (3 or more +1s and more +1s than -1s).
> > >
> > > [1]: https://s.apache.org/ieB0
> > > [2]: Committers will need to flag a github account as theirs via the
> > account linking utility:
> > > https://gitbox.apache.org/setup/
> > >
> > > The git-wip repository will also cease, so folks who currently use it
> > will need to update their remote url for existing git clones. We can call
> > this out in our contribution docs.
> > >
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Travel Assistance applications open.

[Niels Basjes]

—
The Travel Assistance Committee (TAC) are pleased to announce that travel
assistance applications for ApacheCon NA 2018 are now open!

We will be supporting ApacheCon NA Montreal, Canada on 24th - 29th
September 2018

 TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons.
For more info on this years applications and qualifying criteria, please
visit the TAC website at < http://www.apache.org/travel/ <
http://www.apache.org/travel/> >. Applications are now open and will close
1st May.

Important: Applications close on May 1st, 2018. Applicants have until the
closing date above to submit their applications (which should contain as
much supporting material as required to efficiently and accurately process
their request), this will enable TAC to announce successful awards shortly
afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.
We look forward to greeting many of you in Montreal

Kind Regards,
Gavin - (On behalf of the Travel Assistance Committee)
—

Re: Avro serialization in streaming context

[Niels Basjes]

HI,

In Avro 1.8.2 we have added a "Message" format specification.
In the Java classes there are now extra methods generated like
toByteBuffer() and fromByteBuffer() that are intended for exactly the
purpose of using Avro as the serialization specification in a streaming
context.
In fact at bol.com (where I work) we already do this in production.

Please have a look at this because I have the feeling this already does a
lot of what you mention.

Niels Basjes


On Thu, Feb 22, 2018 at 7:18 AM, Commeau, Gabriel <gabriel.comm...@gmail.com
> wrote:

> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Committed.

On Fri, Dec 22, 2017 at 1:27 PM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> Because this is tricky I decided to first let you guys review everything
> before actually committing.
> https://github.com/apache/avro/pull/271
>
> This is the following set of commits:
> AVRO-2118
> 1) Reverting the problematic commits by Thiruvalluvan M G
> 2) Applying Zoltans new license headers
> 3) Me removing the author tag.
> AVRO-2119
> 4) Making sure Apache Rat runs 'always'
>
> - At this point I found the build took ages because it was running rat
> also in all child projects AND failing on many files.
>
> 5) Upgrading Apache Rat and making sure it only runs in the top level
> project.
>
> I would appreciate it if you guys could check if this set of commits is
> correct now.
>
> Thanks.
>
> Niels Basjes
>
>
> On Wed, Dec 20, 2017 at 11:56 PM, Sean Busbey <bus...@apache.org> wrote:
>
>> I think so.
>>
>> It'd be better for someone who hasn't seen the current file to do the
>> code cleanup. But I don't think that's a blocker.
>>
>> On Wed, Dec 20, 2017 at 4:45 PM, Niels Basjes <ni...@basjes.nl> wrote:
>> > Hi all,
>> >
>> > It has been about 1 week now.
>> > Zoltan has put up a pull request correcting the copyright issue very
>> > quickly last week.
>> >
>> > Unfortunately I haven't seen any response from Thiruvalluvan M G <
>> > th...@startsmartlabs.com> yet.
>> > Since his changes were very simple (basic code cleanup) I propose we do
>> the
>> > following so we can cleanup the codebase and close the issues/pull
>> requests
>> > related to all of this.
>> >
>> > 1) I do a git revert on the two affected files and simply rollback his
>> > changes. So "the changes made under the wrong license" are gone.
>> > 2) I merge the fix by Zoltan to correct the licenses.
>> > 3) I cleanup the code (now under the correct license).
>> > 4) I include the related things that have already been checked.
>> >
>> > At this point I am unsure: Is this a valid way (from a licensing
>> > perspective) to fix this?
>> >
>> > Niels
>> >
>> >
>> > On Thu, Dec 14, 2017 at 6:36 PM, Suraj Acharya <su...@apache.org>
>> wrote:
>> >
>> >> As part of the release we do run the rat plugin.
>> >> So it is a highly unlikely this would have been run through a release.
>> >> However, changing it now is a great addition since the release manager
>> has
>> >> to go through the whole license check for all of the files.
>> >> Also, as Sean mentioned anyone who has made any changes to the file
>> after
>> >> the addition of the license will also need to be informed of the
>> change to
>> >> the license.
>> >>
>> >> Thanks
>> >>
>> >> Suraj
>> >>
>> >>
>> >> On Thu, Dec 14, 2017 at 2:49 AM, Niels Basjes <ni...@basjes.nl> wrote:
>> >>
>> >> > Hi all,
>> >> >
>> >> > After we hear back from Thiru I would like Zoltan to fix these 4
>> files.
>> >> >
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/SchemaVisitorAction.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/SchemaVisitor.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/Schemas.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/CloningVisitor.java
>> >> >
>> >> > See: https://issues.apache.org/jira/browse/AVRO-2118
>> >> >
>> >> > After those have been fixed we can commit this change (guys, please
>> >> review
>> >> > this. Thanks.)
>> >> > https://issues.apache.org/jira/browse/AVRO-2119
>> >> >
>> >> > Niels Basjes
>> >> >
>> >> >
>> >> > On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl>
>> wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I had a closer look at the code base.
>> >> > >
>> >> > > Most important:
>> >> > > 1) I have found these files only in the master branch.
>> >> > > 2) I checked 

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

It has been about 1 week now.
Zoltan has put up a pull request correcting the copyright issue very
quickly last week.

Unfortunately I haven't seen any response from Thiruvalluvan M G <
th...@startsmartlabs.com> yet.
Since his changes were very simple (basic code cleanup) I propose we do the
following so we can cleanup the codebase and close the issues/pull requests
related to all of this.

1) I do a git revert on the two affected files and simply rollback his
changes. So "the changes made under the wrong license" are gone.
2) I merge the fix by Zoltan to correct the licenses.
3) I cleanup the code (now under the correct license).
4) I include the related things that have already been checked.

At this point I am unsure: Is this a valid way (from a licensing
perspective) to fix this?

Niels


On Thu, Dec 14, 2017 at 6:36 PM, Suraj Acharya <su...@apache.org> wrote:

> As part of the release we do run the rat plugin.
> So it is a highly unlikely this would have been run through a release.
> However, changing it now is a great addition since the release manager has
> to go through the whole license check for all of the files.
> Also, as Sean mentioned anyone who has made any changes to the file after
> the addition of the license will also need to be informed of the change to
> the license.
>
> Thanks
>
> Suraj
>
>
> On Thu, Dec 14, 2017 at 2:49 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi all,
> >
> > After we hear back from Thiru I would like Zoltan to fix these 4 files.
> >
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/SchemaVisitorAction.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/SchemaVisitor.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/Schemas.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/CloningVisitor.java
> >
> > See: https://issues.apache.org/jira/browse/AVRO-2118
> >
> > After those have been fixed we can commit this change (guys, please
> review
> > this. Thanks.)
> > https://issues.apache.org/jira/browse/AVRO-2119
> >
> > Niels Basjes
> >
> >
> > On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > > Hi all,
> > >
> > > I had a closer look at the code base.
> > >
> > > Most important:
> > > 1) I have found these files only in the master branch.
> > > 2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
> > > present in any of those releases. (
> > > So we're ok on this part.
> > >
> > > I have found exactly 2 files with this problem:
> > > ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > > schema/SchemaVisitorAction.java
> > > ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > > schema/SchemaVisitor.java
> > >
> > > I have found 1 additional commit that touches these two files:
> > >
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9
> > >
> > > commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
> > >> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
> > >> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
> > >> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
> > >> CommitDate: Sun Apr 30 23:31:29 2017 +0530
> > >> Added more tests and fixed a couple of bugs. Also formatted the
> code
> > >
> > >
> > > In both these files the only changes are:
> > > - Removing the author tag
> > > - Whitespace changes.
> > >
> > > See:
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92
> > >
> > > @Thiru: To be 100% sure: You agree with changing these two files to the
> > > Apache license?
> > >
> > > Niels Basjes
> > >
> > >
> > > On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org>
> wrote:
> > >
> > >> In addition to Zoltan we'll need to confirm anyone else who has
> modified
> > >> the files.
> > >>
> > >> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
> > >>
> > >> > Have these files made it into a release?
> >

Re: Jenkins

[Niels Basjes]

I fully agree with automated builds.

I'm just curious what makes Jenkins more appropriate?

Niels
On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko  wrote:
>
> Hi all,
>
> I'd like to revive the Jenkins build on Avro. The last runs are a bit over
> a year old: https://builds.apache.org/view/A/view/Avro/
>
> I've opened a PR based on Apache Yetus
> <https://github.com/apache/avro/pull/344> to configure the precommit task
> to run CI jobs, as Sean Busbey initially suggested. Initially I would like
> to go for a Travis CI, as we have with Apache Airflow, but I think the Jenkins
> of Apache itself <https://builds.apache.org/> might be more appropriate.
>
> The current Yetus scripts that I've written are still a bit crude, but I
> think we should have the automated process from end to end up as soon as
> possible, and then we can start polishing. Would it be possible to get
> access to the Apache Jenkins
> <https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount>,
> or is someone able to set this up?
>
> I would like to configure the Github pull request builder, based on
> the CloudBees
> plugin as suggested on the Apache wiki
> <https://cwiki.apache.org/confluence/display/INFRA/GitHub+Pull+Request+Builder>
> .
>
> Hope to get your feedback on the plan, and hope to move forward :-)
>
> Cheers, Fokko



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Jenkins

[Niels Basjes]

> have a build matrix ... Java 10 and 11. ... easier to configure in, ... 
> Travis  ... native support.

This is exactly why I asked about the preference for Jenkins.
Listening to these arguments I would tend to go to Travis as it
integrates fine with Github and is already used by many Apache
projects.

Niels
On Fri, Oct 12, 2018 at 7:16 AM Driesprong, Fokko  wrote:
>
> Thanks for the reply Niels.
>
> The only thing that makes Jenkins more appropriate right now is that Apache
> offers Jenkins hosts For now we only build against Java8. But in the future
> it might also make sense to have a build matrix to build also against Java
> 10 and 11. That would be easier to configure in, for example, Travis since
> there is native support. Personally, as long it has proper Github
> integration (support for PR's and easy links to the build logs), I don't
> really care which CI service we're using. Since we rely on Docker and
> Yetus, we're not really locked in to a specific service.
>
> Cheers, Fokko
>
>
>
>
> Op do 11 okt. 2018 om 21:53 schreef Niels Basjes :
>
> > I fully agree with automated builds.
> >
> > I'm just curious what makes Jenkins more appropriate?
> >
> > Niels
> > On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to revive the Jenkins build on Avro. The last runs are a bit
> > over
> > > a year old: https://builds.apache.org/view/A/view/Avro/
> > >
> > > I've opened a PR based on Apache Yetus
> > > <https://github.com/apache/avro/pull/344> to configure the precommit
> > task
> > > to run CI jobs, as Sean Busbey initially suggested. Initially I would
> > like
> > > to go for a Travis CI, as we have with Apache Airflow, but I think the
> > Jenkins
> > > of Apache itself <https://builds.apache.org/> might be more appropriate.
> > >
> > > The current Yetus scripts that I've written are still a bit crude, but I
> > > think we should have the automated process from end to end up as soon as
> > > possible, and then we can start polishing. Would it be possible to get
> > > access to the Apache Jenkins
> > > <
> > https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount
> > >,
> > > or is someone able to set this up?
> > >
> > > I would like to configure the Github pull request builder, based on
> > > the CloudBees
> > > plugin as suggested on the Apache wiki
> > > <
> > https://cwiki.apache.org/confluence/display/INFRA/GitHub+Pull+Request+Builder
> > >
> > > .
> > >
> > > Hope to get your feedback on the plan, and hope to move forward :-)
> > >
> > > Cheers, Fokko
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Merge strategy

[Niels Basjes]

In general I'm for Squashing  +1.

In some exceptional cases I would have a small number of commits to
facilitate easier reviewing (like in AVRO-2117).

Niels.
On Thu, Oct 25, 2018 at 7:31 PM Doug Cutting  wrote:
>
> I'm +1 for squashing.
>
> Doug
>
> On Tue, Oct 23, 2018 at 2:35 PM Michael A. Smith 
> wrote:
>
> > I’m generally in favor of linear commits iff the committers can be trusted
> > to follow good git hygiene in general, like with small, proprietary
> > projects with strong leadership.
> >
> > With Avro, with its variegated multilang repo and commits coming from the
> > community at large, I’m not sure if linear-rebase or squash is a great
> > idea.
> >
> > If you think we can put an automated check in place to make sure commits in
> > PRs follow good hygiene by always having ticket tags, signed commits,
> > “real” authors, or whatever criteria makes sense, then I’m all for it.
> >
> > Cheers,
> > Michael Smith/kojiromike
> >
> > On Tue, Oct 23, 2018 at 17:19 Driesprong, Fokko 
> > wrote:
> >
> > > Hi all,
> > >
> > > Allow me to start the great debate.
> > >
> > > [image: image.png]
> > >
> > > I would like to get the Avro's dev-community's opinion on the merge
> > > strategy of pull requests. By default Github supports three options:
> > > - Create a merge commit
> > > - Squash and merge
> > > - Rebase and merge
> > > https://help.github.com/articles/about-merge-methods-on-github/
> > >
> > > Currently I see the PR's being merged into master using the merge commit
> > > (with some exceptions). From my perspective this has two major
> > > disadvantages:
> > > - It will add ugly merge commits on top of master, which don't add any
> > > value.
> > > - The git commit tree history isn't lineair. For example, a PR that has
> > > been merged recently <https://github.com/apache/avro/pull/270>, added a
> > > commit back in 19 dec 2017.
> > > - It makes it tricky to revert commits, since a PR merged using a merge
> > > commit might add one or more commits over the history of master.
> > >
> > > Therefore I would only allow two merge strategies:
> > > - Squash and merge: This will squash all the commits of the PR into one
> > > single commit, and push it on top of master. This should be the default
> > > strategy.
> > > - Rebase and merge: This can be used for very big PR's, in which you
> > don't
> > > want to squash the original.
> > >
> > > Github allows us to disable merge-options. My suggestion would be to
> > > disable the merge commit option, but I'd like to get the community's
> > > opinion on it.
> > >
> > > Cheers, Fokko
> > >
> > >
> > >
> >



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Cleaning up the logo?

[Niels Basjes]

Yes very nice indeed.
Also having a vector format really helps in looking good in presentations
at conferences.

So +1 from me.

Niels Basjes

On Sun, Sep 2, 2018 at 7:54 PM Doug Cutting  wrote:

> +1 looks great!
>
> Thanks,
>
> Doug
>
> On Wed, Aug 29, 2018, 10:54 AM Ryan Blue 
> wrote:
>
> > I think it looks great! The new one has my +1.
> >
> > On Wed, Aug 29, 2018 at 9:26 AM Bridger Howell  wrote:
> >
> > > I think this proposed logo looks better than the current one. Any
> > official
> > > avro members have an opinion?
> > >
> > > On Tue, Aug 28, 2018 at 11:48 AM Daniel Gruno 
> > > wrote:
> > >
> > > > Hi Avro folks!
> > > >
> > > > I was wondering if it's worth cleaning up the Avro logo, making it a
> > tad
> > > > sharper?
> > > > What I'm proposing is retracing the logo, and I've made a proposal
> that
> > > is
> > > > available at:
> > > > http://www.apache.org/logos/res/avro/avro-proposed.svg
> > > >
> > > > If this sounds like a good idea, you are free to use the proposed
> > change
> > > > (I hereby license it under ALv2), the decision is of course yours :)
> If
> > > you
> > > > do decide to change it, do let me know or commit it to the comdev
> logo
> > > > central (see http://www.apache.org/logos/about.html for
> instructions)
> > > >
> > > > With regards,
> > > > Daniel.
> > > >
> > >
> > > --
> > >
> > >
> > > The information contained in this email message is PRIVATE and intended
> > > only for the personal and confidential use of the recipient named
> above.
> > > If
> > > the reader of this message is not the intended recipient or an agent
> > > responsible for delivering it to the intended recipient, you are hereby
> > > notified that you have received this message in error and that any
> > review,
> > > dissemination, distribution or copying of this message is strictly
> > > prohibited.  If you have received this communication in error, please
> > > notify us immediately by email, and delete the original message.
> > >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [ANNOUNCE] Please welcome Ismaël Mejía to the Apache Avro PMC

[Niels Basjes]

Welcome!

On Tue, 11 Jun 2019, 00:01 Brian Lachniet,  wrote:

> Congratulations, Ismaël!
>
> On Mon, Jun 10, 2019 at 5:48 PM Jesse Anderson 
> wrote:
>
>> Congrats!
>>
>> On Mon, Jun 10, 2019, 4:41 PM Sean Busbey  wrote:
>>
>>> Hi folks!
>>>
>>> On behalf of the Apache Avro PMC I am pleased to announce that Ismaël
>>> Mejía has accepted our invitation to become a PMC member. We
>>> appreciate Ismaël stepping up to take more responsibility in the
>>> project.
>>>
>>> Please join me in welcoming Ismaël to the Avro PMC!
>>>
>>> As a reminder, if anyone would like to nominate another person as a
>>> committer or PMC member, even if you are not currently a committer or
>>> PMC member, you can always drop a note to priv...@avro.apache.org to
>>> let us know.
>>>
>>> -busbey
>>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com 
>
>  
>

Re: [ANNOUNCE] Please welcome Fokko Driesprong to the Apache Avro PMC

[Niels Basjes]

Welcome aboard!

On Tue, May 14, 2019, 09:28 Sean Busbey  wrote:

> Hi folks!
>
> On behalf of the Apache Avro PMC I am pleased to announce that Fokko
> Driesprong has accepted our invitation to become a PMC member on the
> Avro project. We appreciate Fokko stepping up to take more
> responsibility in the project.
>
> Please join me in welcoming Fokko to the Avro PMC!
>
>
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or
> PMC member, you can always drop a note to priv...@avro.apache.org to
> let us know.
>
> -busbey
>

Re: supporting a "unit" field for avro schema

[Niels Basjes]

I think we should approach this idea in two parts:

1) The schema. Things like does a different unit mean a different schema
fingerprint even though the bytes remain the same. What does a different
unit mean for schema evolution.

2) Language specifics. Scala has different possibilities than Java.

On Sat, Jun 29, 2019, 18:59 Erik Erlandson  wrote:

> I've been puzzling over what can be done to support this in more
> widely-used languages. The dilemma relative to the current language
> ecosystem is that languages with "modern" type systems (Haskell, Rust,
> Scala, etc) capable of supporting compile-time unit checking, in the
> particular style I've been exploring, are not yet widely used.
>
> With respect to Java, a couple approaches are plausible. One is to enhance
> the language, for example with Java-8 compiler plugins. Another might be to
> implement a unit type system similar to squants
> <https://github.com/typelevel/squants>. This style of unit type system is
> not as flexible or intuitive as what can be done with Scala's latest type
> system sorcery, but it would allow the community to build out a Java native
> type system that supports compile-time unit analysis. And its coverage of
> standard units could be made very good, as squants itself demonstrates.
>
> Python would also be a high-coverage target. I'm even less sure what to do
> for python, as it has no compile-time type checking, but perhaps a
> squants-like python class system would add value. Maybe python's new
> type-hints feature could be leveraged?
>
> Regarding unit expression representation, I'm not unhappy with what I've
> prototyped in `coulomb-avro`, in broad strokes. It has deficiencies that
> would need addressing. It doesn't yet support standard unit abbreviations,
> nor does it understand plurals (e.g. it can parse "second" but not
> "seconds"). Since it's "unit" field is just a custom metadata key, there is
> no enforcement. Parsers are currently instantiated via explicit lists of
> types, which is a property I like, but that may not work well in a world
> where multiple language bindings must be supported in a portable manner.
>
>
>
> On Sat, Jun 29, 2019 at 1:46 AM Niels Basjes  wrote:
>
> > Hi,
> >
> > I attended your talk in Berlin and at the end I thought "too bad this is
> > only Scala".
> >
> > I think it's a good idea to have this in Avro.
> >
> > The details will be tricky: How to encode the units in the schema for
> > example.
> > Especially because of the automatic conversion you spoke about.
> >
> > Niels
> >
> > On Fri, Jun 28, 2019, 23:58 Erik Erlandson  wrote:
> >
> > > Hi Avro community,
> > >
> > > Recently I have been experimenting with avro schema that are extended
> > with
> > > a "unit" field. By "unit" I mean expressions like "second", or
> > "megabyte" -
> > > that is "units of measure".
> > >
> > > I delivered a short talk on my experiments at Berlin Buzzwords, which
> can
> > > be viewed here:
> > > https://www.youtube.com/watch?v=qrQmB2KFKE8
> > > I also wrote a short blog post that may be faster to ingest:
> > >
> > >
> >
> http://erikerlandson.github.io/blog/2019/05/23/unit-types-for-avro-schema-integrating-avro-with-coulomb/
> > >
> > > I received some audience interest in making this concept "first class"
> > for
> > > avro, and so I'm writing to see what the avro dev community thinks of
> the
> > > idea. One issue is that this kind of unit checking is currently only
> > > available for Scala (and specifically scala 2.13 +).
> > >
> > > The Scala project itself is here:
> > > https://github.com/erikerlandson/coulomb
> > >
> > > Cheers,
> > > Erik
> > >
> >
>

Re: supporting a "unit" field for avro schema

[Niels Basjes]

Hi,

I attended your talk in Berlin and at the end I thought "too bad this is
only Scala".

I think it's a good idea to have this in Avro.

The details will be tricky: How to encode the units in the schema for
example.
Especially because of the automatic conversion you spoke about.

Niels

On Fri, Jun 28, 2019, 23:58 Erik Erlandson  wrote:

> Hi Avro community,
>
> Recently I have been experimenting with avro schema that are extended with
> a "unit" field. By "unit" I mean expressions like "second", or "megabyte" -
> that is "units of measure".
>
> I delivered a short talk on my experiments at Berlin Buzzwords, which can
> be viewed here:
> https://www.youtube.com/watch?v=qrQmB2KFKE8
> I also wrote a short blog post that may be faster to ingest:
>
> http://erikerlandson.github.io/blog/2019/05/23/unit-types-for-avro-schema-integrating-avro-with-coulomb/
>
> I received some audience interest in making this concept "first class" for
> avro, and so I'm writing to see what the avro dev community thinks of the
> idea. One issue is that this kind of unit checking is currently only
> available for Scala (and specifically scala 2.13 +).
>
> The Scala project itself is here:
> https://github.com/erikerlandson/coulomb
>
> Cheers,
> Erik
>

Re: [ANNOUNCE] Please welcome Brian Lachniet to the Apache Avro PMC

[Niels Basjes]

Welcome aboard

On Sat, Aug 24, 2019, 17:40 Sean Busbey  wrote:

> Hi folks!
>
> On behalf of the Apache Avro PMC I am pleased to announce that Brian
> Lachniet has accepted our invitation to become a PMC member. We
> appreciate Brian stepping up to take more responsibility in the
> project.
>
> Please join me in welcoming Brian to the Avro PMC!
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or
> PMC member, you can always drop a note to priv...@avro.apache.org to
> let us know.
>
> -busbey
>

Re: [Announce] Please welcome Nándor Kollár to the Apache Avro PMC

[Niels Basjes]

Welcome!

On Fri, Aug 30, 2019, 23:39 Brian Lachniet  wrote:

> Congratulations, Nándor!
>
> On Fri, Aug 30, 2019, 5:37 PM Sean Busbey  wrote:
>
>> Hi folks!
>>
>> On behalf of the Apache Avro PMC I am pleased to announce that Nándor
>> Kollár has accepted our invitation to become a PMC member. We
>> appreciate Nándor stepping up to take more responsibility in the
>> project.
>>
>> Please join me in welcoming Nándor to the Avro PMC!
>>
>> As a reminder, if anyone would like to nominate another person as a
>> committer or PMC member, even if you are not currently a committer or
>> PMC member, you can always drop a note to priv...@avro.apache.org to
>> let us know.
>>
>

Re: [ANNOUNCE] new committer Michael Smith

[Niels Basjes]

Welcome!

On Sun, Oct 20, 2019, 05:20 Sean Busbey  wrote:

> Hi folks!
>
> I'm very pleased to announce that Michael Smith has accepted the PMC's
> invitation to become a committer on the Apache Avro project.
>
> Please join me in congratulating Michael on this recognition of their
> great work thus far in our community.
>

Re: DRAFT ASF Board Report - Apache Avro October 2019

[Niels Basjes]

LGTM.
+1

On Wed, 9 Oct 2019, 16:06 Sean Busbey,  wrote:

> I went ahead and posted the report, with one minor change noted below. I
> can still edit it over the next few days if needed.
>
> On Wed, Oct 9, 2019 at 12:00 AM Sean Busbey  wrote:
>
> >
> > ## Activity:
> > The Avro project slowly is improving on our gains from the last year.
> >
> > We've gotten a fair bit of feedback on the 1.9.0 release and are working
> > towards establishing a regular cadence for getting the result of that
> > feedback
> > into our downstream communities. As noted in the summary, we've gotten
> the
> > follow-up 1.9.1 release out after a period of approximately 4 months.
> >
> >
> I added a parenthetical here to make clear that 1.9.1 is a minor release
> (rather than a maintenance release).
>

Re: [VOTE] Drop Python 2 support

[Niels Basjes]

Yes, fine by me.
If people need an old version of python they can still get an old version
of Avro to help them out.

So for me
+1

Niels

On Tue, 28 Jul 2020, 16:55 Ryan Skraba,  wrote:

> Hello!  Thanks to all the preparatory work you've done over the last
> major releases, it seems alright to me!
>
> [X] +1 Drop support for Python 2 (non-binding)
>
> This leaves the path open to dropping avro-python3 in the next major
> release, which (at this point) should be low impact for anyone who has
> been watching the avro releases.
>
> All my best, Ryan
>
>
> On Tue, Jul 28, 2020 at 3:15 PM Michael A. Smith 
> wrote:
> >
> > Python 2 has been officially obsolete since January 1. I propose we drop
> > support for it.
> >
> > If we decide to continue supporting it, note that pip, the (main) Python
> > package management tool, will itself only support python3 in its next
> > release. So installing avro with python 2 will have to be done more
> > manually after that point.
> >
> > Here are the Apache Voting Rules:
> > http://www.apache.org/foundation/voting.html
> >
> > This vote will remain open for at least 72 hours, more probably until
> next
> > week.
> >
> > [ ] +1 Drop support for Python 2
> > [ ] +0
> > [ ] -1 Continue supporting Python 2 because/until…
>

Re: New committer: Martin Grigorov

[Niels Basjes]

Welcome!

On Wed, Jan 5, 2022, 11:58 Oscar Westra van Holthe - Kind <
os...@westravanholthe.nl> wrote:

> Congratulations Martin! This is well deserved.
>
>
> Oscar
>
> On Tue, 4 Jan 2022 at 17:50, Ryan Skraba  wrote:
>
> > The Project Management Committee (PMC) for Apache Avro
> > has invited Martin Grigorov to become a committer and we are pleased
> > to announce that he has accepted.
> >
> > Over the last few months, he has been active, reliable and easy to
> > work with on PRs and on the mailing list.  His work is of high
> > quality, and he has a breadth of experience in many of the SDK languages.
> > I'm especially keen to point out the work he's been doing on the website!
> >
> > Being a committer enables easier contribution to the
> > project since there is no need to go via the patch
> > submission process. This should enable better productivity.
> >
> > It's great to have you as part of the team, Martin!
> >
>
>
> --
>
> ✉️ Oscar Westra van Holthe - Kind 
>

Re: New website

[Niels Basjes]

Hi,

To me this already looks a lot better than the default website, especially
because now it also supports mobile devices.
The exact look and feel for sites like this is always a discussion thing
as a step 1: I don't have any input on this right now.

What I am thinking about are things like:
Where do we want to host this one?
- On the existing Apache infrastructure?
- Using Github pages? This would make it possible to automatically
regenerate the site on a push to the master/main branch.
- Somewhere else like the netlify this demo is hosted on?

Also:
Do we want this to be a separate repository?
Or do we want this to be part of the main code repository?

Niels



On Thu, Oct 28, 2021 at 10:44 AM Martin Grigorov 
wrote:

> Hi all,
>
> Please check the new candidate for Apache Avro website:
> https://avro-website.netlify.app/
>
> It is based on Hugo and uses Docsy theme.
> Its source code and instructions how to build could be found at
> https://github.com/martin-g/avro-website.
> The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175
>
> I am not web designer, so some things may look not finished.
> I've just copied the HTML content from the old site (
> https://avro.apache.org/) and converted it to Markdown for Hugo.
>
> Any feedback is welcome! With Pull Requests would be awesome!
>
> Regards,
> Martin
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] New Avro logo (final)

[Niels Basjes]

vote, but
> I had
> > > > the
> > > > > > > > opportunity to attend the marketing and publicity office
> hours
> > > > during
> > > > > > the
> > > > > > > > community over Code conference. They had some pretty specific
> > > > > > suggestions
> > > > > > > > (but not requirements of course). I'm on my phone but I just
> want
> > > > to
> > > > > > > > capture them here!
> > > > > > > >
> > > > > > > > 1. Plane going up is better than down!
> > > > > > > > 2. PT Mono is peculiar for our specific word because the r
> has
> > > > serifs
> > > > > > and
> > > > > > > > the others don't.
> > > > > > > > 3. If we use the arrow, it would look great to have the word
> Avro
> > > > in
> > > > > > the
> > > > > > > > plane's dark blue!
> > > > > > > >
> > > > > > > > It was a neat conversation and they said some really nice
> things
> > > > > about
> > > > > > > the
> > > > > > > > initiative!
> > > > > > > >
> > > > > > > > If we do anything about 2 or 3, we can talk about any
> adjustments
> > > > > after
> > > > > > > the
> > > > > > > > vote. To be honest, I'm 100% confident with these current
> logos
> > > and
> > > > > > > leaving
> > > > > > > > any changes to Emma's discretion!
> > > > > > > >
> > > > > > > > All my bets Ryan
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023, 13:08 Ryan Blue 
> wrote:
> > > > > > > >
> > > > > > > > > My vote is for 10.2.
> > > > > > > > >
> > > > > > > > > On Mon, Oct 2, 2023 at 9:02 AM Oscar Westra van Holthe -
> Kind <
> > > > > > > > > os...@westravanholthe.nl> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Since the start, I've been a fan of the pigeon. It looks
> > > good,
> > > > > and
> > > > > > > it's
> > > > > > > > > > very distinctive.
> > > > > > > > > >
> > > > > > > > > > And it's completely different from any send icon, so it
> > > cannot
> > > > be
> > > > > > > > > conflated
> > > > > > > > > > with a message protocol icon.
> > > > > > > > > >
> > > > > > > > > > My vote is for V3.1
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > > Oscar
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > ✉️ Oscar Westra van Holthe - Kind 
> > > > > > > > > >  https://github.com/opwvhk/
> > > > > > > > > >
> > > > > > > > > > Op ma 2 okt. 2023 16:47 schreef Ryan Skraba <
> r...@skraba.com
> > > >:
> > > > > > > > > >
> > > > > > > > > > > I can't stop changing my mind, but I'm going to go with
> > > V9.2
> > > > > -- I
> > > > > > > > > > > really like the pigeon too, but I think the paper plane
> > > fits
> > > > > too
> > > > > > > > > > > nicely to pass up!
> > > > > > > > > > >
> > > > > > > > > > > Excellent work, Emma, and thanks Oscar for calling the
> > > vote.
> > > > I
> > > > > > > think
> > > > > > > > > > > your voting proposal makes sense!
> > > > > > > > > > >
> > > > > > > > > > > Ryan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Oct 2, 2023 at 3:59 PM Ismaël Mejía <
> > > > ieme...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > V9.2
> > > > > > > > > > > >
> > > > > > > > > > > > Like Fokko I was also not so close to the subject
> and it
> > > is
> > > > > > great
> > > > > > > > to
> > > > > > > > > > > > see the final selection.
> > > > > > > > > > > > It looks so great. Thanks for all the work Emma and
> > > > everyone!
> > > > > > > > > > > >
> > > > > > > > > > > > Ismaël
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Oct 2, 2023 at 3:23 PM Fokko Driesprong <
> > > > > > > fo...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hey Oscar,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for raising this vote. I wasn't active in
> the
> > > > > > > conversation
> > > > > > > > > > > > > around the new logo, but I followed it from the
> > > sideline.
> > > > > The
> > > > > > > new
> > > > > > > > > > > logos are
> > > > > > > > > > > > > fantastic. Kudos to Emma.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I really like the pigeon, especially the one with
> the
> > > > > > envelope!
> > > > > > > > My
> > > > > > > > > > vote
> > > > > > > > > > > > > goes to V6.1 if I can't vote on the alternative
> version
> > > > :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers, Fokko
> > > > > > > > > > > > >
> > > > > > > > > > > > > Op ma 2 okt 2023 om 14:14 schreef Oscar Westra van
> > > > Holthe -
> > > > > > > Kind
> > > > > > > > <
> > > > > > > > > > > > > os...@westravanholthe.nl>:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The past month, there has been new progress on
> our
> > > > search
> > > > > > > for a
> > > > > > > > > new
> > > > > > > > > > > logo (
> > > > > > > > > > > > > > AVRO-3554 <
> > > > > https://issues.apache.org/jira/browse/AVRO-3554
> > > > > > > >).
> > > > > > > > > > > Thanks to
> > > > > > > > > > > > > > Emma Kellam and the feedback we've all given, we
> can
> > > > now
> > > > > > > > proceed
> > > > > > > > > to
> > > > > > > > > > > vote on
> > > > > > > > > > > > > > the matter.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > These are the finalists to choose from (if the
> image
> > > is
> > > > > > > > > unreadable,
> > > > > > > > > > > it
> > > > > > > > > > > > > > links to the comment where the finalists are
> > > > announced):
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/browse/AVRO-3554?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17770969#comment-17770969
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So now, we can vote. Because a logo lasts longer
> > > than a
> > > > > > > typical
> > > > > > > > > > > release,
> > > > > > > > > > > > > > the rules deviate from the usual rules
> > > > > > > > > > > > > > <https://www.apache.org/foundation/voting>:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >- the vote lasts 2 weeks
> > > > > > > > > > > > > >- to avoid confusion, please vote with the
> V...
> > > > > > > ifentifier:
> > > > > > > > > > V9.2,
> > > > > > > > > > > V10.2,
> > > > > > > > > > > > > >V11.2, V3.1
> > > > > > > > > > > > > >- the winner is the logo with at least two (2)
> > > votes
> > > > > > more
> > > > > > > > than
> > > > > > > > > > the
> > > > > > > > > > > > > >runner-up
> > > > > > > > > > > > > >- if there is no clear winner, we'll vote by
> > > simple
> > > > > > > majority
> > > > > > > > > > > between the
> > > > > > > > > > > > > >top picks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please vote for your preference!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > Oscar
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > ✉️ Oscar Westra van Holthe - Kind <
> opw...@apache.org
> > > >
> > > > > > > > > > > > > >  https://github.com/opwvhk/
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Ryan Blue
> > > > > > > > > Tabular
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > ✉️ Oscar Westra van Holthe - Kind 
> > > > > > >
> > > > > > > --
> > > > > > > Oscar Westra van Holthe - Kind 
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > David M. Carr
> > da...@carrclan.us
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Changing the Avro logo

[Niels Basjes]

Hi,

I was thinking about some requirements for the logo.

For starters I was wondering where the name Avro actually came from. Is it
an abbreviation? Or does it have a different source (I'll always
like the Hadoop story)?

Note that in addition to the (defunct) Aircraft builder Avro there is also
a Dutch tv broadcaster called Avro ( https://en.wikipedia.org/wiki/AVRO and
https://en.wikipedia.org/wiki/AVROTROS ).

Regarding the logo itself I have these proposal points which are partially
generic and partially my personal opinion:

- Must be unique and not similar to anything "out there"
- Show either something of the name (Like  Beam, Airflow, Ant, Drill) or
what it does (like Mahout)
- Must be available in a vector format (SVG) so it can be scaled with full
accuracy.
- I like a few colors and for practical reasons I propose to limit it to
about 5-10. (so like Iceberg, not like Flex with the gradients)
- I think we will need 2 variants: Square/Circle (social media, favicon,
etc) and rectangle (top banner).

Niels Basjes



On Fri, Jun 24, 2022 at 5:40 PM Ryan Skraba  wrote:

> I think the square (or round) canvas is a really good idea -- it would
> allow an Avro file to have a meaningful icon, for example, in a file
> explorer.
>
> For me, I'm not particularly attached to the current style, or colour,
> and I think giving some freedom to a designer could really bring
> something neat!
>
> Even if I'm fond of our existing logo, I'm always amazed at the
> beautiful logos that have been contributed to different projects
> (https://www.apache.org/logos/)
>
> The keywords I'd like to see evoked are: fast, small (efficient),
> reliable.  Wings fit those keywords but they aren't the only option
> (nor are those the only keywords).
>
> Ryan
>
> On Thu, Jun 23, 2022 at 10:16 AM Oscar Westra van Holthe - Kind
>  wrote:
> >
> > Hi,
> >
> > To change the logo, I propose we gather both requirements and ideas.
> >
> > For example, I'd like a logo that fits better on a square or round
> canvas.
> > Especially at small sizes (icons), the current logo is too flat IMHO.
> >
> > On the same note, I like that the current logo allows varying degrees of
> > detail in the wings (for various sizes).
> >
> > As for ideas, mixing aviation with data flow usually gives you a mix of
> > wings and a wave. But there are already many logos in that theme...
> >
> > These are my 2ct..
> >
> >
> > Oscar Westra van Holthe - Kind 
> >
> > Op wo 22 jun. 2022 21:29 schreef Ryan Skraba :
> >
> > > Hello all!
> > >
> > > Even if our current logo is a nice homage, there are many, many
> > > advantages to using original art for a project logo, and it's
> > > important that there's no question of confusion with other marks.
> > >
> > > If we were to change the Avro logo, how would the community propose we
> > > go about doing it?  What would we like to see?
> > >
> > > Despite the horrible cliché of "paying a designer in exposure", it's
> > > kind of what we all do in Open Source!  I think it would be
> > > appropriate to give a nice shout-out on the project web page.
> > >
> > > My opinion: this doesn't block the 1.11.1 release, but it's something
> > > we should do in parallel, sooner rather than later, in order to be
> > > able to have project materials for ApacheCon in October.
> > >
> > > All my best, Ryan
> > >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Java 8 compatibility?

[Niels Basjes]

Hi,

Because of the harder maintenance I usually do not favor having an LTS
to manage as well.

I like your last sentence more.

Let me summarize how I see this:

   1. The core libraries (i.e. core, compiler,... ) remain Java 8 because
   that is all that is needed... I think.
   2. For each context that is specific for something else (like thrift,
   trevni, mapred, etc.) the java version is highest of java 8 and what that
   context needs.
  1. So for thrift this means Java 11. If a downstream dependency
  cannot handle Java 11 then they are stuck and cannot upgrade Avro anymore
  because they should but cannot upgrade thrift.
  2. We can still build our code into java 8 classes, but because of
  the required context that won't help anyone.

Consequences I see now:

   - We only have 1 release.
  - There is no LTS to maintain separately which makes it all easier.
  - All dependencies are always up to date.
 - If you need the component that works with X then you get the
 integration for the latest version of X.
 - If someone finds a security problem in "the last jdk8 version"
 of a dependency that will no longer get any updates: we don't have any
 problems.
 - Alternatively if we would have an LTS then we WILL have problems
 because there will not be an update available.
  - The build has different JDK settings depending on the module.
  - We can run the entire build under Java 17 (i.e. only once)
 - Which should make this part of the build a lot faster.
 - Toolchains can be used to control the exact JDK needed for the
 build.
  - We can use integration tests with toolchains and something like
  this
  https://www.mojohaus.org/extra-enforcer-rules/enforceBytecodeVersion.html
  to make sure it all works with the correct Java version.


Note: I use this model also in this project of mine where I have a core and
a lot of UDFs in which this core is wrapped.
https://github.com/nielsbasjes/yauaa



On Thu, Mar 9, 2023 at 10:07 PM Ryan Skraba  wrote:

> This seems like something that an LTS version might serve well!  For
> as long as I've used Avro, only the most recent version has active
> support, but we've talked about having more than one version.
>
> If we go this route, I'd be comfortable dropping Java 8 on Avro
> 1.12.x!  I'd also be comfortable only supporting core avro with only
> Java 8.
>
> All my best, Ryan
>
> On Thu, Mar 9, 2023 at 10:28 AM Christophe Le Saëc 
> wrote:
> >
> > Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11
> runtime
> > support is completed."
> > For Hive, it seems to be a question of time
> > <https://issues.apache.org/jira/browse/HIVE-22415> , to remove
> joda.time
> > lib
> >
> > More generally, could we consider that projects which need Java8 or old
> > library have to use old avro version (like beam uses Avro 1.8.2 version
> > <
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527
> >
> > ).
> >
> > So, if we maintain Java8 compatibility for current avro versions, make
> the
> > future "major" version in Java 11 or 17 only is not a pb i think.
> >
> > Best regards,
> > Christophe Le Saëc
> >
> >
> > Le mer. 8 mars 2023 à 23:22, Niels Basjes  a écrit :
> >
> > > Hi,
> > >
> > > I was looking into making the build run in only Java 11 or 17 because
> there
> > > are more and more maven plugins that only run in those.
> > > Newer versions of for example the maven-checkstyle-plugin need Java 11.
> > > https://github.com/apache/avro/pull/2118
> > >
> > > I consider that to be a low risk change because we can make sure that
> the
> > > code still runs under java 8 to support all of the older systems that
> use
> > > Avro.
> > > Having some of the integration tests run via invoker and then use
> > > toolchains is a perfect way to make something like that work.
> > >
> > > Just now I noticed that key dependencies (not plugins: actual
> dependencies)
> > > have started dropping Java 8 support.
> > >
> > > The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
> > > 0.18.1 in /lang/java  https://github.com/apache/avro/pull/2124 ) fails
> > > with:
> > > *[INFO] Restricted to JDK 8 yet
> > > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> > >
> > >
> > > *I have asked the guys at Thrift if they can have the next release be
&g

Re: Does thrift need Java 11?

[Niels Basjes]

Hi,

Thanks for the info.

Niels Basjes


On Thu, Mar 9, 2023 at 4:02 AM Jiayu Liu  wrote:

> Hi Niels,
>
> Yes we moved on to Java 11 last year and configured the build to be
> targeting Java 11 minimal (using gradle release config, the building JDK
> is 17 but the target release is 11), you can find more info here:
>
> https://github.com/apache/thrift/blob/master/lib/java/gradle/sourceConfiguration.gradle#L43
> .
>
> On March 9, 2023, Niels Basjes  wrote:
> > Hi,
> >
> > I'm one of the PMCs of Apache Avro.
> > Dependabot created a merge request to update your library at our end
> > https://github.com/apache/avro/pull/2124
> >
> > This failed because apparently this newer thrift version needs Java
> > 11.
> > We see this message just prior to our build failing
> > *[INFO] Restricted to JDK 8 yet
> > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> >
> > I recognise the need for building under Java 11 or even Java 17
> > because
> > more and more maven/gradle plugins need this.
> > We have the same problem.
> > My question is does the binary release of thrift also need java 11 ?
> > Or can you tweak your build to produce Java 8 compatible binaries ?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[DISCUSS] Java 8 compatibility?

[Niels Basjes]

Hi,

I was looking into making the build run in only Java 11 or 17 because there
are more and more maven plugins that only run in those.
Newer versions of for example the maven-checkstyle-plugin need Java 11.
https://github.com/apache/avro/pull/2118

I consider that to be a low risk change because we can make sure that the
code still runs under java 8 to support all of the older systems that use
Avro.
Having some of the integration tests run via invoker and then use
toolchains is a perfect way to make something like that work.

Just now I noticed that key dependencies (not plugins: actual dependencies)
have started dropping Java 8 support.

The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
0.18.1 in /lang/java  https://github.com/apache/avro/pull/2124 ) fails with:
*[INFO] Restricted to JDK 8 yet
org.apache.thrift:libthrift:jar:0.18.1:compile contains
org/apache/thrift/TBaseProcessor.class targeted to JDK 11*


*I have asked the guys at Thrift if they can have the next release be Java
8 again, let's see if they can.*

Avro is a dependency for many projects of which a significant group (small
projects like Hadoop and Hive) still need Java 8 and cannot build under
Java 11.
So we cannot simply switch to Java 11.

One thing I thought of is to switch "partially".
Something like all modules be Java 8 compatible, except the Thrift one
which is Java 11.

Like to hear your thoughts on this.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Does thrift need Java 11?

[Niels Basjes]

Hi,

I'm one of the PMCs of Apache Avro.
Dependabot created a merge request to update your library at our end
https://github.com/apache/avro/pull/2124

This failed because apparently this newer thrift version needs Java 11.
We see this message just prior to our build failing
*[INFO] Restricted to JDK 8 yet
org.apache.thrift:libthrift:jar:0.18.1:compile contains
org/apache/thrift/TBaseProcessor.class targeted to JDK 11*

I recognise the need for building under Java 11 or even Java 17 because
more and more maven/gradle plugins need this.
We have the same problem.
My question is does the binary release of thrift also need java 11 ?
Or can you tweak your build to produce Java 8 compatible binaries ?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[AVRO-3713] Why does the default values cache need weak keys?

[Niels Basjes]

Hi,

Earlier this week a few colleagues of mine ran into a concurrency problem
in Avro which is a regression of the fix applied in
https://issues.apache.org/jira/browse/AVRO-1760.

I created a new ticket to track this
https://issues.apache.org/jira/browse/AVRO-3713

Summary of the problem:
In 2016 this change was made to solve the concurrency problem (too much
synchronization/locking):
*Original code:*

  private final Map defaultValueCache
= Collections.synchronizedMap(new WeakHashMap());

*New code (using Guava):*

  private final Map defaultValueCache
= new MapMaker().weakKeys().makeMap();

Then in 2018 guava was dropped
*New code:*

  private final Map defaultValueCache
= Collections.synchronizedMap(new WeakHashMap<>());

So we are back where we started and the same problem has returned.

Looking at this last code change I found that when dropping Guava in many
places a ConcurrentHashMap is used.
Only in this place because of the "Weak Keys" a different setup was used.

I looked back into the version history and this code dates back to 2013 (
https://issues.apache.org/jira/browse/AVRO-1228 ) .
I found no explanation about why the weak keys are used (I checked Jira,
Mailing list and the code).

My question is simply: *Why use weak keys? Is that really needed?*

If we do not need it I propose to simply go for a ConcurrentHashMap.
If we do need it I think the fix should probably look something like this:

private final WeakConcurrentMap defaultValueCache = new
WeakConcurrentMap<>();

private static class WeakConcurrentMap {
  ConcurrentHashMap, V> map = new ConcurrentHashMap<>();

  public V get(K key) {
WeakReference weakKey = new WeakReference<>(key);
V value = map.get(weakKey);
if(value != null) {
  return value;
} else {
  map.remove(weakKey);
  return null;
}
  }

  V put(K key, V value) {
return map.put(new WeakReference<>(key), value);
  }
}




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Release Avro 1.11.3

[Niels Basjes]

Yes, +1 from me for having a regression release.

In my opinion this regression fix should also be included in this:
https://issues.apache.org/jira/browse/AVRO-3713

Niels Basjes

On Wed, Jul 12, 2023 at 2:30 PM Christophe Le Saëc 
wrote:

> +1 for me too.
> (*Are votes reserved for commiters or PMC ?*)
>
> Le mer. 12 juil. 2023 à 13:38, Martin Grigorov  a
> écrit :
>
> > +1 from me !
> >
> > On Wed, Jul 12, 2023 at 2:34 PM Ryan Skraba  wrote:
> >
> > > The regression from AVRO-3789 looks like it a sufficient regression to
> > > merit releasing 1.11.3, which should also cover some of the minor
> > > bumps necessary for the Rust SDK.
> > >
> > > Any objection to doing another while the branch-1.11 is still in a
> > > pretty releasable state?
> > >
> > > I would rather prioritize getting this minor release out, over getting
> > > the website prepared for the 1.11.2 release that we just did!
> > >
> > > All my best, Ryan
> > >
> > > [AVRO-3789]: https://issues.apache.org/jira/browse/AVRO-3789
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Curious: what is the advantage of the FieldAccessHandle you wrote over the
(now present) FieldAccessReflect?
i.e. : Why did you create this alternative?

Niels

On Fri, May 5, 2023 at 1:47 PM Christophe Le Saëc 
wrote:

> Hello Niels,
> This is a nice idea !!
> In same spirit, few time ago, i wrote class FieldAccessHandle (see joined
> file), another extension of FieldAccess, that use
> "java.lang.invoke.MethodHandle" instead of Unsafe.
> And change ReflectionUtil class with it
>
> final FieldAccess unsafeAccess;
> if (javaVersion != null && javaVersion.startsWith("1.8")) {
>   unsafeAccess = load("org.apache.avro.reflect.FieldAccessUnsafe",
> FieldAccess.class);
> } else {
>   unsafeAccess = load("org.apache.avro.reflect.FieldAccessHandle",
> FieldAccess.class);
>
> }
>
> Hope it could help
>
> Le ven. 5 mai 2023 à 12:45, Niels Basjes  a écrit :
>
>> Hi,
>>
>> Currently several build plugins cannot be upgraded because the newer
>> versions require Java 11+.
>> So I'm working on this and I have a partially working pull request
>>
>> https://issues.apache.org/jira/browse/AVRO-3716
>> https://github.com/apache/avro/pull/2118
>>
>> One of the things I ran into is that currently the main library MUST be
>> built with JDK 8 because otherwise the code referring to theUnsafe simply
>> won't build.
>> Now there is already special code in place to use a reflection based
>> system
>> that is used if you are not running on Java 8.
>>
>> Since "everyone" I know is already running on Java 11 and 17 I propose to
>> - kick theUnsafe code, making the code in that area a lot simpler.
>> - Build the entire project with JDK 17 (or 11, but I prefer going to the
>> latest LTS version)
>> - Make it still produce Java 8 compliant code where possible (so just
>> about
>> everywhere but not Thrift).
>> - Have checks in place to do a simple basic test using Java 8 (toolchains)
>>
>> One of the effects is that the build on Github will no longer use the
>> "matrix" build and will build the project only once (I expect it to
>> become faster because of that).
>>
>> Your opinions on this?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Yes, the fact that the build passes on JDK 11 is a mystery to me at this
point.
I'll have a look at the links you mentioned.
I also really like your Java 8 effort towards Thrift. Thanks

Niels

On Fri, May 5, 2023 at 11:08 PM Fokko Driesprong  wrote:

> Hi Niels,
>
> Thanks for raising this. I'm having trouble understanding what you're
> running into. Looks like the CI pass on Java 11+
> <
> https://github.com/apache/avro/blob/master/.github/workflows/test-lang-java.yml#L41-L46
> >?
> Are the Unsafe paths not hit by the tests?
>
> If it helps, four years ago, I also did some similar changes in Parquet
> <https://github.com/apache/parquet-mr/pull/654>. I think we can port these
> changes also to Avro. Unfortunately, we still need to support Java8, since
> big
> projects like Hive <https://issues.apache.org/jira/browse/HIVE-22415> are
> still not on Java11. Related to this, I also opened up a PR to bring back
> Java 8 support for Thrift <https://github.com/apache/thrift/pull/2785>, I
> don't think that we can drop this soon.
>
> Kind regards,
> Fokko Driesprong
>
>
> Op vr 5 mei 2023 om 15:08 schreef Christophe Le Saëc :
>
> > As far as i understood, MethodHandle is design to replace Unsafe call
> > <
> >
> http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html
> > >
> > If not, forget my first message.
> >
> > Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
> > os...@westravanholthe.nl> a écrit :
> >
> > > Hi Niels,
> > >
> > > This is a very good plan. Though personally I prefer using the latest
> LTS
> > > version, I think we should require the oldest, actively supported LST
> > > version. For the next 5 months, this means Java 11.
> > >
> > > This link may be of interest here: https://endoflife.date/java
> > >
> > >
> > > Kind regards,
> > > Oscar
> > >
> > > On Fri, 5 May 2023 at 12:44, Niels Basjes  wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently several build plugins cannot be upgraded because the newer
> > > > versions require Java 11+.
> > > > So I'm working on this and I have a partially working pull request
> > > >
> > > > https://issues.apache.org/jira/browse/AVRO-3716
> > > > https://github.com/apache/avro/pull/2118
> > > >
> > > > One of the things I ran into is that currently the main library MUST
> be
> > > > built with JDK 8 because otherwise the code referring to theUnsafe
> > simply
> > > > won't build.
> > > > Now there is already special code in place to use a reflection based
> > > system
> > > > that is used if you are not running on Java 8.
> > > >
> > > > Since "everyone" I know is already running on Java 11 and 17 I
> propose
> > to
> > > > - kick theUnsafe code, making the code in that area a lot simpler.
> > > > - Build the entire project with JDK 17 (or 11, but I prefer going to
> > the
> > > > latest LTS version)
> > > > - Make it still produce Java 8 compliant code where possible (so just
> > > about
> > > > everywhere but not Thrift).
> > > > - Have checks in place to do a simple basic test using Java 8
> > > (toolchains)
> > > >
> > > > One of the effects is that the build on Github will no longer use the
> > > > "matrix" build and will build the project only once (I expect it to
> > > > become faster because of that).
> > > >
> > > > Your opinions on this?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> > >
> > > --
> > >
> > > ✉️ Oscar Westra van Holthe - Kind 
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Currently several build plugins cannot be upgraded because the newer
versions require Java 11+.
So I'm working on this and I have a partially working pull request

https://issues.apache.org/jira/browse/AVRO-3716
https://github.com/apache/avro/pull/2118

One of the things I ran into is that currently the main library MUST be
built with JDK 8 because otherwise the code referring to theUnsafe simply
won't build.
Now there is already special code in place to use a reflection based system
that is used if you are not running on Java 8.

Since "everyone" I know is already running on Java 11 and 17 I propose to
- kick theUnsafe code, making the code in that area a lot simpler.
- Build the entire project with JDK 17 (or 11, but I prefer going to the
latest LTS version)
- Make it still produce Java 8 compliant code where possible (so just about
everywhere but not Thrift).
- Have checks in place to do a simple basic test using Java 8 (toolchains)

One of the effects is that the build on Github will no longer use the
"matrix" build and will build the project only once (I expect it to
become faster because of that).

Your opinions on this?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Flaky Github ARM builds

[Niels Basjes]

We reported it at the same time
https://issues.apache.org/jira/browse/INFRA-25462


On Mon, Feb 5, 2024 at 10:40 AM Martin Grigorov 
wrote:

> https://issues.apache.org/jira/browse/INFRA-25464
>
> On Fri, Feb 2, 2024 at 10:23 AM Niels Basjes  wrote:
>
> > Yes, that sounds indeed like a very good approach.
> > Can you please report this (I don't know where to report this)?
> >
> > Niels
> >
> >
> >
> > On Thu, Feb 1, 2024 at 9:05 PM Martin Grigorov 
> > wrote:
> >
> > > Hi,
> > >
> > > Another option is to report to the Apache Infra team that the ARM64
> > > self-hosted nodes have these communication problems.
> > > The ARM64 self-hosted nodes are an experimental feature provided by the
> > > Infra team. Any feedback would be welcome!
> > >
> > > Martin
> > >
> > > On Thu, Feb 1, 2024 at 5:20 PM Niels Basjes  wrote:
> > >
> > > > Yes, so we are apparently using an ASF hosted server.
> > > >
> > > > As a quick way of handling this I put up this proposed change where
> we
> > > run
> > > > the ARM tests separately.
> > > > https://github.com/apache/avro/pull/2721
> > > >
> > > > That with an updated README we should for now at least have a simple
> > way
> > > to
> > > > see which part went wrong.
> > > >
> > > > WDYT?
> > > >
> > > > Niels
> > > >
> > > >
> > > >
> > > > On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia <
> zcsizma...@gmail.com>
> > > > wrote:
> > > >
> > > > > There are several options to build for ARM using github workflows:
> > > hosted
> > > > > server , qemu and using macos-14 image.
> > > > >
> > > > > 1. Hosted server: dependency on an external server
> > > > > 2. qemu: extremely slow
> > > > > 3. macos14: Transparent usage compared to other run images, eg.
> > > > > ubuntu-latest, windows-latest. Con is that I am not sure if avro
> can
> > be
> > > > > compiled on macos-latest (because of size constraints). There are
> > > > > macos-large and xlarge images, however those are not free.
> > > > >
> > > > > I have put together an example project which builds a simple C#
> > project
> > > > on
> > > > > Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin
> 23.2
> > > > > (arm64). It demonstrates a unified way to build and run a sample
> > > project
> > > > on
> > > > > the 4 above images. Check the Run step output, which shows the OS
> and
> > > > > architecture info.
> > > > >
> > > > > https://github.com/zcsizmadia/test-arm64
> > > > > https://github.com/zcsizmadia/test-arm64/actions
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
> > > > >
> > > > > Maybe something similar could be used for the avro build.
> > > > >
> > > > > Regards,
> > > > > Zoltan
> > > > >
> > > > > On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes 
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I noticed that most of the build failures are currently related
> to
> > > the
> > > > > ARM
> > > > > > build system being flaky.
> > > > > > So we see that the Java build or the C# build has failed ... but
> > > really
> > > > > it
> > > > > > was the ARM server not responding. Not a real error in the code
> > > and/or
> > > > > > build itself.
> > > > > >
> > > > > > How should we handle this?
> > > > > >
> > > > > > Can we ignore it if the build server is dead/slow/not responding?
> > > > > > Or should we simply disable the ARM builds?
> > > > > >
> > > > > > --
> > > > > > Best regards / Met vriendelijke groeten,
> > > > > >
> > > > > > Niels Basjes
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Flaky Github ARM builds

[Niels Basjes]

Hi,

I noticed that most of the build failures are currently related to the ARM
build system being flaky.
So we see that the Java build or the C# build has failed ... but really it
was the ARM server not responding. Not a real error in the code and/or
build itself.

How should we handle this?

Can we ignore it if the build server is dead/slow/not responding?
Or should we simply disable the ARM builds?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Flaky Github ARM builds

[Niels Basjes]

Yes, so we are apparently using an ASF hosted server.

As a quick way of handling this I put up this proposed change where we run
the ARM tests separately.
https://github.com/apache/avro/pull/2721

That with an updated README we should for now at least have a simple way to
see which part went wrong.

WDYT?

Niels



On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia 
wrote:

> There are several options to build for ARM using github workflows: hosted
> server , qemu and using macos-14 image.
>
> 1. Hosted server: dependency on an external server
> 2. qemu: extremely slow
> 3. macos14: Transparent usage compared to other run images, eg.
> ubuntu-latest, windows-latest. Con is that I am not sure if avro can be
> compiled on macos-latest (because of size constraints). There are
> macos-large and xlarge images, however those are not free.
>
> I have put together an example project which builds a simple C# project on
> Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin 23.2
> (arm64). It demonstrates a unified way to build and run a sample project on
> the 4 above images. Check the Run step output, which shows the OS and
> architecture info.
>
> https://github.com/zcsizmadia/test-arm64
> https://github.com/zcsizmadia/test-arm64/actions
>
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
>
> Maybe something similar could be used for the avro build.
>
> Regards,
> Zoltan
>
> On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes  wrote:
>
> > Hi,
> >
> > I noticed that most of the build failures are currently related to the
> ARM
> > build system being flaky.
> > So we see that the Java build or the C# build has failed ... but really
> it
> > was the ARM server not responding. Not a real error in the code and/or
> > build itself.
> >
> > How should we handle this?
> >
> > Can we ignore it if the build server is dead/slow/not responding?
> > Or should we simply disable the ARM builds?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[Java] New build environment requirements

[Niels Basjes]

Hi,

Just a heads up for all who want to build Avro locally.
The build and test environment requirements have been updated.

Now you'll need to have JDK 8, 11, 17 and 21 (yes, all of them) installed
and the correct toolchains.xml must be present. Also the requirements for
Apache Maven has been upgraded to 3.9.6

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Flaky Github ARM builds

[Niels Basjes]

Yes, that sounds indeed like a very good approach.
Can you please report this (I don't know where to report this)?

Niels



On Thu, Feb 1, 2024 at 9:05 PM Martin Grigorov  wrote:

> Hi,
>
> Another option is to report to the Apache Infra team that the ARM64
> self-hosted nodes have these communication problems.
> The ARM64 self-hosted nodes are an experimental feature provided by the
> Infra team. Any feedback would be welcome!
>
> Martin
>
> On Thu, Feb 1, 2024 at 5:20 PM Niels Basjes  wrote:
>
> > Yes, so we are apparently using an ASF hosted server.
> >
> > As a quick way of handling this I put up this proposed change where we
> run
> > the ARM tests separately.
> > https://github.com/apache/avro/pull/2721
> >
> > That with an updated README we should for now at least have a simple way
> to
> > see which part went wrong.
> >
> > WDYT?
> >
> > Niels
> >
> >
> >
> > On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia 
> > wrote:
> >
> > > There are several options to build for ARM using github workflows:
> hosted
> > > server , qemu and using macos-14 image.
> > >
> > > 1. Hosted server: dependency on an external server
> > > 2. qemu: extremely slow
> > > 3. macos14: Transparent usage compared to other run images, eg.
> > > ubuntu-latest, windows-latest. Con is that I am not sure if avro can be
> > > compiled on macos-latest (because of size constraints). There are
> > > macos-large and xlarge images, however those are not free.
> > >
> > > I have put together an example project which builds a simple C# project
> > on
> > > Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin 23.2
> > > (arm64). It demonstrates a unified way to build and run a sample
> project
> > on
> > > the 4 above images. Check the Run step output, which shows the OS and
> > > architecture info.
> > >
> > > https://github.com/zcsizmadia/test-arm64
> > > https://github.com/zcsizmadia/test-arm64/actions
> > >
> > >
> >
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
> > >
> > > Maybe something similar could be used for the avro build.
> > >
> > > Regards,
> > > Zoltan
> > >
> > > On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes  wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed that most of the build failures are currently related to
> the
> > > ARM
> > > > build system being flaky.
> > > > So we see that the Java build or the C# build has failed ... but
> really
> > > it
> > > > was the ARM server not responding. Not a real error in the code
> and/or
> > > > build itself.
> > > >
> > > > How should we handle this?
> > > >
> > > > Can we ignore it if the build server is dead/slow/not responding?
> > > > Or should we simply disable the ARM builds?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Make Apache Avro a proper Java module.

[Niels Basjes]

Yes, Dropping the runtime support for Java 8 is enough of a reason to bump
to AVRO 2.0.0 for me.

On Mon, Jan 8, 2024 at 6:09 PM Stephen Kittelson
 wrote:

> More and more libraries these days are dropping support for JDK 8 (at least
> Spring Boot 3, Jakarta EE 11, Mockito 5, among others), so I personally
> think it would be fine to drop support for JDK 8 in 1.12.0, or maybe even
> bump the Avro release to 2.0.0 with the removal of JDK 8 support?
>
> On Mon, Jan 8, 2024 at 2:38 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I think this is a good idea!
> >
> > Some PRs (mostly by dependabot) are not merged because Avro needs to be
> JDK
> > 8 compatible and the dependencies require a newer JDK...
> > I am not sure whether Avro 1.12.0 still needs to be JDK 8 compatible or
> > not.
> >
> > Martin
> >
> > On Sat, Jan 6, 2024 at 5:55 PM Chad Preisler 
> > wrote:
> >
> > > Hello,
> > >
> > > I'm wondering if there is any interest in making Apache Avro a proper
> > Java
> > > module? The following changes are required.
> > >
> > > - Add or generate the module-info.java file.
> > > - Change the POM file to build a multi-release jar.
> > > - Replace xerial Snappy with Apache commons-compress Snappy (see
> > additional
> > > information below).
> > > - Update dependencies (like slf4) to the current versions.
> > > - Build with newer JDK. I'm using 21.
> > >
> > > Regarding the Snappy compressor, the next version of Apache
> > > commons-compress (1.25.1) can be swapped in for Xerial with no issues.
> > All
> > > of the existing unit tests will work without changes. Xerial is not a
> > > proper Java module at this time, and it uses JNI which could make it
> > tricky
> > > (especially if the goal is to use jlink).
> > >
> > > For me, the motivation here is to use Avro with modularized
> > > applications and custom runtime images using jlink.
> > >
> > > I currently have this working locally, and I can contribute my changes.
> > >
> > > Please let me know what you think.
> > >
> > > Thanks,
> > >
> > > Chad
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Make Apache Avro a proper Java module.

[Niels Basjes]

Hi,

I have been working on this one over the holidays:
https://github.com/apache/avro/pull/2699
I'm now able to build with Java 21 and still maintain backwards
compatibility for the final artifact with Java 8.

This also includes running (almost all) tests for the main avro module
under all JDKs (8, 11, 17 and 21).

>From what I see now this change set would allow updating a lot of things
(dependencies, adding multirelease, etc.) without losing Java 8 support.

There are however a few rough edges; I had to remove the usage Unsafe and I
found that some tests simply fail on newer JDKs because of changes in the
way Java does things (mainly regarding reflection, modules and security).

Note that if we put dropping Java 8 support up for a vote I would +1 it.

Niels Basjes





On Mon, Jan 8, 2024 at 8:39 PM Chad Preisler 
wrote:

> I can do both depending on the timeline for 1.12.0. If you are cutting a
> release this week, that probably won't work. I don't think it will take
> long to switch to JDK 11 as your default, but I'm not familiar with testing
> changes to the github actions. Basically there will be some learning curve
> for me. You may want to consider adding JDK 21 and updating maven to the
> latest also.
>
> I'm happy to help and will have a decent amount of time to work on it this
> month.
>
> Let me know.
>
> On Mon, Jan 8, 2024 at 11:51 AM Ismaël Mejía  wrote:
>
> > +1 to move to Java 11. Most of our downstream projects have moved out of
> > Java 8 and the fact that we are getting behind on our own dependencies
> is a
> > sign of the current state of the ecosystem. So good to move, and no need
> to
> > change the full version. 1.12.0 should be good.
> >
> > We should probably tackle this in two different issues/PRs (1) to change
> > the baseline for development/support to Java 11 (build system, docker
> > image, github actions, enforce plugin, etc)  and (2) for the proper Java
> 11
> > modules. Would you be up to prepare also a PR for (1) if we all agree
> Chad?
> >
> >
> >
> > On Mon, Jan 8, 2024 at 6:08 PM Stephen Kittelson
> >  wrote:
> >
> >> More and more libraries these days are dropping support for JDK 8 (at
> >> least
> >> Spring Boot 3, Jakarta EE 11, Mockito 5, among others), so I personally
> >> think it would be fine to drop support for JDK 8 in 1.12.0, or maybe
> even
> >> bump the Avro release to 2.0.0 with the removal of JDK 8 support?
> >>
> >> On Mon, Jan 8, 2024 at 2:38 AM Martin Grigorov 
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I think this is a good idea!
> >> >
> >> > Some PRs (mostly by dependabot) are not merged because Avro needs to
> be
> >> JDK
> >> > 8 compatible and the dependencies require a newer JDK...
> >> > I am not sure whether Avro 1.12.0 still needs to be JDK 8 compatible
> or
> >> > not.
> >> >
> >> > Martin
> >> >
> >> > On Sat, Jan 6, 2024 at 5:55 PM Chad Preisler  >
> >> > wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > I'm wondering if there is any interest in making Apache Avro a
> proper
> >> > Java
> >> > > module? The following changes are required.
> >> > >
> >> > > - Add or generate the module-info.java file.
> >> > > - Change the POM file to build a multi-release jar.
> >> > > - Replace xerial Snappy with Apache commons-compress Snappy (see
> >> > additional
> >> > > information below).
> >> > > - Update dependencies (like slf4) to the current versions.
> >> > > - Build with newer JDK. I'm using 21.
> >> > >
> >> > > Regarding the Snappy compressor, the next version of Apache
> >> > > commons-compress (1.25.1) can be swapped in for Xerial with no
> issues.
> >> > All
> >> > > of the existing unit tests will work without changes. Xerial is not
> a
> >> > > proper Java module at this time, and it uses JNI which could make it
> >> > tricky
> >> > > (especially if the goal is to use jlink).
> >> > >
> >> > > For me, the motivation here is to use Avro with modularized
> >> > > applications and custom runtime images using jlink.
> >> > >
> >> > > I currently have this working locally, and I can contribute my
> >> changes.
> >> > >
> >> > > Please let me know what you think.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Chad
> >> > >
> >> >
> >>
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Created] (AVRO-1614) Always getting a value...

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-1614:
--

 Summary: Always getting a value...
 Key: AVRO-1614
 URL: https://issues.apache.org/jira/browse/AVRO-1614
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Niels Basjes


Sometimes the Avro structure becomes deeply nested.
If in such a scenario you want to be able to set a specific value deep in this 
tree you want to do this:

public void setSomething(String value) {
myStruct
.getFoo()
.getBar()
.getOne()
.getOther()
.setSomething(value);
}

The 'problem' I ran into is that any of the 4 get methods can return a null 
value so the code I have to write is really huge.
For every step in this method I have to build null checks and create the 
underlying instance if it is null.
I already started writing helper methods to do this for parts of my tree.

To solve this in a way that makes this code readable I came up with the 
following which I want to propose to you guys (before I start working on a 
patch).

My idea is to generate a new 'get' method in addition to the existing normal 
get method for the regular instance of the class.

So in addition to the 

public Foo getFoo() {
return foo;
}

I propose to generate something like this as well in the cases where this is a 
type of structure that you may want to traverse as shown in the example.

public Foo getAlwaysFoo() {
if (foo == null) {
setFoo(Foo.newBuilder().build());
}
return foo;
}

This way the automatically created instance immediately has all the defaults I 
have defined.

Assuming this naming my code will be readable because it will look like this:
public void setSomething(String value) {
myStruct
.getAlwaysFoo()
.getAlwaysBar()
.getAlwaysOne()
.getAlwaysOther()
.setSomething(value);
}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)