Re: Flaky Github ARM builds

[Niels Basjes]

We reported it at the same time
https://issues.apache.org/jira/browse/INFRA-25462


On Mon, Feb 5, 2024 at 10:40 AM Martin Grigorov 
wrote:

> https://issues.apache.org/jira/browse/INFRA-25464
>
> On Fri, Feb 2, 2024 at 10:23 AM Niels Basjes  wrote:
>
> > Yes, that sounds indeed like a very good approach.
> > Can you please report this (I don't know where to report this)?
> >
> > Niels
> >
> >
> >
> > On Thu, Feb 1, 2024 at 9:05 PM Martin Grigorov 
> > wrote:
> >
> > > Hi,
> > >
> > > Another option is to report to the Apache Infra team that the ARM64
> > > self-hosted nodes have these communication problems.
> > > The ARM64 self-hosted nodes are an experimental feature provided by the
> > > Infra team. Any feedback would be welcome!
> > >
> > > Martin
> > >
> > > On Thu, Feb 1, 2024 at 5:20 PM Niels Basjes  wrote:
> > >
> > > > Yes, so we are apparently using an ASF hosted server.
> > > >
> > > > As a quick way of handling this I put up this proposed change where
> we
> > > run
> > > > the ARM tests separately.
> > > > https://github.com/apache/avro/pull/2721
> > > >
> > > > That with an updated README we should for now at least have a simple
> > way
> > > to
> > > > see which part went wrong.
> > > >
> > > > WDYT?
> > > >
> > > > Niels
> > > >
> > > >
> > > >
> > > > On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia <
> zcsizma...@gmail.com>
> > > > wrote:
> > > >
> > > > > There are several options to build for ARM using github workflows:
> > > hosted
> > > > > server , qemu and using macos-14 image.
> > > > >
> > > > > 1. Hosted server: dependency on an external server
> > > > > 2. qemu: extremely slow
> > > > > 3. macos14: Transparent usage compared to other run images, eg.
> > > > > ubuntu-latest, windows-latest. Con is that I am not sure if avro
> can
> > be
> > > > > compiled on macos-latest (because of size constraints). There are
> > > > > macos-large and xlarge images, however those are not free.
> > > > >
> > > > > I have put together an example project which builds a simple C#
> > project
> > > > on
> > > > > Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin
> 23.2
> > > > > (arm64). It demonstrates a unified way to build and run a sample
> > > project
> > > > on
> > > > > the 4 above images. Check the Run step output, which shows the OS
> and
> > > > > architecture info.
> > > > >
> > > > > https://github.com/zcsizmadia/test-arm64
> > > > > https://github.com/zcsizmadia/test-arm64/actions
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
> > > > >
> > > > > Maybe something similar could be used for the avro build.
> > > > >
> > > > > Regards,
> > > > > Zoltan
> > > > >
> > > > > On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes 
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I noticed that most of the build failures are currently related
> to
> > > the
> > > > > ARM
> > > > > > build system being flaky.
> > > > > > So we see that the Java build or the C# build has failed ... but
> > > really
> > > > > it
> > > > > > was the ARM server not responding. Not a real error in the code
> > > and/or
> > > > > > build itself.
> > > > > >
> > > > > > How should we handle this?
> > > > > >
> > > > > > Can we ignore it if the build server is dead/slow/not responding?
> > > > > > Or should we simply disable the ARM builds?
> > > > > >
> > > > > > --
> > > > > > Best regards / Met vriendelijke groeten,
> > > > > >
> > > > > > Niels Basjes
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Flaky Github ARM builds

[Niels Basjes]

Yes, that sounds indeed like a very good approach.
Can you please report this (I don't know where to report this)?

Niels



On Thu, Feb 1, 2024 at 9:05 PM Martin Grigorov  wrote:

> Hi,
>
> Another option is to report to the Apache Infra team that the ARM64
> self-hosted nodes have these communication problems.
> The ARM64 self-hosted nodes are an experimental feature provided by the
> Infra team. Any feedback would be welcome!
>
> Martin
>
> On Thu, Feb 1, 2024 at 5:20 PM Niels Basjes  wrote:
>
> > Yes, so we are apparently using an ASF hosted server.
> >
> > As a quick way of handling this I put up this proposed change where we
> run
> > the ARM tests separately.
> > https://github.com/apache/avro/pull/2721
> >
> > That with an updated README we should for now at least have a simple way
> to
> > see which part went wrong.
> >
> > WDYT?
> >
> > Niels
> >
> >
> >
> > On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia 
> > wrote:
> >
> > > There are several options to build for ARM using github workflows:
> hosted
> > > server , qemu and using macos-14 image.
> > >
> > > 1. Hosted server: dependency on an external server
> > > 2. qemu: extremely slow
> > > 3. macos14: Transparent usage compared to other run images, eg.
> > > ubuntu-latest, windows-latest. Con is that I am not sure if avro can be
> > > compiled on macos-latest (because of size constraints). There are
> > > macos-large and xlarge images, however those are not free.
> > >
> > > I have put together an example project which builds a simple C# project
> > on
> > > Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin 23.2
> > > (arm64). It demonstrates a unified way to build and run a sample
> project
> > on
> > > the 4 above images. Check the Run step output, which shows the OS and
> > > architecture info.
> > >
> > > https://github.com/zcsizmadia/test-arm64
> > > https://github.com/zcsizmadia/test-arm64/actions
> > >
> > >
> >
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
> > >
> > > Maybe something similar could be used for the avro build.
> > >
> > > Regards,
> > > Zoltan
> > >
> > > On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes  wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed that most of the build failures are currently related to
> the
> > > ARM
> > > > build system being flaky.
> > > > So we see that the Java build or the C# build has failed ... but
> really
> > > it
> > > > was the ARM server not responding. Not a real error in the code
> and/or
> > > > build itself.
> > > >
> > > > How should we handle this?
> > > >
> > > > Can we ignore it if the build server is dead/slow/not responding?
> > > > Or should we simply disable the ARM builds?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Flaky Github ARM builds

[Niels Basjes]

Yes, so we are apparently using an ASF hosted server.

As a quick way of handling this I put up this proposed change where we run
the ARM tests separately.
https://github.com/apache/avro/pull/2721

That with an updated README we should for now at least have a simple way to
see which part went wrong.

WDYT?

Niels



On Thu, Feb 1, 2024 at 4:10 PM Zoltan Csizmadia 
wrote:

> There are several options to build for ARM using github workflows: hosted
> server , qemu and using macos-14 image.
>
> 1. Hosted server: dependency on an external server
> 2. qemu: extremely slow
> 3. macos14: Transparent usage compared to other run images, eg.
> ubuntu-latest, windows-latest. Con is that I am not sure if avro can be
> compiled on macos-latest (because of size constraints). There are
> macos-large and xlarge images, however those are not free.
>
> I have put together an example project which builds a simple C# project on
> Ubuntu 22.04 (x64), Windows 10 (x64), Darwin 21.6 (x64), Darwin 23.2
> (arm64). It demonstrates a unified way to build and run a sample project on
> the 4 above images. Check the Run step output, which shows the OS and
> architecture info.
>
> https://github.com/zcsizmadia/test-arm64
> https://github.com/zcsizmadia/test-arm64/actions
>
> https://github.com/zcsizmadia/test-arm64/actions/workflows/test-multi-os.yml
>
> Maybe something similar could be used for the avro build.
>
> Regards,
> Zoltan
>
> On Thu, Feb 1, 2024 at 7:33 AM Niels Basjes  wrote:
>
> > Hi,
> >
> > I noticed that most of the build failures are currently related to the
> ARM
> > build system being flaky.
> > So we see that the Java build or the C# build has failed ... but really
> it
> > was the ARM server not responding. Not a real error in the code and/or
> > build itself.
> >
> > How should we handle this?
> >
> > Can we ignore it if the build server is dead/slow/not responding?
> > Or should we simply disable the ARM builds?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Flaky Github ARM builds

[Niels Basjes]

Hi,

I noticed that most of the build failures are currently related to the ARM
build system being flaky.
So we see that the Java build or the C# build has failed ... but really it
was the ARM server not responding. Not a real error in the code and/or
build itself.

How should we handle this?

Can we ignore it if the build server is dead/slow/not responding?
Or should we simply disable the ARM builds?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[Java] New build environment requirements

[Niels Basjes]

Hi,

Just a heads up for all who want to build Avro locally.
The build and test environment requirements have been updated.

Now you'll need to have JDK 8, 11, 17 and 21 (yes, all of them) installed
and the correct toolchains.xml must be present. Also the requirements for
Apache Maven has been upgraded to 3.9.6

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Make Apache Avro a proper Java module.

[Niels Basjes]

Yes, Dropping the runtime support for Java 8 is enough of a reason to bump
to AVRO 2.0.0 for me.

On Mon, Jan 8, 2024 at 6:09 PM Stephen Kittelson
 wrote:

> More and more libraries these days are dropping support for JDK 8 (at least
> Spring Boot 3, Jakarta EE 11, Mockito 5, among others), so I personally
> think it would be fine to drop support for JDK 8 in 1.12.0, or maybe even
> bump the Avro release to 2.0.0 with the removal of JDK 8 support?
>
> On Mon, Jan 8, 2024 at 2:38 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I think this is a good idea!
> >
> > Some PRs (mostly by dependabot) are not merged because Avro needs to be
> JDK
> > 8 compatible and the dependencies require a newer JDK...
> > I am not sure whether Avro 1.12.0 still needs to be JDK 8 compatible or
> > not.
> >
> > Martin
> >
> > On Sat, Jan 6, 2024 at 5:55 PM Chad Preisler 
> > wrote:
> >
> > > Hello,
> > >
> > > I'm wondering if there is any interest in making Apache Avro a proper
> > Java
> > > module? The following changes are required.
> > >
> > > - Add or generate the module-info.java file.
> > > - Change the POM file to build a multi-release jar.
> > > - Replace xerial Snappy with Apache commons-compress Snappy (see
> > additional
> > > information below).
> > > - Update dependencies (like slf4) to the current versions.
> > > - Build with newer JDK. I'm using 21.
> > >
> > > Regarding the Snappy compressor, the next version of Apache
> > > commons-compress (1.25.1) can be swapped in for Xerial with no issues.
> > All
> > > of the existing unit tests will work without changes. Xerial is not a
> > > proper Java module at this time, and it uses JNI which could make it
> > tricky
> > > (especially if the goal is to use jlink).
> > >
> > > For me, the motivation here is to use Avro with modularized
> > > applications and custom runtime images using jlink.
> > >
> > > I currently have this working locally, and I can contribute my changes.
> > >
> > > Please let me know what you think.
> > >
> > > Thanks,
> > >
> > > Chad
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Make Apache Avro a proper Java module.

[Niels Basjes]

Hi,

I have been working on this one over the holidays:
https://github.com/apache/avro/pull/2699
I'm now able to build with Java 21 and still maintain backwards
compatibility for the final artifact with Java 8.

This also includes running (almost all) tests for the main avro module
under all JDKs (8, 11, 17 and 21).

>From what I see now this change set would allow updating a lot of things
(dependencies, adding multirelease, etc.) without losing Java 8 support.

There are however a few rough edges; I had to remove the usage Unsafe and I
found that some tests simply fail on newer JDKs because of changes in the
way Java does things (mainly regarding reflection, modules and security).

Note that if we put dropping Java 8 support up for a vote I would +1 it.

Niels Basjes





On Mon, Jan 8, 2024 at 8:39 PM Chad Preisler 
wrote:

> I can do both depending on the timeline for 1.12.0. If you are cutting a
> release this week, that probably won't work. I don't think it will take
> long to switch to JDK 11 as your default, but I'm not familiar with testing
> changes to the github actions. Basically there will be some learning curve
> for me. You may want to consider adding JDK 21 and updating maven to the
> latest also.
>
> I'm happy to help and will have a decent amount of time to work on it this
> month.
>
> Let me know.
>
> On Mon, Jan 8, 2024 at 11:51 AM Ismaël Mejía  wrote:
>
> > +1 to move to Java 11. Most of our downstream projects have moved out of
> > Java 8 and the fact that we are getting behind on our own dependencies
> is a
> > sign of the current state of the ecosystem. So good to move, and no need
> to
> > change the full version. 1.12.0 should be good.
> >
> > We should probably tackle this in two different issues/PRs (1) to change
> > the baseline for development/support to Java 11 (build system, docker
> > image, github actions, enforce plugin, etc)  and (2) for the proper Java
> 11
> > modules. Would you be up to prepare also a PR for (1) if we all agree
> Chad?
> >
> >
> >
> > On Mon, Jan 8, 2024 at 6:08 PM Stephen Kittelson
> >  wrote:
> >
> >> More and more libraries these days are dropping support for JDK 8 (at
> >> least
> >> Spring Boot 3, Jakarta EE 11, Mockito 5, among others), so I personally
> >> think it would be fine to drop support for JDK 8 in 1.12.0, or maybe
> even
> >> bump the Avro release to 2.0.0 with the removal of JDK 8 support?
> >>
> >> On Mon, Jan 8, 2024 at 2:38 AM Martin Grigorov 
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I think this is a good idea!
> >> >
> >> > Some PRs (mostly by dependabot) are not merged because Avro needs to
> be
> >> JDK
> >> > 8 compatible and the dependencies require a newer JDK...
> >> > I am not sure whether Avro 1.12.0 still needs to be JDK 8 compatible
> or
> >> > not.
> >> >
> >> > Martin
> >> >
> >> > On Sat, Jan 6, 2024 at 5:55 PM Chad Preisler  >
> >> > wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > I'm wondering if there is any interest in making Apache Avro a
> proper
> >> > Java
> >> > > module? The following changes are required.
> >> > >
> >> > > - Add or generate the module-info.java file.
> >> > > - Change the POM file to build a multi-release jar.
> >> > > - Replace xerial Snappy with Apache commons-compress Snappy (see
> >> > additional
> >> > > information below).
> >> > > - Update dependencies (like slf4) to the current versions.
> >> > > - Build with newer JDK. I'm using 21.
> >> > >
> >> > > Regarding the Snappy compressor, the next version of Apache
> >> > > commons-compress (1.25.1) can be swapped in for Xerial with no
> issues.
> >> > All
> >> > > of the existing unit tests will work without changes. Xerial is not
> a
> >> > > proper Java module at this time, and it uses JNI which could make it
> >> > tricky
> >> > > (especially if the goal is to use jlink).
> >> > >
> >> > > For me, the motivation here is to use Avro with modularized
> >> > > applications and custom runtime images using jlink.
> >> > >
> >> > > I currently have this working locally, and I can contribute my
> >> changes.
> >> > >
> >> > > Please let me know what you think.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Chad
> >> > >
> >> >
> >>
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] New Avro logo (final)

[Niels Basjes]

vote, but
> I had
> > > > the
> > > > > > > > opportunity to attend the marketing and publicity office
> hours
> > > > during
> > > > > > the
> > > > > > > > community over Code conference. They had some pretty specific
> > > > > > suggestions
> > > > > > > > (but not requirements of course). I'm on my phone but I just
> want
> > > > to
> > > > > > > > capture them here!
> > > > > > > >
> > > > > > > > 1. Plane going up is better than down!
> > > > > > > > 2. PT Mono is peculiar for our specific word because the r
> has
> > > > serifs
> > > > > > and
> > > > > > > > the others don't.
> > > > > > > > 3. If we use the arrow, it would look great to have the word
> Avro
> > > > in
> > > > > > the
> > > > > > > > plane's dark blue!
> > > > > > > >
> > > > > > > > It was a neat conversation and they said some really nice
> things
> > > > > about
> > > > > > > the
> > > > > > > > initiative!
> > > > > > > >
> > > > > > > > If we do anything about 2 or 3, we can talk about any
> adjustments
> > > > > after
> > > > > > > the
> > > > > > > > vote. To be honest, I'm 100% confident with these current
> logos
> > > and
> > > > > > > leaving
> > > > > > > > any changes to Emma's discretion!
> > > > > > > >
> > > > > > > > All my bets Ryan
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023, 13:08 Ryan Blue 
> wrote:
> > > > > > > >
> > > > > > > > > My vote is for 10.2.
> > > > > > > > >
> > > > > > > > > On Mon, Oct 2, 2023 at 9:02 AM Oscar Westra van Holthe -
> Kind <
> > > > > > > > > os...@westravanholthe.nl> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Since the start, I've been a fan of the pigeon. It looks
> > > good,
> > > > > and
> > > > > > > it's
> > > > > > > > > > very distinctive.
> > > > > > > > > >
> > > > > > > > > > And it's completely different from any send icon, so it
> > > cannot
> > > > be
> > > > > > > > > conflated
> > > > > > > > > > with a message protocol icon.
> > > > > > > > > >
> > > > > > > > > > My vote is for V3.1
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > > Oscar
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > ✉️ Oscar Westra van Holthe - Kind 
> > > > > > > > > >  https://github.com/opwvhk/
> > > > > > > > > >
> > > > > > > > > > Op ma 2 okt. 2023 16:47 schreef Ryan Skraba <
> r...@skraba.com
> > > >:
> > > > > > > > > >
> > > > > > > > > > > I can't stop changing my mind, but I'm going to go with
> > > V9.2
> > > > > -- I
> > > > > > > > > > > really like the pigeon too, but I think the paper plane
> > > fits
> > > > > too
> > > > > > > > > > > nicely to pass up!
> > > > > > > > > > >
> > > > > > > > > > > Excellent work, Emma, and thanks Oscar for calling the
> > > vote.
> > > > I
> > > > > > > think
> > > > > > > > > > > your voting proposal makes sense!
> > > > > > > > > > >
> > > > > > > > > > > Ryan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Oct 2, 2023 at 3:59 PM Ismaël Mejía <
> > > > ieme...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > V9.2
> > > > > > > > > > > >
> > > > > > > > > > > > Like Fokko I was also not so close to the subject
> and it
> > > is
> > > > > > great
> > > > > > > > to
> > > > > > > > > > > > see the final selection.
> > > > > > > > > > > > It looks so great. Thanks for all the work Emma and
> > > > everyone!
> > > > > > > > > > > >
> > > > > > > > > > > > Ismaël
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Oct 2, 2023 at 3:23 PM Fokko Driesprong <
> > > > > > > fo...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hey Oscar,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for raising this vote. I wasn't active in
> the
> > > > > > > conversation
> > > > > > > > > > > > > around the new logo, but I followed it from the
> > > sideline.
> > > > > The
> > > > > > > new
> > > > > > > > > > > logos are
> > > > > > > > > > > > > fantastic. Kudos to Emma.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I really like the pigeon, especially the one with
> the
> > > > > > envelope!
> > > > > > > > My
> > > > > > > > > > vote
> > > > > > > > > > > > > goes to V6.1 if I can't vote on the alternative
> version
> > > > :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers, Fokko
> > > > > > > > > > > > >
> > > > > > > > > > > > > Op ma 2 okt 2023 om 14:14 schreef Oscar Westra van
> > > > Holthe -
> > > > > > > Kind
> > > > > > > > <
> > > > > > > > > > > > > os...@westravanholthe.nl>:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The past month, there has been new progress on
> our
> > > > search
> > > > > > > for a
> > > > > > > > > new
> > > > > > > > > > > logo (
> > > > > > > > > > > > > > AVRO-3554 <
> > > > > https://issues.apache.org/jira/browse/AVRO-3554
> > > > > > > >).
> > > > > > > > > > > Thanks to
> > > > > > > > > > > > > > Emma Kellam and the feedback we've all given, we
> can
> > > > now
> > > > > > > > proceed
> > > > > > > > > to
> > > > > > > > > > > vote on
> > > > > > > > > > > > > > the matter.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > These are the finalists to choose from (if the
> image
> > > is
> > > > > > > > > unreadable,
> > > > > > > > > > > it
> > > > > > > > > > > > > > links to the comment where the finalists are
> > > > announced):
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/browse/AVRO-3554?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17770969#comment-17770969
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So now, we can vote. Because a logo lasts longer
> > > than a
> > > > > > > typical
> > > > > > > > > > > release,
> > > > > > > > > > > > > > the rules deviate from the usual rules
> > > > > > > > > > > > > > <https://www.apache.org/foundation/voting>:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >- the vote lasts 2 weeks
> > > > > > > > > > > > > >- to avoid confusion, please vote with the
> V...
> > > > > > > ifentifier:
> > > > > > > > > > V9.2,
> > > > > > > > > > > V10.2,
> > > > > > > > > > > > > >V11.2, V3.1
> > > > > > > > > > > > > >- the winner is the logo with at least two (2)
> > > votes
> > > > > > more
> > > > > > > > than
> > > > > > > > > > the
> > > > > > > > > > > > > >runner-up
> > > > > > > > > > > > > >- if there is no clear winner, we'll vote by
> > > simple
> > > > > > > majority
> > > > > > > > > > > between the
> > > > > > > > > > > > > >top picks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please vote for your preference!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Kind regards,
> > > > > > > > > > > > > > Oscar
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > ✉️ Oscar Westra van Holthe - Kind <
> opw...@apache.org
> > > >
> > > > > > > > > > > > > >  https://github.com/opwvhk/
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Ryan Blue
> > > > > > > > > Tabular
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > ✉️ Oscar Westra van Holthe - Kind 
> > > > > > >
> > > > > > > --
> > > > > > > Oscar Westra van Holthe - Kind 
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > David M. Carr
> > da...@carrclan.us
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Release Avro 1.11.3

[Niels Basjes]

Yes, +1 from me for having a regression release.

In my opinion this regression fix should also be included in this:
https://issues.apache.org/jira/browse/AVRO-3713

Niels Basjes

On Wed, Jul 12, 2023 at 2:30 PM Christophe Le Saëc 
wrote:

> +1 for me too.
> (*Are votes reserved for commiters or PMC ?*)
>
> Le mer. 12 juil. 2023 à 13:38, Martin Grigorov  a
> écrit :
>
> > +1 from me !
> >
> > On Wed, Jul 12, 2023 at 2:34 PM Ryan Skraba  wrote:
> >
> > > The regression from AVRO-3789 looks like it a sufficient regression to
> > > merit releasing 1.11.3, which should also cover some of the minor
> > > bumps necessary for the Rust SDK.
> > >
> > > Any objection to doing another while the branch-1.11 is still in a
> > > pretty releasable state?
> > >
> > > I would rather prioritize getting this minor release out, over getting
> > > the website prepared for the 1.11.2 release that we just did!
> > >
> > > All my best, Ryan
> > >
> > > [AVRO-3789]: https://issues.apache.org/jira/browse/AVRO-3789
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Yes, the fact that the build passes on JDK 11 is a mystery to me at this
point.
I'll have a look at the links you mentioned.
I also really like your Java 8 effort towards Thrift. Thanks

Niels

On Fri, May 5, 2023 at 11:08 PM Fokko Driesprong  wrote:

> Hi Niels,
>
> Thanks for raising this. I'm having trouble understanding what you're
> running into. Looks like the CI pass on Java 11+
> <
> https://github.com/apache/avro/blob/master/.github/workflows/test-lang-java.yml#L41-L46
> >?
> Are the Unsafe paths not hit by the tests?
>
> If it helps, four years ago, I also did some similar changes in Parquet
> <https://github.com/apache/parquet-mr/pull/654>. I think we can port these
> changes also to Avro. Unfortunately, we still need to support Java8, since
> big
> projects like Hive <https://issues.apache.org/jira/browse/HIVE-22415> are
> still not on Java11. Related to this, I also opened up a PR to bring back
> Java 8 support for Thrift <https://github.com/apache/thrift/pull/2785>, I
> don't think that we can drop this soon.
>
> Kind regards,
> Fokko Driesprong
>
>
> Op vr 5 mei 2023 om 15:08 schreef Christophe Le Saëc :
>
> > As far as i understood, MethodHandle is design to replace Unsafe call
> > <
> >
> http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html
> > >
> > If not, forget my first message.
> >
> > Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
> > os...@westravanholthe.nl> a écrit :
> >
> > > Hi Niels,
> > >
> > > This is a very good plan. Though personally I prefer using the latest
> LTS
> > > version, I think we should require the oldest, actively supported LST
> > > version. For the next 5 months, this means Java 11.
> > >
> > > This link may be of interest here: https://endoflife.date/java
> > >
> > >
> > > Kind regards,
> > > Oscar
> > >
> > > On Fri, 5 May 2023 at 12:44, Niels Basjes  wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently several build plugins cannot be upgraded because the newer
> > > > versions require Java 11+.
> > > > So I'm working on this and I have a partially working pull request
> > > >
> > > > https://issues.apache.org/jira/browse/AVRO-3716
> > > > https://github.com/apache/avro/pull/2118
> > > >
> > > > One of the things I ran into is that currently the main library MUST
> be
> > > > built with JDK 8 because otherwise the code referring to theUnsafe
> > simply
> > > > won't build.
> > > > Now there is already special code in place to use a reflection based
> > > system
> > > > that is used if you are not running on Java 8.
> > > >
> > > > Since "everyone" I know is already running on Java 11 and 17 I
> propose
> > to
> > > > - kick theUnsafe code, making the code in that area a lot simpler.
> > > > - Build the entire project with JDK 17 (or 11, but I prefer going to
> > the
> > > > latest LTS version)
> > > > - Make it still produce Java 8 compliant code where possible (so just
> > > about
> > > > everywhere but not Thrift).
> > > > - Have checks in place to do a simple basic test using Java 8
> > > (toolchains)
> > > >
> > > > One of the effects is that the build on Github will no longer use the
> > > > "matrix" build and will build the project only once (I expect it to
> > > > become faster because of that).
> > > >
> > > > Your opinions on this?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> > >
> > > --
> > >
> > > ✉️ Oscar Westra van Holthe - Kind 
> > >
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Curious: what is the advantage of the FieldAccessHandle you wrote over the
(now present) FieldAccessReflect?
i.e. : Why did you create this alternative?

Niels

On Fri, May 5, 2023 at 1:47 PM Christophe Le Saëc 
wrote:

> Hello Niels,
> This is a nice idea !!
> In same spirit, few time ago, i wrote class FieldAccessHandle (see joined
> file), another extension of FieldAccess, that use
> "java.lang.invoke.MethodHandle" instead of Unsafe.
> And change ReflectionUtil class with it
>
> final FieldAccess unsafeAccess;
> if (javaVersion != null && javaVersion.startsWith("1.8")) {
>   unsafeAccess = load("org.apache.avro.reflect.FieldAccessUnsafe",
> FieldAccess.class);
> } else {
>   unsafeAccess = load("org.apache.avro.reflect.FieldAccessHandle",
> FieldAccess.class);
>
> }
>
> Hope it could help
>
> Le ven. 5 mai 2023 à 12:45, Niels Basjes  a écrit :
>
>> Hi,
>>
>> Currently several build plugins cannot be upgraded because the newer
>> versions require Java 11+.
>> So I'm working on this and I have a partially working pull request
>>
>> https://issues.apache.org/jira/browse/AVRO-3716
>> https://github.com/apache/avro/pull/2118
>>
>> One of the things I ran into is that currently the main library MUST be
>> built with JDK 8 because otherwise the code referring to theUnsafe simply
>> won't build.
>> Now there is already special code in place to use a reflection based
>> system
>> that is used if you are not running on Java 8.
>>
>> Since "everyone" I know is already running on Java 11 and 17 I propose to
>> - kick theUnsafe code, making the code in that area a lot simpler.
>> - Build the entire project with JDK 17 (or 11, but I prefer going to the
>> latest LTS version)
>> - Make it still produce Java 8 compliant code where possible (so just
>> about
>> everywhere but not Thrift).
>> - Have checks in place to do a simple basic test using Java 8 (toolchains)
>>
>> One of the effects is that the build on Github will no longer use the
>> "matrix" build and will build the project only once (I expect it to
>> become faster because of that).
>>
>> Your opinions on this?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[JAVA] Proposal: Dropping theUnsafe

[Niels Basjes]

Hi,

Currently several build plugins cannot be upgraded because the newer
versions require Java 11+.
So I'm working on this and I have a partially working pull request

https://issues.apache.org/jira/browse/AVRO-3716
https://github.com/apache/avro/pull/2118

One of the things I ran into is that currently the main library MUST be
built with JDK 8 because otherwise the code referring to theUnsafe simply
won't build.
Now there is already special code in place to use a reflection based system
that is used if you are not running on Java 8.

Since "everyone" I know is already running on Java 11 and 17 I propose to
- kick theUnsafe code, making the code in that area a lot simpler.
- Build the entire project with JDK 17 (or 11, but I prefer going to the
latest LTS version)
- Make it still produce Java 8 compliant code where possible (so just about
everywhere but not Thrift).
- Have checks in place to do a simple basic test using Java 8 (toolchains)

One of the effects is that the build on Github will no longer use the
"matrix" build and will build the project only once (I expect it to
become faster because of that).

Your opinions on this?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Does thrift need Java 11?

[Niels Basjes]

Hi,

Thanks for the info.

Niels Basjes


On Thu, Mar 9, 2023 at 4:02 AM Jiayu Liu  wrote:

> Hi Niels,
>
> Yes we moved on to Java 11 last year and configured the build to be
> targeting Java 11 minimal (using gradle release config, the building JDK
> is 17 but the target release is 11), you can find more info here:
>
> https://github.com/apache/thrift/blob/master/lib/java/gradle/sourceConfiguration.gradle#L43
> .
>
> On March 9, 2023, Niels Basjes  wrote:
> > Hi,
> >
> > I'm one of the PMCs of Apache Avro.
> > Dependabot created a merge request to update your library at our end
> > https://github.com/apache/avro/pull/2124
> >
> > This failed because apparently this newer thrift version needs Java
> > 11.
> > We see this message just prior to our build failing
> > *[INFO] Restricted to JDK 8 yet
> > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> >
> > I recognise the need for building under Java 11 or even Java 17
> > because
> > more and more maven/gradle plugins need this.
> > We have the same problem.
> > My question is does the binary release of thrift also need java 11 ?
> > Or can you tweak your build to produce Java 8 compatible binaries ?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [DISCUSS] Java 8 compatibility?

[Niels Basjes]

Hi,

Because of the harder maintenance I usually do not favor having an LTS
to manage as well.

I like your last sentence more.

Let me summarize how I see this:

   1. The core libraries (i.e. core, compiler,... ) remain Java 8 because
   that is all that is needed... I think.
   2. For each context that is specific for something else (like thrift,
   trevni, mapred, etc.) the java version is highest of java 8 and what that
   context needs.
  1. So for thrift this means Java 11. If a downstream dependency
  cannot handle Java 11 then they are stuck and cannot upgrade Avro anymore
  because they should but cannot upgrade thrift.
  2. We can still build our code into java 8 classes, but because of
  the required context that won't help anyone.

Consequences I see now:

   - We only have 1 release.
  - There is no LTS to maintain separately which makes it all easier.
  - All dependencies are always up to date.
 - If you need the component that works with X then you get the
 integration for the latest version of X.
 - If someone finds a security problem in "the last jdk8 version"
 of a dependency that will no longer get any updates: we don't have any
 problems.
 - Alternatively if we would have an LTS then we WILL have problems
 because there will not be an update available.
  - The build has different JDK settings depending on the module.
  - We can run the entire build under Java 17 (i.e. only once)
 - Which should make this part of the build a lot faster.
 - Toolchains can be used to control the exact JDK needed for the
 build.
  - We can use integration tests with toolchains and something like
  this
  https://www.mojohaus.org/extra-enforcer-rules/enforceBytecodeVersion.html
  to make sure it all works with the correct Java version.


Note: I use this model also in this project of mine where I have a core and
a lot of UDFs in which this core is wrapped.
https://github.com/nielsbasjes/yauaa



On Thu, Mar 9, 2023 at 10:07 PM Ryan Skraba  wrote:

> This seems like something that an LTS version might serve well!  For
> as long as I've used Avro, only the most recent version has active
> support, but we've talked about having more than one version.
>
> If we go this route, I'd be comfortable dropping Java 8 on Avro
> 1.12.x!  I'd also be comfortable only supporting core avro with only
> Java 8.
>
> All my best, Ryan
>
> On Thu, Mar 9, 2023 at 10:28 AM Christophe Le Saëc 
> wrote:
> >
> > Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11
> runtime
> > support is completed."
> > For Hive, it seems to be a question of time
> > <https://issues.apache.org/jira/browse/HIVE-22415> , to remove
> joda.time
> > lib
> >
> > More generally, could we consider that projects which need Java8 or old
> > library have to use old avro version (like beam uses Avro 1.8.2 version
> > <
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527
> >
> > ).
> >
> > So, if we maintain Java8 compatibility for current avro versions, make
> the
> > future "major" version in Java 11 or 17 only is not a pb i think.
> >
> > Best regards,
> > Christophe Le Saëc
> >
> >
> > Le mer. 8 mars 2023 à 23:22, Niels Basjes  a écrit :
> >
> > > Hi,
> > >
> > > I was looking into making the build run in only Java 11 or 17 because
> there
> > > are more and more maven plugins that only run in those.
> > > Newer versions of for example the maven-checkstyle-plugin need Java 11.
> > > https://github.com/apache/avro/pull/2118
> > >
> > > I consider that to be a low risk change because we can make sure that
> the
> > > code still runs under java 8 to support all of the older systems that
> use
> > > Avro.
> > > Having some of the integration tests run via invoker and then use
> > > toolchains is a perfect way to make something like that work.
> > >
> > > Just now I noticed that key dependencies (not plugins: actual
> dependencies)
> > > have started dropping Java 8 support.
> > >
> > > The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
> > > 0.18.1 in /lang/java  https://github.com/apache/avro/pull/2124 ) fails
> > > with:
> > > *[INFO] Restricted to JDK 8 yet
> > > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> > >
> > >
> > > *I have asked the guys at Thrift if they can have the next release be
&g

[DISCUSS] Java 8 compatibility?

[Niels Basjes]

Hi,

I was looking into making the build run in only Java 11 or 17 because there
are more and more maven plugins that only run in those.
Newer versions of for example the maven-checkstyle-plugin need Java 11.
https://github.com/apache/avro/pull/2118

I consider that to be a low risk change because we can make sure that the
code still runs under java 8 to support all of the older systems that use
Avro.
Having some of the integration tests run via invoker and then use
toolchains is a perfect way to make something like that work.

Just now I noticed that key dependencies (not plugins: actual dependencies)
have started dropping Java 8 support.

The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
0.18.1 in /lang/java  https://github.com/apache/avro/pull/2124 ) fails with:
*[INFO] Restricted to JDK 8 yet
org.apache.thrift:libthrift:jar:0.18.1:compile contains
org/apache/thrift/TBaseProcessor.class targeted to JDK 11*


*I have asked the guys at Thrift if they can have the next release be Java
8 again, let's see if they can.*

Avro is a dependency for many projects of which a significant group (small
projects like Hadoop and Hive) still need Java 8 and cannot build under
Java 11.
So we cannot simply switch to Java 11.

One thing I thought of is to switch "partially".
Something like all modules be Java 8 compatible, except the Thrift one
which is Java 11.

Like to hear your thoughts on this.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Does thrift need Java 11?

[Niels Basjes]

Hi,

I'm one of the PMCs of Apache Avro.
Dependabot created a merge request to update your library at our end
https://github.com/apache/avro/pull/2124

This failed because apparently this newer thrift version needs Java 11.
We see this message just prior to our build failing
*[INFO] Restricted to JDK 8 yet
org.apache.thrift:libthrift:jar:0.18.1:compile contains
org/apache/thrift/TBaseProcessor.class targeted to JDK 11*

I recognise the need for building under Java 11 or even Java 17 because
more and more maven/gradle plugins need this.
We have the same problem.
My question is does the binary release of thrift also need java 11 ?
Or can you tweak your build to produce Java 8 compatible binaries ?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Created] (AVRO-3718) Flaky NettyServer test

[Niels Basjes (Jira)]

Niels Basjes created AVRO-3718:
--

 Summary: Flaky NettyServer test
 Key: AVRO-3718
 URL: https://issues.apache.org/jira/browse/AVRO-3718
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes


On a regular basis the TestNettyServer.connectionsCount fails in the Github 
Actions CI.

When you rerun the same test is usually passes.
{code:java}
[INFO] --- maven-surefire-plugin:3.0.0-M8:test (default-test) @ avro-ipc-netty 
---
3498[INFO] Tests will run in random order. To reproduce ordering use flag 
-Dsurefire.runOrder.random.seed=371358370820
3499[INFO] Using auto detected provider 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
3500[INFO] 
3501[INFO] ---
3502[INFO]  T E S T S
3503[INFO] ---
3504[INFO] Running org.apache.avro.ipc.netty.TestNettyServer
3505Error:  Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
4.626 s <<< FAILURE! - in org.apache.avro.ipc.netty.TestNettyServer
3506Error:  org.apache.avro.ipc.netty.TestNettyServer.connectionsCount  Time 
elapsed: 0.01 s  <<< FAILURE!
3507org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
3508at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
3509at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
3510at 
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
3511{code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (AVRO-3717) NPE when basic type with Nullable annotation.

[Niels Basjes (Jira)]

Niels Basjes created AVRO-3717:
--

 Summary: NPE when basic type with Nullable annotation.
 Key: AVRO-3717
 URL: https://issues.apache.org/jira/browse/AVRO-3717
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes
Assignee: Niels Basjes


Reported by [https://github.com/horizonzy]

When using a @Nullable annotation on a primitive type there is an NPE in Avro.

Although this is a strange construct it may not give an NPE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (AVRO-3716) Allow building with newer Maven plugins

[Niels Basjes (Jira)]

Niels Basjes created AVRO-3716:
--

 Summary: Allow building with newer Maven plugins
 Key: AVRO-3716
 URL: https://issues.apache.org/jira/browse/AVRO-3716
 Project: Apache Avro
  Issue Type: Improvement
Reporter: Niels Basjes
Assignee: Niels Basjes


Several of the dependabot updates fail on Java 8 because of newer versions of 
plugins needing Java 11 or Java 17 to run.

{code:java}
Warning:  Error injecting: 
org.apache.maven.plugins.checkstyle.CheckstyleViolationCheckMojo 
[206|https://github.com/apache/avro/actions/runs/4197990575/jobs/7281026269#step:7:207]java.lang.UnsupportedClassVersionError:
 com/puppycrawl/tools/checkstyle/api/AuditListener has been compiled by a more 
recent version of the Java Runtime (class file version 55.0), this version of 
the Java Runtime only recognizes class file versions up to 52.0{code}

My proposal is to restructure the java maven build as follows:
# The build requires Java 17 (or 11 if the next statement is incorrect)
## This means all plugins should work again.
# The compiler must create Java 8 compatible classes (source=8, release=8)
# To ensure it actually runs on all Java versions a basic set of tests are 
executed with all LTS Java versions (8, 11 and 17) and the newest intermediate 
(currently 19)
#- Similar to what I have done here 
#-- https://github.com/nielsbasjes/yauaa/blob/main/.github/workflows/build.yml
#-- https://github.com/nielsbasjes/yauaa/tree/main/analyzer/src/it/JavaPlain
#-- https://github.com/nielsbasjes/ToolChainsInCiBuilds

Additional things
- The build environment for developers change which requires updated  
documentation and docker build.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (AVRO-3715) The build consistently fails.

[Niels Basjes (Jira)]

Niels Basjes created AVRO-3715:
--

 Summary: The build consistently fails.
 Key: AVRO-3715
 URL: https://issues.apache.org/jira/browse/AVRO-3715
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes
Assignee: Niels Basjes


At least 3 reasons:
- Oracle Java 18 no longer exists.
- CycloneDX does not generate reproducible BOM
   
https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/doc/BUILDSPEC.md#cyclonedx-does-not-generate-reproducible-bom
- During ./build.sh dist this dependency does not exist 
org.apache.maven.plugin-tools:maven-plugin-tools-javadoc:3.7.0




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[AVRO-3713] Why does the default values cache need weak keys?

[Niels Basjes]

Hi,

Earlier this week a few colleagues of mine ran into a concurrency problem
in Avro which is a regression of the fix applied in
https://issues.apache.org/jira/browse/AVRO-1760.

I created a new ticket to track this
https://issues.apache.org/jira/browse/AVRO-3713

Summary of the problem:
In 2016 this change was made to solve the concurrency problem (too much
synchronization/locking):
*Original code:*

  private final Map defaultValueCache
= Collections.synchronizedMap(new WeakHashMap());

*New code (using Guava):*

  private final Map defaultValueCache
= new MapMaker().weakKeys().makeMap();

Then in 2018 guava was dropped
*New code:*

  private final Map defaultValueCache
= Collections.synchronizedMap(new WeakHashMap<>());

So we are back where we started and the same problem has returned.

Looking at this last code change I found that when dropping Guava in many
places a ConcurrentHashMap is used.
Only in this place because of the "Weak Keys" a different setup was used.

I looked back into the version history and this code dates back to 2013 (
https://issues.apache.org/jira/browse/AVRO-1228 ) .
I found no explanation about why the weak keys are used (I checked Jira,
Mailing list and the code).

My question is simply: *Why use weak keys? Is that really needed?*

If we do not need it I propose to simply go for a ConcurrentHashMap.
If we do need it I think the fix should probably look something like this:

private final WeakConcurrentMap defaultValueCache = new
WeakConcurrentMap<>();

private static class WeakConcurrentMap {
  ConcurrentHashMap, V> map = new ConcurrentHashMap<>();

  public V get(K key) {
WeakReference weakKey = new WeakReference<>(key);
V value = map.get(weakKey);
if(value != null) {
  return value;
} else {
  map.remove(weakKey);
  return null;
}
  }

  V put(K key, V value) {
return map.put(new WeakReference<>(key), value);
  }
}




-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Created] (AVRO-3713) Regression AVRO-1760 : Thread scalability problem with the use of SynchronizedMap

[Niels Basjes (Jira)]

Niels Basjes created AVRO-3713:
--

 Summary: Regression AVRO-1760 : Thread scalability problem with 
the use of SynchronizedMap
 Key: AVRO-3713
 URL: https://issues.apache.org/jira/browse/AVRO-3713
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes


This is a regression of AVRO-1760 which is causing production performance 
problems.

In AVRO-1760 the issue was fixed that there is a scalability problem with AVRO 
being used in multithreaded situations.

The solution that was implemented for this ticket uses a different Map 
implementation (part of Guava) that does not have these locking issues:

[https://github.com/apache/avro/commit/363360229e7ea5f816d87f9d5747dd4494d3b862#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL970-L973]

Investigating the performance of our internal application has made us conclude 
the original problem was reintroduced with this commit:

[https://github.com/apache/avro/commit/c78a3d5ea4e7e511ba9f6182157956db3e65e1a0#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL986-L988]

which seems to be related to this jira AVRO-2265 "Remove Guava as a dependency".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (AVRO-3546) Add the standard apache favicons for the site

[Niels Basjes (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559232#comment-17559232
 ] 

Niels Basjes commented on AVRO-3546:


In the 'copied" snippet I see the hostname has been pasted in ... and the first 
4 have a / too many (it says     /https://    ) .

> Add the standard apache favicons for the site
> -
>
> Key: AVRO-3546
> URL: https://issues.apache.org/jira/browse/AVRO-3546
> Project: Apache Avro
>  Issue Type: Sub-task
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Copied from https://apache.org home page source.
> {code}
>  href="/https://apache.org/favicons/apple-touch-icon-57x57.png;>
>  href="/https://apache.org/favicons/apple-touch-icon-60x60.png;>
>  href="/https://apache.org/favicons/apple-touch-icon-72x72.png;>
>  href="/https://apache.org/favicons/apple-touch-icon-76x76.png;>
>  href="https://apache.org/favicons/apple-touch-icon-114x114.png;>
>  href="https://apache.org/favicons/apple-touch-icon-120x120.png;>
>  href="https://apache.org/favicons/apple-touch-icon-144x144.png;>
>  href="https://apache.org/favicons/apple-touch-icon-152x152.png;>
>  href="https://apache.org/favicons/apple-touch-icon-180x180.png;>
>  href="https://apache.org/favicons/favicon-32x32.png; sizes="32x32">
>  href="https://apache.org/favicons/favicon-194x194.png; sizes="194x194">
>  href="https://apache.org/favicons/favicon-96x96.png; sizes="96x96">
>  href="https://apache.org/favicons/android-chrome-192x192.png; sizes="192x192">
>  href="https://apache.org/favicons/favicon-16x16.png; sizes="16x16">
> https://apache.org/favicons/manifest.json;>
> https://apache.org/favicons/favicon.ico;>
> {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Changing the Avro logo

[Niels Basjes]

Hi,

I was thinking about some requirements for the logo.

For starters I was wondering where the name Avro actually came from. Is it
an abbreviation? Or does it have a different source (I'll always
like the Hadoop story)?

Note that in addition to the (defunct) Aircraft builder Avro there is also
a Dutch tv broadcaster called Avro ( https://en.wikipedia.org/wiki/AVRO and
https://en.wikipedia.org/wiki/AVROTROS ).

Regarding the logo itself I have these proposal points which are partially
generic and partially my personal opinion:

- Must be unique and not similar to anything "out there"
- Show either something of the name (Like  Beam, Airflow, Ant, Drill) or
what it does (like Mahout)
- Must be available in a vector format (SVG) so it can be scaled with full
accuracy.
- I like a few colors and for practical reasons I propose to limit it to
about 5-10. (so like Iceberg, not like Flex with the gradients)
- I think we will need 2 variants: Square/Circle (social media, favicon,
etc) and rectangle (top banner).

Niels Basjes



On Fri, Jun 24, 2022 at 5:40 PM Ryan Skraba  wrote:

> I think the square (or round) canvas is a really good idea -- it would
> allow an Avro file to have a meaningful icon, for example, in a file
> explorer.
>
> For me, I'm not particularly attached to the current style, or colour,
> and I think giving some freedom to a designer could really bring
> something neat!
>
> Even if I'm fond of our existing logo, I'm always amazed at the
> beautiful logos that have been contributed to different projects
> (https://www.apache.org/logos/)
>
> The keywords I'd like to see evoked are: fast, small (efficient),
> reliable.  Wings fit those keywords but they aren't the only option
> (nor are those the only keywords).
>
> Ryan
>
> On Thu, Jun 23, 2022 at 10:16 AM Oscar Westra van Holthe - Kind
>  wrote:
> >
> > Hi,
> >
> > To change the logo, I propose we gather both requirements and ideas.
> >
> > For example, I'd like a logo that fits better on a square or round
> canvas.
> > Especially at small sizes (icons), the current logo is too flat IMHO.
> >
> > On the same note, I like that the current logo allows varying degrees of
> > detail in the wings (for various sizes).
> >
> > As for ideas, mixing aviation with data flow usually gives you a mix of
> > wings and a wave. But there are already many logos in that theme...
> >
> > These are my 2ct..
> >
> >
> > Oscar Westra van Holthe - Kind 
> >
> > Op wo 22 jun. 2022 21:29 schreef Ryan Skraba :
> >
> > > Hello all!
> > >
> > > Even if our current logo is a nice homage, there are many, many
> > > advantages to using original art for a project logo, and it's
> > > important that there's no question of confusion with other marks.
> > >
> > > If we were to change the Avro logo, how would the community propose we
> > > go about doing it?  What would we like to see?
> > >
> > > Despite the horrible cliché of "paying a designer in exposure", it's
> > > kind of what we all do in Open Source!  I think it would be
> > > appropriate to give a nice shout-out on the project web page.
> > >
> > > My opinion: this doesn't block the 1.11.1 release, but it's something
> > > we should do in parallel, sooner rather than later, in order to be
> > > able to have project materials for ApacheCon in October.
> > >
> > > All my best, Ryan
> > >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: New committer: Martin Grigorov

[Niels Basjes]

Welcome!

On Wed, Jan 5, 2022, 11:58 Oscar Westra van Holthe - Kind <
os...@westravanholthe.nl> wrote:

> Congratulations Martin! This is well deserved.
>
>
> Oscar
>
> On Tue, 4 Jan 2022 at 17:50, Ryan Skraba  wrote:
>
> > The Project Management Committee (PMC) for Apache Avro
> > has invited Martin Grigorov to become a committer and we are pleased
> > to announce that he has accepted.
> >
> > Over the last few months, he has been active, reliable and easy to
> > work with on PRs and on the mailing list.  His work is of high
> > quality, and he has a breadth of experience in many of the SDK languages.
> > I'm especially keen to point out the work he's been doing on the website!
> >
> > Being a committer enables easier contribution to the
> > project since there is no need to go via the patch
> > submission process. This should enable better productivity.
> >
> > It's great to have you as part of the team, Martin!
> >
>
>
> --
>
> ✉️ Oscar Westra van Holthe - Kind 
>

Re: New website

[Niels Basjes]

Hi,

To me this already looks a lot better than the default website, especially
because now it also supports mobile devices.
The exact look and feel for sites like this is always a discussion thing
as a step 1: I don't have any input on this right now.

What I am thinking about are things like:
Where do we want to host this one?
- On the existing Apache infrastructure?
- Using Github pages? This would make it possible to automatically
regenerate the site on a push to the master/main branch.
- Somewhere else like the netlify this demo is hosted on?

Also:
Do we want this to be a separate repository?
Or do we want this to be part of the main code repository?

Niels



On Thu, Oct 28, 2021 at 10:44 AM Martin Grigorov 
wrote:

> Hi all,
>
> Please check the new candidate for Apache Avro website:
> https://avro-website.netlify.app/
>
> It is based on Hugo and uses Docsy theme.
> Its source code and instructions how to build could be found at
> https://github.com/martin-g/avro-website.
> The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175
>
> I am not web designer, so some things may look not finished.
> I've just copied the HTML content from the old site (
> https://avro.apache.org/) and converted it to Markdown for Hugo.
>
> Any feedback is welcome! With Pull Requests would be awesome!
>
> Regards,
> Martin
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: [VOTE] Drop Python 2 support

[Niels Basjes]

Yes, fine by me.
If people need an old version of python they can still get an old version
of Avro to help them out.

So for me
+1

Niels

On Tue, 28 Jul 2020, 16:55 Ryan Skraba,  wrote:

> Hello!  Thanks to all the preparatory work you've done over the last
> major releases, it seems alright to me!
>
> [X] +1 Drop support for Python 2 (non-binding)
>
> This leaves the path open to dropping avro-python3 in the next major
> release, which (at this point) should be low impact for anyone who has
> been watching the avro releases.
>
> All my best, Ryan
>
>
> On Tue, Jul 28, 2020 at 3:15 PM Michael A. Smith 
> wrote:
> >
> > Python 2 has been officially obsolete since January 1. I propose we drop
> > support for it.
> >
> > If we decide to continue supporting it, note that pip, the (main) Python
> > package management tool, will itself only support python3 in its next
> > release. So installing avro with python 2 will have to be done more
> > manually after that point.
> >
> > Here are the Apache Voting Rules:
> > http://www.apache.org/foundation/voting.html
> >
> > This vote will remain open for at least 72 hours, more probably until
> next
> > week.
> >
> > [ ] +1 Drop support for Python 2
> > [ ] +0
> > [ ] -1 Continue supporting Python 2 because/until…
>

Re: [ANNOUNCE] new committer Michael Smith

[Niels Basjes]

Welcome!

On Sun, Oct 20, 2019, 05:20 Sean Busbey  wrote:

> Hi folks!
>
> I'm very pleased to announce that Michael Smith has accepted the PMC's
> invitation to become a committer on the Apache Avro project.
>
> Please join me in congratulating Michael on this recognition of their
> great work thus far in our community.
>

Re: DRAFT ASF Board Report - Apache Avro October 2019

[Niels Basjes]

LGTM.
+1

On Wed, 9 Oct 2019, 16:06 Sean Busbey,  wrote:

> I went ahead and posted the report, with one minor change noted below. I
> can still edit it over the next few days if needed.
>
> On Wed, Oct 9, 2019 at 12:00 AM Sean Busbey  wrote:
>
> >
> > ## Activity:
> > The Avro project slowly is improving on our gains from the last year.
> >
> > We've gotten a fair bit of feedback on the 1.9.0 release and are working
> > towards establishing a regular cadence for getting the result of that
> > feedback
> > into our downstream communities. As noted in the summary, we've gotten
> the
> > follow-up 1.9.1 release out after a period of approximately 4 months.
> >
> >
> I added a parenthetical here to make clear that 1.9.1 is a minor release
> (rather than a maintenance release).
>

Re: [Announce] Please welcome Nándor Kollár to the Apache Avro PMC

[Niels Basjes]

Welcome!

On Fri, Aug 30, 2019, 23:39 Brian Lachniet  wrote:

> Congratulations, Nándor!
>
> On Fri, Aug 30, 2019, 5:37 PM Sean Busbey  wrote:
>
>> Hi folks!
>>
>> On behalf of the Apache Avro PMC I am pleased to announce that Nándor
>> Kollár has accepted our invitation to become a PMC member. We
>> appreciate Nándor stepping up to take more responsibility in the
>> project.
>>
>> Please join me in welcoming Nándor to the Avro PMC!
>>
>> As a reminder, if anyone would like to nominate another person as a
>> committer or PMC member, even if you are not currently a committer or
>> PMC member, you can always drop a note to priv...@avro.apache.org to
>> let us know.
>>
>

Re: [ANNOUNCE] Please welcome Brian Lachniet to the Apache Avro PMC

[Niels Basjes]

Welcome aboard

On Sat, Aug 24, 2019, 17:40 Sean Busbey  wrote:

> Hi folks!
>
> On behalf of the Apache Avro PMC I am pleased to announce that Brian
> Lachniet has accepted our invitation to become a PMC member. We
> appreciate Brian stepping up to take more responsibility in the
> project.
>
> Please join me in welcoming Brian to the Avro PMC!
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or
> PMC member, you can always drop a note to priv...@avro.apache.org to
> let us know.
>
> -busbey
>

Re: supporting a "unit" field for avro schema

[Niels Basjes]

I think we should approach this idea in two parts:

1) The schema. Things like does a different unit mean a different schema
fingerprint even though the bytes remain the same. What does a different
unit mean for schema evolution.

2) Language specifics. Scala has different possibilities than Java.

On Sat, Jun 29, 2019, 18:59 Erik Erlandson  wrote:

> I've been puzzling over what can be done to support this in more
> widely-used languages. The dilemma relative to the current language
> ecosystem is that languages with "modern" type systems (Haskell, Rust,
> Scala, etc) capable of supporting compile-time unit checking, in the
> particular style I've been exploring, are not yet widely used.
>
> With respect to Java, a couple approaches are plausible. One is to enhance
> the language, for example with Java-8 compiler plugins. Another might be to
> implement a unit type system similar to squants
> <https://github.com/typelevel/squants>. This style of unit type system is
> not as flexible or intuitive as what can be done with Scala's latest type
> system sorcery, but it would allow the community to build out a Java native
> type system that supports compile-time unit analysis. And its coverage of
> standard units could be made very good, as squants itself demonstrates.
>
> Python would also be a high-coverage target. I'm even less sure what to do
> for python, as it has no compile-time type checking, but perhaps a
> squants-like python class system would add value. Maybe python's new
> type-hints feature could be leveraged?
>
> Regarding unit expression representation, I'm not unhappy with what I've
> prototyped in `coulomb-avro`, in broad strokes. It has deficiencies that
> would need addressing. It doesn't yet support standard unit abbreviations,
> nor does it understand plurals (e.g. it can parse "second" but not
> "seconds"). Since it's "unit" field is just a custom metadata key, there is
> no enforcement. Parsers are currently instantiated via explicit lists of
> types, which is a property I like, but that may not work well in a world
> where multiple language bindings must be supported in a portable manner.
>
>
>
> On Sat, Jun 29, 2019 at 1:46 AM Niels Basjes  wrote:
>
> > Hi,
> >
> > I attended your talk in Berlin and at the end I thought "too bad this is
> > only Scala".
> >
> > I think it's a good idea to have this in Avro.
> >
> > The details will be tricky: How to encode the units in the schema for
> > example.
> > Especially because of the automatic conversion you spoke about.
> >
> > Niels
> >
> > On Fri, Jun 28, 2019, 23:58 Erik Erlandson  wrote:
> >
> > > Hi Avro community,
> > >
> > > Recently I have been experimenting with avro schema that are extended
> > with
> > > a "unit" field. By "unit" I mean expressions like "second", or
> > "megabyte" -
> > > that is "units of measure".
> > >
> > > I delivered a short talk on my experiments at Berlin Buzzwords, which
> can
> > > be viewed here:
> > > https://www.youtube.com/watch?v=qrQmB2KFKE8
> > > I also wrote a short blog post that may be faster to ingest:
> > >
> > >
> >
> http://erikerlandson.github.io/blog/2019/05/23/unit-types-for-avro-schema-integrating-avro-with-coulomb/
> > >
> > > I received some audience interest in making this concept "first class"
> > for
> > > avro, and so I'm writing to see what the avro dev community thinks of
> the
> > > idea. One issue is that this kind of unit checking is currently only
> > > available for Scala (and specifically scala 2.13 +).
> > >
> > > The Scala project itself is here:
> > > https://github.com/erikerlandson/coulomb
> > >
> > > Cheers,
> > > Erik
> > >
> >
>

Re: supporting a "unit" field for avro schema

[Niels Basjes]

Hi,

I attended your talk in Berlin and at the end I thought "too bad this is
only Scala".

I think it's a good idea to have this in Avro.

The details will be tricky: How to encode the units in the schema for
example.
Especially because of the automatic conversion you spoke about.

Niels

On Fri, Jun 28, 2019, 23:58 Erik Erlandson  wrote:

> Hi Avro community,
>
> Recently I have been experimenting with avro schema that are extended with
> a "unit" field. By "unit" I mean expressions like "second", or "megabyte" -
> that is "units of measure".
>
> I delivered a short talk on my experiments at Berlin Buzzwords, which can
> be viewed here:
> https://www.youtube.com/watch?v=qrQmB2KFKE8
> I also wrote a short blog post that may be faster to ingest:
>
> http://erikerlandson.github.io/blog/2019/05/23/unit-types-for-avro-schema-integrating-avro-with-coulomb/
>
> I received some audience interest in making this concept "first class" for
> avro, and so I'm writing to see what the avro dev community thinks of the
> idea. One issue is that this kind of unit checking is currently only
> available for Scala (and specifically scala 2.13 +).
>
> The Scala project itself is here:
> https://github.com/erikerlandson/coulomb
>
> Cheers,
> Erik
>

[jira] [Updated] (AVRO-2440) Allow BinaryMessageDecoder to read (generic) records with the same schema they were written with.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2440:
---
Fix Version/s: 1.9.1

> Allow BinaryMessageDecoder to read (generic) records with the same schema 
> they were written with.
> -
>
> Key: AVRO-2440
> URL: https://issues.apache.org/jira/browse/AVRO-2440
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Oscar Westra van Holthe - Kind
>Priority: Minor
> Fix For: 1.10.0, 1.9.1
>
>
> At work we have a use case to read Avro records from Kafka, and to write them 
> to one of several BigQuery tables (based on compatibility). This means that 
> there is no single schema (with all data) that is compatible with all records 
> we'll be reading.
> In order to allow this, we'd like to be able to use the BinaryMessageDecoder 
> without any read schema, and have it use the write schema of each record as 
> the read schema for that record.
> -Pull request will follow.-
> Pull request submitted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2448) Flaky test for open file descriptors

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2448:
--

 Summary: Flaky test for open file descriptors
 Key: AVRO-2448
 URL: https://issues.apache.org/jira/browse/AVRO-2448
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes
Assignee: Niels Basjes


In random situations (like here 
[https://travis-ci.org/apache/avro/jobs/548635040] ) the test 
TestDataFileReader : testForLeakingFileDescriptors fails with a message like 
this
 
{quote}[ERROR] 
testForLeakingFileDescriptors(org.apache.avro.TestDataFileReader) Time elapsed: 
0.026 s <<< FAILURE!
java.lang.AssertionError: File descriptor leaked from new DataFileReader() 
expected:<38> but was:<35>{quote}
The assessment is that during the test run the garbage collector cleaned up 
some of the open file descriptors and thus the expected number of open file 
descriptors is lower than expected by the test.

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2440) Allow BinaryMessageDecoder to read (generic) records with the same schema they were written with.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2440.

   Resolution: Fixed
Fix Version/s: 1.10.0

> Allow BinaryMessageDecoder to read (generic) records with the same schema 
> they were written with.
> -
>
> Key: AVRO-2440
> URL: https://issues.apache.org/jira/browse/AVRO-2440
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Oscar Westra van Holthe - Kind
>Priority: Minor
> Fix For: 1.10.0
>
>
> At work we have a use case to read Avro records from Kafka, and to write them 
> to one of several BigQuery tables (based on compatibility). This means that 
> there is no single schema (with all data) that is compatible with all records 
> we'll be reading.
> In order to allow this, we'd like to be able to use the BinaryMessageDecoder 
> without any read schema, and have it use the write schema of each record as 
> the read schema for that record.
> -Pull request will follow.-
> Pull request submitted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2404) Security sweep of build scripts and urls

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868763#comment-16868763
 ] 

Niels Basjes commented on AVRO-2404:


I have merged this into master.

In which other branches do we want this to be merged into too?

Only master? Also in branch-1.9 ? Older?

> Security sweep of build scripts and urls
> 
>
> Key: AVRO-2404
> URL: https://issues.apache.org/jira/browse/AVRO-2404
> Project: Apache Avro
>  Issue Type: Task
>  Components: build
>    Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>
> In some places we still use insecure URLs.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Please welcome Ismaël Mejía to the Apache Avro PMC

[Niels Basjes]

Welcome!

On Tue, 11 Jun 2019, 00:01 Brian Lachniet,  wrote:

> Congratulations, Ismaël!
>
> On Mon, Jun 10, 2019 at 5:48 PM Jesse Anderson 
> wrote:
>
>> Congrats!
>>
>> On Mon, Jun 10, 2019, 4:41 PM Sean Busbey  wrote:
>>
>>> Hi folks!
>>>
>>> On behalf of the Apache Avro PMC I am pleased to announce that Ismaël
>>> Mejía has accepted our invitation to become a PMC member. We
>>> appreciate Ismaël stepping up to take more responsibility in the
>>> project.
>>>
>>> Please join me in welcoming Ismaël to the Avro PMC!
>>>
>>> As a reminder, if anyone would like to nominate another person as a
>>> committer or PMC member, even if you are not currently a committer or
>>> PMC member, you can always drop a note to priv...@avro.apache.org to
>>> let us know.
>>>
>>> -busbey
>>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com 
>
>  
>

[jira] [Created] (AVRO-2408) Make Forrest work again in the Docker image for local development

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2408:
--

 Summary: Make Forrest work again in the Docker image for local 
development
 Key: AVRO-2408
 URL: https://issues.apache.org/jira/browse/AVRO-2408
 Project: Apache Avro
  Issue Type: Bug
Reporter: Niels Basjes
Assignee: Niels Basjes






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2405) Make Travis green again

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848692#comment-16848692
 ] 

Niels Basjes commented on AVRO-2405:


I think I may have found the cause: In some places it still says 1.9.0-SNAPSHOT

[https://github.com/nielsbasjes/avro/commit/73acffe5d20f32559665d53f3e2e060b677818e9]

> Make Travis green again
> ---
>
> Key: AVRO-2405
> URL: https://issues.apache.org/jira/browse/AVRO-2405
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 1.9.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2404) Security sweep of build scripts and urls

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2404:
--

 Summary: Security sweep of build scripts and urls
 Key: AVRO-2404
 URL: https://issues.apache.org/jira/browse/AVRO-2404
 Project: Apache Avro
  Issue Type: Task
  Components: build
Reporter: Niels Basjes
Assignee: Niels Basjes


In some places we still use insecure URLs.

We should fix that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Please welcome Fokko Driesprong to the Apache Avro PMC

[Niels Basjes]

Welcome aboard!

On Tue, May 14, 2019, 09:28 Sean Busbey  wrote:

> Hi folks!
>
> On behalf of the Apache Avro PMC I am pleased to announce that Fokko
> Driesprong has accepted our invitation to become a PMC member on the
> Avro project. We appreciate Fokko stepping up to take more
> responsibility in the project.
>
> Please join me in welcoming Fokko to the Avro PMC!
>
>
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or
> PMC member, you can always drop a note to priv...@avro.apache.org to
> let us know.
>
> -busbey
>

Re: Merge strategy

[Niels Basjes]

In general I'm for Squashing  +1.

In some exceptional cases I would have a small number of commits to
facilitate easier reviewing (like in AVRO-2117).

Niels.
On Thu, Oct 25, 2018 at 7:31 PM Doug Cutting  wrote:
>
> I'm +1 for squashing.
>
> Doug
>
> On Tue, Oct 23, 2018 at 2:35 PM Michael A. Smith 
> wrote:
>
> > I’m generally in favor of linear commits iff the committers can be trusted
> > to follow good git hygiene in general, like with small, proprietary
> > projects with strong leadership.
> >
> > With Avro, with its variegated multilang repo and commits coming from the
> > community at large, I’m not sure if linear-rebase or squash is a great
> > idea.
> >
> > If you think we can put an automated check in place to make sure commits in
> > PRs follow good hygiene by always having ticket tags, signed commits,
> > “real” authors, or whatever criteria makes sense, then I’m all for it.
> >
> > Cheers,
> > Michael Smith/kojiromike
> >
> > On Tue, Oct 23, 2018 at 17:19 Driesprong, Fokko 
> > wrote:
> >
> > > Hi all,
> > >
> > > Allow me to start the great debate.
> > >
> > > [image: image.png]
> > >
> > > I would like to get the Avro's dev-community's opinion on the merge
> > > strategy of pull requests. By default Github supports three options:
> > > - Create a merge commit
> > > - Squash and merge
> > > - Rebase and merge
> > > https://help.github.com/articles/about-merge-methods-on-github/
> > >
> > > Currently I see the PR's being merged into master using the merge commit
> > > (with some exceptions). From my perspective this has two major
> > > disadvantages:
> > > - It will add ugly merge commits on top of master, which don't add any
> > > value.
> > > - The git commit tree history isn't lineair. For example, a PR that has
> > > been merged recently <https://github.com/apache/avro/pull/270>, added a
> > > commit back in 19 dec 2017.
> > > - It makes it tricky to revert commits, since a PR merged using a merge
> > > commit might add one or more commits over the history of master.
> > >
> > > Therefore I would only allow two merge strategies:
> > > - Squash and merge: This will squash all the commits of the PR into one
> > > single commit, and push it on top of master. This should be the default
> > > strategy.
> > > - Rebase and merge: This can be used for very big PR's, in which you
> > don't
> > > want to squash the original.
> > >
> > > Github allows us to disable merge-options. My suggestion would be to
> > > disable the merge commit option, but I'd like to get the community's
> > > opinion on it.
> > >
> > > Cheers, Fokko
> > >
> > >
> > >
> >



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Jenkins

[Niels Basjes]

> have a build matrix ... Java 10 and 11. ... easier to configure in, ... 
> Travis  ... native support.

This is exactly why I asked about the preference for Jenkins.
Listening to these arguments I would tend to go to Travis as it
integrates fine with Github and is already used by many Apache
projects.

Niels
On Fri, Oct 12, 2018 at 7:16 AM Driesprong, Fokko  wrote:
>
> Thanks for the reply Niels.
>
> The only thing that makes Jenkins more appropriate right now is that Apache
> offers Jenkins hosts For now we only build against Java8. But in the future
> it might also make sense to have a build matrix to build also against Java
> 10 and 11. That would be easier to configure in, for example, Travis since
> there is native support. Personally, as long it has proper Github
> integration (support for PR's and easy links to the build logs), I don't
> really care which CI service we're using. Since we rely on Docker and
> Yetus, we're not really locked in to a specific service.
>
> Cheers, Fokko
>
>
>
>
> Op do 11 okt. 2018 om 21:53 schreef Niels Basjes :
>
> > I fully agree with automated builds.
> >
> > I'm just curious what makes Jenkins more appropriate?
> >
> > Niels
> > On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to revive the Jenkins build on Avro. The last runs are a bit
> > over
> > > a year old: https://builds.apache.org/view/A/view/Avro/
> > >
> > > I've opened a PR based on Apache Yetus
> > > <https://github.com/apache/avro/pull/344> to configure the precommit
> > task
> > > to run CI jobs, as Sean Busbey initially suggested. Initially I would
> > like
> > > to go for a Travis CI, as we have with Apache Airflow, but I think the
> > Jenkins
> > > of Apache itself <https://builds.apache.org/> might be more appropriate.
> > >
> > > The current Yetus scripts that I've written are still a bit crude, but I
> > > think we should have the automated process from end to end up as soon as
> > > possible, and then we can start polishing. Would it be possible to get
> > > access to the Apache Jenkins
> > > <
> > https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount
> > >,
> > > or is someone able to set this up?
> > >
> > > I would like to configure the Github pull request builder, based on
> > > the CloudBees
> > > plugin as suggested on the Apache wiki
> > > <
> > https://cwiki.apache.org/confluence/display/INFRA/GitHub+Pull+Request+Builder
> > >
> > > .
> > >
> > > Hope to get your feedback on the plan, and hope to move forward :-)
> > >
> > > Cheers, Fokko
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Jenkins

[Niels Basjes]

I fully agree with automated builds.

I'm just curious what makes Jenkins more appropriate?

Niels
On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko  wrote:
>
> Hi all,
>
> I'd like to revive the Jenkins build on Avro. The last runs are a bit over
> a year old: https://builds.apache.org/view/A/view/Avro/
>
> I've opened a PR based on Apache Yetus
> <https://github.com/apache/avro/pull/344> to configure the precommit task
> to run CI jobs, as Sean Busbey initially suggested. Initially I would like
> to go for a Travis CI, as we have with Apache Airflow, but I think the Jenkins
> of Apache itself <https://builds.apache.org/> might be more appropriate.
>
> The current Yetus scripts that I've written are still a bit crude, but I
> think we should have the automated process from end to end up as soon as
> possible, and then we can start polishing. Would it be possible to get
> access to the Apache Jenkins
> <https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount>,
> or is someone able to set this up?
>
> I would like to configure the Github pull request builder, based on
> the CloudBees
> plugin as suggested on the Apache wiki
> <https://cwiki.apache.org/confluence/display/INFRA/GitHub+Pull+Request+Builder>
> .
>
> Hope to get your feedback on the plan, and hope to move forward :-)
>
> Cheers, Fokko



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Resolved] (AVRO-2196) Build fails on a clean machine

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2196.

Resolution: Fixed

> Build fails on a clean machine
> --
>
> Key: AVRO-2196
> URL: https://issues.apache.org/jira/browse/AVRO-2196
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Niels Basjes
>    Assignee: Niels Basjes
>Priority: Major
> Fix For: 1.9.0
>
>
> On a clean machine (i.e. empty ~/.m2/reporsitory) when I simply run mvn clean 
> package the build fails with
> {code:java}
> {code:java}
> [INFO] 
> 
> [INFO] Building Apache Avro Tools 1.9.0-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Avro Toplevel .. SUCCESS [1.009s]
> [INFO] Apache Avro Java .. SUCCESS [0.930s]
> [INFO] Apache Avro Guava Dependencies  SUCCESS [1.377s]
> [INFO] Apache Avro ... SUCCESS [1:02.460s]
> [INFO] Apache Avro Compiler .. SUCCESS [10.705s]
> [INFO] Apache Avro Maven Plugin .. SUCCESS [5.190s]
> [INFO] Apache Avro IPC ... SUCCESS [47.988s]
> [INFO] Trevni Java ... SUCCESS [0.089s]
> [INFO] Trevni Java Core .. SUCCESS [4.526s]
> [INFO] Apache Avro Mapred API  SUCCESS [1:29.866s]
> [INFO] Trevni Java Avro .. SUCCESS [14.612s]
> [INFO] Trevni Specification .. SUCCESS [0.229s]
> [INFO] Apache Avro Tools . FAILURE [0.042s]
> [INFO] Apache Avro Protobuf Compatibility  SKIPPED
> [INFO] Apache Avro Thrift Compatibility .. SKIPPED
> [INFO] Apache Avro Maven Archetypes .. SKIPPED
> [INFO] Apache Avro Maven Service Archetype ... SKIPPED
> [INFO] Apache Avro gRPC .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 3:59.636s
> [INFO] Finished at: Fri Jun 29 09:14:04 UTC 2018
> [INFO] Final Memory: 80M/1309M
> [INFO] 
> 
> [ERROR] Failed to execute goal on project avro-tools: Could not resolve 
> dependencies for project org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT: Could 
> not find artifact org.apache.avro:avro-mapred:jar:tests:1.9.0-SNAPSHOT -> 
> [Help 1]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR] 
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :avro-tools
> {code}
>  
> Apparently the 
> {code:java}
> org.apache.avro:avro-mapred:jar:tests:1.9.0-SNAPSHOT{code}
> is not created (but seems needed).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Cleaning up the logo?

[Niels Basjes]

Yes very nice indeed.
Also having a vector format really helps in looking good in presentations
at conferences.

So +1 from me.

Niels Basjes

On Sun, Sep 2, 2018 at 7:54 PM Doug Cutting  wrote:

> +1 looks great!
>
> Thanks,
>
> Doug
>
> On Wed, Aug 29, 2018, 10:54 AM Ryan Blue 
> wrote:
>
> > I think it looks great! The new one has my +1.
> >
> > On Wed, Aug 29, 2018 at 9:26 AM Bridger Howell  wrote:
> >
> > > I think this proposed logo looks better than the current one. Any
> > official
> > > avro members have an opinion?
> > >
> > > On Tue, Aug 28, 2018 at 11:48 AM Daniel Gruno 
> > > wrote:
> > >
> > > > Hi Avro folks!
> > > >
> > > > I was wondering if it's worth cleaning up the Avro logo, making it a
> > tad
> > > > sharper?
> > > > What I'm proposing is retracing the logo, and I've made a proposal
> that
> > > is
> > > > available at:
> > > > http://www.apache.org/logos/res/avro/avro-proposed.svg
> > > >
> > > > If this sounds like a good idea, you are free to use the proposed
> > change
> > > > (I hereby license it under ALv2), the decision is of course yours :)
> If
> > > you
> > > > do decide to change it, do let me know or commit it to the comdev
> logo
> > > > central (see http://www.apache.org/logos/about.html for
> instructions)
> > > >
> > > > With regards,
> > > > Daniel.
> > > >
> > >
> > > --
> > >
> > >
> > > The information contained in this email message is PRIVATE and intended
> > > only for the personal and confidential use of the recipient named
> above.
> > > If
> > > the reader of this message is not the intended recipient or an agent
> > > responsible for delivering it to the intended recipient, you are hereby
> > > notified that you have received this message in error and that any
> > review,
> > > dissemination, distribution or copying of this message is strictly
> > > prohibited.  If you have received this communication in error, please
> > > notify us immediately by email, and delete the original message.
> > >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Commented] (AVRO-2195) Add Zstandard Codec

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561043#comment-16561043
 ] 

Niels Basjes commented on AVRO-2195:


I only had concerns about the option of making it possible for the other 
languages and the license of the used implementation. Both of these point check 
out fine.

I haven't been able to check the code itself so I trust Nandors judgement.

> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
>  Issue Type: New Feature
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
>  Labels: patch
> Attachments: AVRO-2195.patch, AVRO-2195.patch.v2, AVRO-2195.v3.patch, 
> AVRO-2195.v4.patch
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the 
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than 
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2173) remove CHANGES.txt

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2173:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Removed from both the master and branch-1.8

> remove CHANGES.txt
> --
>
> Key: AVRO-2173
> URL: https://issues.apache.org/jira/browse/AVRO-2173
> Project: Avro
>  Issue Type: Improvement
>Reporter: Doug Cutting
>Priority: Major
>
> The CHANGES.txt file is not well maintained and redundant with information in 
> Jira.
> Let's remove this file, and instead generate release notes from Jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2195) Add Zstandard Codec

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534218#comment-16534218
 ] 

Niels Basjes commented on AVRO-2195:


You can set the dependency in the pom.xml with scope set to test to achieve 
this.

> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
>  Issue Type: New Feature
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
>  Labels: patch
> Attachments: AVRO-2195.patch, AVRO-2195.patch.v2
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the 
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than 
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2195) Add Zstandard Codec

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534128#comment-16534128
 ] 

Niels Basjes commented on AVRO-2195:


According to this page [https://www.apache.org/legal/resolved.html#category-a]  
the BSD license is ok to include. The originating issue came to the same 
conclusion: COMPRESS-423

Looking at the provided patch I noticed that the zstd-jni still needed to be 
included separately in addition to the commons-compress because has explicitly 
been marked as optional 
[https://github.com/apache/commons-compress/blob/master/pom.xml#L87]

COMPRESS-423 does not mention why this was done.

I'm in doubt here; should we do the same or just include it as shown in the 
current patch?

What I also find remarkable is that the zstd-jni is not just a dependency, it 
is actually shaded into the avro jar.

I've tried the second patch but it misses the 
lang/java/avro/src/main/java/org/apache/avro/file/ZstandardCodec.java 
completely.

 

> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
>  Issue Type: New Feature
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
>  Labels: patch
> Attachments: AVRO-2195.patch, AVRO-2195.patch.v2
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the 
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than 
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Request for review: AVRO-2195

[Niels Basjes]

Thanks for putting up this contribution.

I put some minor comments in the Jira.

There is one thing I want to mention here also

An important feature of Avro is being crossplatform/crosslanguage so I am
very curious regarding the portability of this compressor across languages.

Simply put: does it exist outside the Java world?

Niels Basjes



On Mon, Jul 2, 2018 at 8:48 AM, Benson Qiu  wrote:

> Hi,
>
> Can a committer please assign AVRO-2195
> <https://issues.apache.org/jira/browse/AVRO-2195> to me and possibly leave
> comments on my patch?
>
> Thank you.
> - Benson
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Commented] (AVRO-2195) Add Zstandard Codec

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529497#comment-16529497
 ] 

Niels Basjes commented on AVRO-2195:


Thanks for putting up this contribution.

An important feature of Avro is being crossplatform/crosslanguage so I am very 
curious regarding the portability of this compressor across languages.

Simply put: does it exist outside the Java world?

I have not yet had a good look at your code.

A few small first feedback points:
 * A tiny typo in the comment  /** zstanard codec.*/
 * That dependency you mentioned seems to be a known limitation: 
[https://commons.apache.org/proper/commons-compress/limitations.html]
 * I would really like to see the tests.

> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
>  Labels: patch
> Attachments: AVRO-2195.patch
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the 
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than 
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AVRO-2173) remove CHANGES.txt

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527531#comment-16527531
 ] 

Niels Basjes edited comment on AVRO-2173 at 6/29/18 1:09 PM:
-

It is removed now from master.

I propose we kick is from branch-1.8 too and let all other branches remain 
untouched.

[~nkollar] / [~cutting] what do you think?

 


was (Author: nielsbasjes):
I is removed now from master. I propose we kick is from branch-1.8 too and let 
all other branches remain untouched.

[~nkollar] / [~cutting] what do you think?

 

> remove CHANGES.txt
> --
>
> Key: AVRO-2173
> URL: https://issues.apache.org/jira/browse/AVRO-2173
> Project: Avro
>  Issue Type: Improvement
>Reporter: Doug Cutting
>Priority: Major
>
> The CHANGES.txt file is not well maintained and redundant with information in 
> Jira.
> Let's remove this file, and instead generate release notes from Jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2173) remove CHANGES.txt

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527531#comment-16527531
 ] 

Niels Basjes commented on AVRO-2173:


I is removed now from master. I propose we kick is from branch-1.8 too and let 
all other branches remain untouched.

[~nkollar] / [~cutting] what do you think?

 

> remove CHANGES.txt
> --
>
> Key: AVRO-2173
> URL: https://issues.apache.org/jira/browse/AVRO-2173
> Project: Avro
>  Issue Type: Improvement
>Reporter: Doug Cutting
>Priority: Major
>
> The CHANGES.txt file is not well maintained and redundant with information in 
> Jira.
> Let's remove this file, and instead generate release notes from Jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2173) remove CHANGES.txt

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527508#comment-16527508
 ] 

Niels Basjes commented on AVRO-2173:


[~nkollar] Perhaps fixing the 'wiki' can be put under the umbrella of AVRO-2175 
?

> remove CHANGES.txt
> --
>
> Key: AVRO-2173
> URL: https://issues.apache.org/jira/browse/AVRO-2173
> Project: Avro
>  Issue Type: Improvement
>Reporter: Doug Cutting
>Priority: Major
>
> The CHANGES.txt file is not well maintained and redundant with information in 
> Jira.
> Let's remove this file, and instead generate release notes from Jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2196) Build fails on a clean machine

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2196:
--

 Summary: Build fails on a clean machine
 Key: AVRO-2196
 URL: https://issues.apache.org/jira/browse/AVRO-2196
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.9.0
Reporter: Niels Basjes
Assignee: Niels Basjes
 Fix For: 1.9.0


On a clean machine (i.e. empty ~/.m2/reporsitory) when I simply run mvn clean 
package the build fails with

{code:java}
{code:java}
[INFO] 
[INFO] Building Apache Avro Tools 1.9.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Avro Toplevel .. SUCCESS [1.009s]
[INFO] Apache Avro Java .. SUCCESS [0.930s]
[INFO] Apache Avro Guava Dependencies  SUCCESS [1.377s]
[INFO] Apache Avro ... SUCCESS [1:02.460s]
[INFO] Apache Avro Compiler .. SUCCESS [10.705s]
[INFO] Apache Avro Maven Plugin .. SUCCESS [5.190s]
[INFO] Apache Avro IPC ... SUCCESS [47.988s]
[INFO] Trevni Java ... SUCCESS [0.089s]
[INFO] Trevni Java Core .. SUCCESS [4.526s]
[INFO] Apache Avro Mapred API  SUCCESS [1:29.866s]
[INFO] Trevni Java Avro .. SUCCESS [14.612s]
[INFO] Trevni Specification .. SUCCESS [0.229s]
[INFO] Apache Avro Tools . FAILURE [0.042s]
[INFO] Apache Avro Protobuf Compatibility  SKIPPED
[INFO] Apache Avro Thrift Compatibility .. SKIPPED
[INFO] Apache Avro Maven Archetypes .. SKIPPED
[INFO] Apache Avro Maven Service Archetype ... SKIPPED
[INFO] Apache Avro gRPC .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 3:59.636s
[INFO] Finished at: Fri Jun 29 09:14:04 UTC 2018
[INFO] Final Memory: 80M/1309M
[INFO] 
[ERROR] Failed to execute goal on project avro-tools: Could not resolve 
dependencies for project org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT: Could 
not find artifact org.apache.avro:avro-mapred:jar:tests:1.9.0-SNAPSHOT -> [Help 
1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :avro-tools
{code}
 

Apparently the 
{code:java}
org.apache.avro:avro-mapred:jar:tests:1.9.0-SNAPSHOT{code}
is not created (but seems needed).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2175) website refactor

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466908#comment-16466908
 ] 

Niels Basjes commented on AVRO-2175:


Have a look at what the team of Apache Beam did with Jekyll and the additional 
tooling to make it work smoothly with Jenkins.
I submitted a few changes a while ago and this was very easy to do.

> website refactor
> 
>
> Key: AVRO-2175
> URL: https://issues.apache.org/jira/browse/AVRO-2175
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
>
> https://lists.apache.org/thread.html/4b804f4e99dc9975c42af59485225808846648b90015f04ab9787246@%3Cdev.avro.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Avro serialization in streaming context

[Niels Basjes]

HI,

In Avro 1.8.2 we have added a "Message" format specification.
In the Java classes there are now extra methods generated like
toByteBuffer() and fromByteBuffer() that are intended for exactly the
purpose of using Avro as the serialization specification in a streaming
context.
In fact at bol.com (where I work) we already do this in production.

Please have a look at this because I have the feeling this already does a
lot of what you mention.

Niels Basjes


On Thu, Feb 22, 2018 at 7:18 AM, Commeau, Gabriel <gabriel.comm...@gmail.com
> wrote:

> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Resolved] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2041.

Resolution: Fixed

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Niels Basjes
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372096#comment-16372096
 ] 

Niels Basjes commented on AVRO-2041:


I updated http://avro.apache.org/version_control.html

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Niels Basjes
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (AVRO-2144) 404 - Not found - Invalid Url for CSharp documentation

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes reopened AVRO-2144:


Reopening: At this point nothing has been committed yet.

> 404 - Not found - Invalid Url for CSharp documentation
> --
>
> Key: AVRO-2144
> URL: https://issues.apache.org/jira/browse/AVRO-2144
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.2
>Reporter: Selva Chinnasamy
>Assignee: Selva Chinnasamy
>Priority: Minor
>
> Invalid Url for C# Api documentation
> Selecting the below link
> [C# API|http://avro.apache.org/docs/current/api/csharp/index.html]
> leads to 404. Below is the current result:
> h1. Not Found
> The requested URL /docs/current/api/csharp/index.html was not found on this 
> server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371215#comment-16371215
 ] 

Niels Basjes commented on AVRO-2041:


The site is in subversion and uses ant/forrest to be generated.

I tried to use the docker image we have to generate it but ran into a lot of 
errors like these
{code:java}
[exec] 
/usr/local/apache-forrest/main/webapp/resources/schema/relaxng/sitemap-v06.rng:781:29:
 error: datatype library "http://www.w3.org/2001/XMLSchema-datatypes; not 
recognized
{code}
I do not know too much about Forrest so I tried if updating it works and it 
does. So I updated forrest in the docker image from 0.8 to 0.9 (the latest).

Should I commit this update to 0.9 or is there a better way to solve this?

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Niels Basjes
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Travel Assistance applications open.

[Niels Basjes]

—
The Travel Assistance Committee (TAC) are pleased to announce that travel
assistance applications for ApacheCon NA 2018 are now open!

We will be supporting ApacheCon NA Montreal, Canada on 24th - 29th
September 2018

 TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons.
For more info on this years applications and qualifying criteria, please
visit the TAC website at < http://www.apache.org/travel/ <
http://www.apache.org/travel/> >. Applications are now open and will close
1st May.

Important: Applications close on May 1st, 2018. Applicants have until the
closing date above to submit their applications (which should contain as
much supporting material as required to efficiently and accurately process
their request), this will enable TAC to announce successful awards shortly
afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.
We look forward to greeting many of you in Montreal

Kind Regards,
Gavin - (On behalf of the Travel Assistance Committee)
—

[jira] [Commented] (AVRO-2144) 404 - Not found - Invalid Url for CSharp documentation

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362548#comment-16362548
 ] 

Niels Basjes commented on AVRO-2144:


Where did you find this incorrect URL?

> 404 - Not found - Invalid Url for CSharp documentation
> --
>
> Key: AVRO-2144
> URL: https://issues.apache.org/jira/browse/AVRO-2144
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.2
>Reporter: Selva Chinnasamy
>Priority: Minor
>
> Invalid Url for C# Api documentation
> Selecting the below link
> [C# API|http://avro.apache.org/docs/current/api/csharp/index.html]
> leads to 404. Below is the current result:
> h1. Not Found
> The requested URL /docs/current/api/csharp/index.html was not found on this 
> server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361208#comment-16361208
 ] 

Niels Basjes commented on AVRO-2041:


Seems like the transition was already done.

TODO: Update [http://avro.apache.org/version_control.html] and other places (if 
any) where we mention the git repo for developers.

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Niels Basjes
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [RESULT] Convert to gitbox

[Niels Basjes]

FYI:

Just now I put in the request with the Apache Infra team via ticket
https://issues.apache.org/jira/browse/INFRA-15913

Niels

On Thu, Jan 25, 2018 at 9:05 AM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> First of all apologies for the delay.
>
> For the vote to migrate Apache Avro to use gitbox we have seen
> 5   "+1" votes
> 0   "0" votes
> 0   "-1" votes
>
> This means that the vote has PASSED.
>
> Because of the extreme delay since we started the vote I'm waiting an
> additional 24 hours before actually submitting a ticket with INFRA to start
> the migration to gitbox.
>
> So unless we see good reasons not to migrate I'm starting the migration
> tomorrow.
>
> Best regards,
>
> Niels Basjes
> Apache Avro PMC
>
>
> On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:
>
>> +1
>>
>> On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org> wrote:
>>
>> > +1
>> >
>> > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
>> > > Way back in May we had consensus[1] on using gitbox so that we could
>> > start making use of pull requests.
>> > >
>> > > In the subsequent JIRAs to implement, it became clear that the entry
>> bar
>> > to start is convert our repo to use gitbox. this means the writeable
>> > repository will be on github and some committer set up will be
>> needed[2]. I
>> > don't think this got adequately conveyed originally, so I wanted to
>> call it
>> > out here.
>> > >
>> > > Additionally, the infra process for converting to gitbox requires a
>> link
>> > to a community consensus thread. While I think we already had that back
>> in
>> > [1], May was a long time ago and some folks might not be watching for
>> > [DISCUSS] threads.
>> > >
>> > > Please vote:
>> > >
>> > > +1: we should convert to gitbox and enable pull request merging
>> > > -1: we should not convert to gitbox because...
>> > >
>> > > Vote will be open for at least 72 hours and will be subject to
>> Majority
>> > Approval (3 or more +1s and more +1s than -1s).
>> > >
>> > > [1]: https://s.apache.org/ieB0
>> > > [2]: Committers will need to flag a github account as theirs via the
>> > account linking utility:
>> > > https://gitbox.apache.org/setup/
>> > >
>> > > The git-wip repository will also cease, so folks who currently use it
>> > will need to update their remote url for existing git clones. We can
>> call
>> > this out in our contribution docs.
>> > >
>> >
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340890#comment-16340890
 ] 

Niels Basjes commented on AVRO-2041:


I put in the request with the Apache Infra team via ticket INFRA-15913

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes reassigned AVRO-2041:
--

Assignee: Niels Basjes  (was: Sean Busbey)

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Niels Basjes
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339107#comment-16339107
 ] 

Niels Basjes commented on AVRO-2041:


Vote:

[https://lists.apache.org/thread.html/03d761432d99fd0cb9c7c935e45177cf3d8e4efec3066f45bb1f4f43@%3Cdev.avro.apache.org%3E]

Result:

[https://lists.apache.org/thread.html/e332c49cd25b3659f3820455974f0e6fad66edb47fed3a0be905cb22@%3Cdev.avro.apache.org%3E]

 

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[RESULT] Convert to gitbox

[Niels Basjes]

Hi all,

First of all apologies for the delay.

For the vote to migrate Apache Avro to use gitbox we have seen
5   "+1" votes
0   "0" votes
0   "-1" votes

This means that the vote has PASSED.

Because of the extreme delay since we started the vote I'm waiting an
additional 24 hours before actually submitting a ticket with INFRA to start
the migration to gitbox.

So unless we see good reasons not to migrate I'm starting the migration
tomorrow.

Best regards,

Niels Basjes
Apache Avro PMC


On Tue, Aug 29, 2017 at 9:14 PM, Suraj Acharya <su...@apache.org> wrote:

> +1
>
> On Tue, Aug 29, 2017 at 11:49 AM, Sean Busbey <bus...@apache.org> wrote:
>
> > +1
> >
> > On 2017-08-29 11:29, "Sean Busbey"<bus...@apache.org> wrote:
> > > Way back in May we had consensus[1] on using gitbox so that we could
> > start making use of pull requests.
> > >
> > > In the subsequent JIRAs to implement, it became clear that the entry
> bar
> > to start is convert our repo to use gitbox. this means the writeable
> > repository will be on github and some committer set up will be
> needed[2]. I
> > don't think this got adequately conveyed originally, so I wanted to call
> it
> > out here.
> > >
> > > Additionally, the infra process for converting to gitbox requires a
> link
> > to a community consensus thread. While I think we already had that back
> in
> > [1], May was a long time ago and some folks might not be watching for
> > [DISCUSS] threads.
> > >
> > > Please vote:
> > >
> > > +1: we should convert to gitbox and enable pull request merging
> > > -1: we should not convert to gitbox because...
> > >
> > > Vote will be open for at least 72 hours and will be subject to Majority
> > Approval (3 or more +1s and more +1s than -1s).
> > >
> > > [1]: https://s.apache.org/ieB0
> > > [2]: Committers will need to flag a github account as theirs via the
> > account linking utility:
> > > https://gitbox.apache.org/setup/
> > >
> > > The git-wip repository will also cease, so folks who currently use it
> > will need to update their remote url for existing git clones. We can call
> > this out in our contribution docs.
> > >
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Commented] (AVRO-2041) set up gitbox integration

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337589#comment-16337589
 ] 

Niels Basjes commented on AVRO-2041:


Seems the correct URL for the ticket is 

https://issues.apache.org/jira/browse/INFRA-14667
It was closed as "Invalid" months ago stating that Avro is not a gitbox project.
I thought this ticket was to convert Avro into gitbox ?!?!?

> set up gitbox integration
> -
>
> Key: AVRO-2041
> URL: https://issues.apache.org/jira/browse/AVRO-2041
> Project: Avro
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> We got consensus back in may about [turning on gitbox 
> integration|https://lists.apache.org/thread.html/cdd8ba14c1bf8aca2d71d09862e9780f2dc46af414ed78b1e3fd9c56@%3Cdev.avro.apache.org%3E]
>  so do it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2120) NullPointerException thrown by Schema.Parser#parse(literal)

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2120.

Resolution: Fixed

> NullPointerException thrown by Schema.Parser#parse(literal)
> ---
>
> Key: AVRO-2120
> URL: https://issues.apache.org/jira/browse/AVRO-2120
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sebastien Dubois
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>
> Calling the parse method with an invalid input (e.g., "") instead of a valid 
> schema throws a NullPointerException.
> Expected behavior: SchemaParseException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2120) NullPointerException thrown by Schema.Parser#parse(literal)

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2120:
---
Fix Version/s: 1.8.3
   1.9.0

> NullPointerException thrown by Schema.Parser#parse(literal)
> ---
>
> Key: AVRO-2120
> URL: https://issues.apache.org/jira/browse/AVRO-2120
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sebastien Dubois
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>
> Calling the parse method with an invalid input (e.g., "") instead of a valid 
> schema throws a NullPointerException.
> Expected behavior: SchemaParseException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

AVRO-2117: Code cleanup

[Niels Basjes]

Hi,

Merry X-Mas to you all.


When there are a lot of warnings that are 'trivial' they tend to hide the
warnings that are important.
So in most projects I aim for a 'zero warning' situation so you can spot
any problem in de code base much faster.


I recently put some effort into cleaning up some of the simple and needless
warnings that various tools give on the current codebase. This is not 'all
warnings' but it is a big step in the right direction.

https://issues.apache.org/jira/browse/AVRO-2117
https://github.com/apache/avro/pull/266
To keep an overview of what was changed I used a single commit per "type of
fix".

I would really appreciate feedback from you guys if and how to proceed with
this one.

Thanks.

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Assigned] (AVRO-2120) NullPointerException thrown by Schema.Parser#parse(literal)

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes reassigned AVRO-2120:
--

Assignee: Niels Basjes

> NullPointerException thrown by Schema.Parser#parse(literal)
> ---
>
> Key: AVRO-2120
> URL: https://issues.apache.org/jira/browse/AVRO-2120
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sebastien Dubois
>    Assignee: Niels Basjes
>
> Calling the parse method with an invalid input (e.g., "") instead of a valid 
> schema throws a NullPointerException.
> Expected behavior: SchemaParseException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-1967) NPE calling getXyzBuilder on instance where the xyz is null

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-1967:
---
Fix Version/s: (was: 1.8.2)
   1.8.3
   1.9.0

> NPE calling getXyzBuilder on instance where the xyz is null
> ---
>
> Key: AVRO-1967
> URL: https://issues.apache.org/jira/browse/AVRO-1967
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
>Priority: Critical
> Fix For: 1.9.0, 1.8.3
>
>
> Assume a Record with a nested nullable record that has been set to the value 
> 'null'.
> Then call the getXxxBuilder method on that record to obtain a builder for 
> that nested record and you get an NPE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-1966) NPE When copying builder with nullable record

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-1966:
---
Fix Version/s: (was: 1.8.2)
   1.8.3
   1.9.0

> NPE When copying builder with nullable record
> -
>
> Key: AVRO-1966
> URL: https://issues.apache.org/jira/browse/AVRO-1966
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>    Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Critical
> Fix For: 1.9.0, 1.8.3
>
>
> Assume a schema with a record that embeds a record that is optional (i.e. the 
> reference is union with null) and has the default value null.
> Then create a builder and copy that builder into a new builder.
> Using this copy will yield a NulPointerException. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Committed.

On Fri, Dec 22, 2017 at 1:27 PM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> Because this is tricky I decided to first let you guys review everything
> before actually committing.
> https://github.com/apache/avro/pull/271
>
> This is the following set of commits:
> AVRO-2118
> 1) Reverting the problematic commits by Thiruvalluvan M G
> 2) Applying Zoltans new license headers
> 3) Me removing the author tag.
> AVRO-2119
> 4) Making sure Apache Rat runs 'always'
>
> - At this point I found the build took ages because it was running rat
> also in all child projects AND failing on many files.
>
> 5) Upgrading Apache Rat and making sure it only runs in the top level
> project.
>
> I would appreciate it if you guys could check if this set of commits is
> correct now.
>
> Thanks.
>
> Niels Basjes
>
>
> On Wed, Dec 20, 2017 at 11:56 PM, Sean Busbey <bus...@apache.org> wrote:
>
>> I think so.
>>
>> It'd be better for someone who hasn't seen the current file to do the
>> code cleanup. But I don't think that's a blocker.
>>
>> On Wed, Dec 20, 2017 at 4:45 PM, Niels Basjes <ni...@basjes.nl> wrote:
>> > Hi all,
>> >
>> > It has been about 1 week now.
>> > Zoltan has put up a pull request correcting the copyright issue very
>> > quickly last week.
>> >
>> > Unfortunately I haven't seen any response from Thiruvalluvan M G <
>> > th...@startsmartlabs.com> yet.
>> > Since his changes were very simple (basic code cleanup) I propose we do
>> the
>> > following so we can cleanup the codebase and close the issues/pull
>> requests
>> > related to all of this.
>> >
>> > 1) I do a git revert on the two affected files and simply rollback his
>> > changes. So "the changes made under the wrong license" are gone.
>> > 2) I merge the fix by Zoltan to correct the licenses.
>> > 3) I cleanup the code (now under the correct license).
>> > 4) I include the related things that have already been checked.
>> >
>> > At this point I am unsure: Is this a valid way (from a licensing
>> > perspective) to fix this?
>> >
>> > Niels
>> >
>> >
>> > On Thu, Dec 14, 2017 at 6:36 PM, Suraj Acharya <su...@apache.org>
>> wrote:
>> >
>> >> As part of the release we do run the rat plugin.
>> >> So it is a highly unlikely this would have been run through a release.
>> >> However, changing it now is a great addition since the release manager
>> has
>> >> to go through the whole license check for all of the files.
>> >> Also, as Sean mentioned anyone who has made any changes to the file
>> after
>> >> the addition of the license will also need to be informed of the
>> change to
>> >> the license.
>> >>
>> >> Thanks
>> >>
>> >> Suraj
>> >>
>> >>
>> >> On Thu, Dec 14, 2017 at 2:49 AM, Niels Basjes <ni...@basjes.nl> wrote:
>> >>
>> >> > Hi all,
>> >> >
>> >> > After we hear back from Thiru I would like Zoltan to fix these 4
>> files.
>> >> >
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/SchemaVisitorAction.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/SchemaVisitor.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/Schemas.java
>> >> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
>> >> > schema/CloningVisitor.java
>> >> >
>> >> > See: https://issues.apache.org/jira/browse/AVRO-2118
>> >> >
>> >> > After those have been fixed we can commit this change (guys, please
>> >> review
>> >> > this. Thanks.)
>> >> > https://issues.apache.org/jira/browse/AVRO-2119
>> >> >
>> >> > Niels Basjes
>> >> >
>> >> >
>> >> > On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl>
>> wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I had a closer look at the code base.
>> >> > >
>> >> > > Most important:
>> >> > > 1) I have found these files only in the master branch.
>> >> > > 2) I checked 

[jira] [Resolved] (AVRO-2119) Run Apache Rat check on every java build

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2119.

Resolution: Fixed

> Run Apache Rat check on every java build
> 
>
> Key: AVRO-2119
> URL: https://issues.apache.org/jira/browse/AVRO-2119
> Project: Avro
>  Issue Type: Improvement
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>
> On a regular basis files get in that do not meet the licensing requirements.
> The root casue seems to be that in order to do the check you need to run the 
> Apache Rat too manually (which many forget).
> This must be dome much more often.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2118.

Resolution: Fixed

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>Assignee: Zoltan Farkas
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

It has been about 1 week now.
Zoltan has put up a pull request correcting the copyright issue very
quickly last week.

Unfortunately I haven't seen any response from Thiruvalluvan M G <
th...@startsmartlabs.com> yet.
Since his changes were very simple (basic code cleanup) I propose we do the
following so we can cleanup the codebase and close the issues/pull requests
related to all of this.

1) I do a git revert on the two affected files and simply rollback his
changes. So "the changes made under the wrong license" are gone.
2) I merge the fix by Zoltan to correct the licenses.
3) I cleanup the code (now under the correct license).
4) I include the related things that have already been checked.

At this point I am unsure: Is this a valid way (from a licensing
perspective) to fix this?

Niels


On Thu, Dec 14, 2017 at 6:36 PM, Suraj Acharya <su...@apache.org> wrote:

> As part of the release we do run the rat plugin.
> So it is a highly unlikely this would have been run through a release.
> However, changing it now is a great addition since the release manager has
> to go through the whole license check for all of the files.
> Also, as Sean mentioned anyone who has made any changes to the file after
> the addition of the license will also need to be informed of the change to
> the license.
>
> Thanks
>
> Suraj
>
>
> On Thu, Dec 14, 2017 at 2:49 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
> > Hi all,
> >
> > After we hear back from Thiru I would like Zoltan to fix these 4 files.
> >
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/SchemaVisitorAction.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/SchemaVisitor.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/Schemas.java
> > lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > schema/CloningVisitor.java
> >
> > See: https://issues.apache.org/jira/browse/AVRO-2118
> >
> > After those have been fixed we can commit this change (guys, please
> review
> > this. Thanks.)
> > https://issues.apache.org/jira/browse/AVRO-2119
> >
> > Niels Basjes
> >
> >
> > On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > > Hi all,
> > >
> > > I had a closer look at the code base.
> > >
> > > Most important:
> > > 1) I have found these files only in the master branch.
> > > 2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
> > > present in any of those releases. (
> > > So we're ok on this part.
> > >
> > > I have found exactly 2 files with this problem:
> > > ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > > schema/SchemaVisitorAction.java
> > > ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> > > schema/SchemaVisitor.java
> > >
> > > I have found 1 additional commit that touches these two files:
> > >
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9
> > >
> > > commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
> > >> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
> > >> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
> > >> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
> > >> CommitDate: Sun Apr 30 23:31:29 2017 +0530
> > >> Added more tests and fixed a couple of bugs. Also formatted the
> code
> > >
> > >
> > > In both these files the only changes are:
> > > - Removing the author tag
> > > - Whitespace changes.
> > >
> > > See:
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
> > > https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> > > 1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92
> > >
> > > @Thiru: To be 100% sure: You agree with changing these two files to the
> > > Apache license?
> > >
> > > Niels Basjes
> > >
> > >
> > > On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org>
> wrote:
> > >
> > >> In addition to Zoltan we'll need to confirm anyone else who has
> modified
> > >> the files.
> > >>
> > >> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
> > >>
> > >> > Have these files made it into a release?
> >

[jira] [Updated] (AVRO-2114) Make missing value exceptions in nested structures easier to read.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2114:
---
Fix Version/s: 1.8.3

> Make missing value exceptions in nested structures easier to read.
> --
>
> Key: AVRO-2114
> URL: https://issues.apache.org/jira/browse/AVRO-2114
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>    Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
> Fix For: 1.9.0, 1.8.3
>
>
> If you have in Java the generated SpecificRecord structure and it has nested 
> fields then finding the cause of the problem takes a while. The cause is that 
> the system wraps the deeper runtime exception in another runtime exception 
> and in the end it can become too much to dig through.
> Goal: make the exception in this kind of scenario easier to read.
> If you let this test fail 
> https://github.com/apache/avro/blob/master/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificBuilderTree.java#L75
>  the stacktrace you get looks like this:
> {code}
> org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException: 
> org.apache.avro.AvroRuntimeException: Field networkAddress type:STRING pos:1 
> not set and has no default value
>   at org.apache.avro.test.http.Request$Builder.build(Request.java:450)
>   at 
> org.apache.avro.specific.TestSpecificBuilderTree.failOnIncompleteTree(TestSpecificBuilderTree.java:77)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.apache.avro.AvroRuntimeException: 
> org.apache.avro.AvroRuntimeException: Field networkAddress type:STRING pos:1 
> not set and has no default value
>   at 
> org.apache.avro.test.http.NetworkConnection$Builder.build(NetworkConnection.java:293)
>   at org.apache.avro.test.http.Request$Builder.build(Request.java:439)
>   ... 23 more
> Caused by: org.apache.avro.AvroRuntimeException: Field networkAddress 
> type:STRING pos:1 not set and has no default value
>   at 
> org.apache.avro.generic.GenericData.getDefaultValue(GenericData.java:998)
>   at 
> org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:138)
>   at 
> org.apache.avro.test.http.NetworkConnection$Builder.build(NetworkConnection.java:290)
>   ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AVRO-2114) Make missing value exceptions in nested structures easier to read.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes resolved AVRO-2114.

Resolution: Fixed

> Make missing value exceptions in nested structures easier to read.
> --
>
> Key: AVRO-2114
> URL: https://issues.apache.org/jira/browse/AVRO-2114
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>    Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
> Fix For: 1.9.0
>
>
> If you have in Java the generated SpecificRecord structure and it has nested 
> fields then finding the cause of the problem takes a while. The cause is that 
> the system wraps the deeper runtime exception in another runtime exception 
> and in the end it can become too much to dig through.
> Goal: make the exception in this kind of scenario easier to read.
> If you let this test fail 
> https://github.com/apache/avro/blob/master/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificBuilderTree.java#L75
>  the stacktrace you get looks like this:
> {code}
> org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException: 
> org.apache.avro.AvroRuntimeException: Field networkAddress type:STRING pos:1 
> not set and has no default value
>   at org.apache.avro.test.http.Request$Builder.build(Request.java:450)
>   at 
> org.apache.avro.specific.TestSpecificBuilderTree.failOnIncompleteTree(TestSpecificBuilderTree.java:77)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.apache.avro.AvroRuntimeException: 
> org.apache.avro.AvroRuntimeException: Field networkAddress type:STRING pos:1 
> not set and has no default value
>   at 
> org.apache.avro.test.http.NetworkConnection$Builder.build(NetworkConnection.java:293)
>   at org.apache.avro.test.http.Request$Builder.build(Request.java:439)
>   ... 23 more
> Caused by: org.apache.avro.AvroRuntimeException: Field networkAddress 
> type:STRING pos:1 not set and has no default value
>   at 
> org.apache.avro.generic.GenericData.getDefaultValue(GenericData.java:998)
>   at 
> org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:138)
>   at 
> org.apache.avro.test.http.NetworkConnection$Builder.build(NetworkConnection.java:290)
>   ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

After we hear back from Thiru I would like Zoltan to fix these 4 files.

lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitorAction.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitor.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/Schemas.java
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/CloningVisitor.java

See: https://issues.apache.org/jira/browse/AVRO-2118

After those have been fixed we can commit this change (guys, please review
this. Thanks.)
https://issues.apache.org/jira/browse/AVRO-2119

Niels Basjes


On Thu, Dec 14, 2017 at 11:11 AM, Niels Basjes <ni...@basjes.nl> wrote:

> Hi all,
>
> I had a closer look at the code base.
>
> Most important:
> 1) I have found these files only in the master branch.
> 2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
> present in any of those releases. (
> So we're ok on this part.
>
> I have found exactly 2 files with this problem:
> ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> schema/SchemaVisitorAction.java
> ./lang/java/compiler/src/main/java/org/apache/avro/compiler/
> schema/SchemaVisitor.java
>
> I have found 1 additional commit that touches these two files:
>
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9
>
> commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
>> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
>> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
>> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
>> CommitDate: Sun Apr 30 23:31:29 2017 +0530
>> Added more tests and fixed a couple of bugs. Also formatted the code
>
>
> In both these files the only changes are:
> - Removing the author tag
> - Whitespace changes.
>
> See:
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
> https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f
> 1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92
>
> @Thiru: To be 100% sure: You agree with changing these two files to the
> Apache license?
>
> Niels Basjes
>
>
> On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org> wrote:
>
>> In addition to Zoltan we'll need to confirm anyone else who has modified
>> the files.
>>
>> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
>>
>> > Have these files made it into a release?
>> >
>> > On Dec 13, 2017 10:18, "Niels Basjes" <ni...@basjes.nl> wrote:
>> >
>> >> Zoltan,
>> >>
>> >> Because the copyright notice now says you own it I guess the best way
>> to
>> >> approach this is is when you put up a pull request with all those files
>> >> files having a new license header.
>> >> That way it is clear that you made the license switch. I think this
>> should
>> >> be a separate jira to document this clearly.
>> >>
>> >> What do you guys think about this approach?
>> >>
>> >> @Nandor / Gabor: I'll put up a ticket that we should run rat much more
>> >> often (for both 1.8 and master). (i.e. no longer only in separate
>> profile
>> >> of maven)
>> >>
>> >> Niels Basjes
>> >>
>> >>
>> >> On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas
>> >> <zolyfar...@yahoo.com.invalid
>> >> > wrote:
>> >>
>> >> > Hi Niels, the license is a mistake made by me.
>> >> > Those files were based from my work on spf4j-avro which is currently
>> >> dual
>> >> > licensed with LGPL and Apache .
>> >> >
>> >> > We should just replace the license headers with the appropriate
>> Apache
>> >> > header.
>> >> > Let me know if you need me to do anything.
>> >> >
>> >> > Thank you
>> >> >
>> >> > --z
>> >> >
>> >> > > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
>> >> > >
>> >> > > Hi all,
>> >> > >
>> >> > > I was going through the codebase and I found that several files are
>> >> not
>> >> > > Apache licensed.
>> >> > >
>> >> > > https://github.com/apache/avro/tree/master/lang/java/
>> >> > compiler/src/main/java/org/apache/avro/compiler/schema
>> >> &g

[jira] [Commented] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290705#comment-16290705
 ] 

Niels Basjes commented on AVRO-2118:


[~zolyfarkas]: These are the 4 remaining files
  
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitorAction.java
  
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitor.java
  lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/Schemas.java
  
lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/CloningVisitor.java



> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>Assignee: Zoltan Farkas
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes reassigned AVRO-2118:
--

Assignee: Zoltan Farkas  (was: Niels Basjes)

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>Assignee: Zoltan Farkas
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2119) Run rat tool as part of the regular (java) build.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2119:
---
Fix Version/s: 1.8.3
   1.9.0

> Run rat tool as part of the regular (java) build.
> -
>
> Key: AVRO-2119
> URL: https://issues.apache.org/jira/browse/AVRO-2119
> Project: Avro
>  Issue Type: Improvement
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>
> On a regular basis files get in that do not meet the licensing requirements.
> The root casue seems to be that in order to do the check you need to run the 
> Apache Rat too manually (which many forget).
> This must be dome much more often.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2118:
---
Fix Version/s: (was: 1.8.3)

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2119) Run Apache Rat check on every java build

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2119:
---
Summary: Run Apache Rat check on every java build  (was: Run rat tool as 
part of the regular (java) build.)

> Run Apache Rat check on every java build
> 
>
> Key: AVRO-2119
> URL: https://issues.apache.org/jira/browse/AVRO-2119
> Project: Avro
>  Issue Type: Improvement
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>
> On a regular basis files get in that do not meet the licensing requirements.
> The root casue seems to be that in order to do the check you need to run the 
> Apache Rat too manually (which many forget).
> This must be dome much more often.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2118:
---
Fix Version/s: 1.8.3
   1.9.0

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AVRO-2119) Run rat tool as part of the regular (java) build.

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2119:
--

 Summary: Run rat tool as part of the regular (java) build.
 Key: AVRO-2119
 URL: https://issues.apache.org/jira/browse/AVRO-2119
 Project: Avro
  Issue Type: Improvement
Reporter: Niels Basjes
Assignee: Niels Basjes


On a regular basis files get in that do not meet the licensing requirements.
The root casue seems to be that in order to do the check you need to run the 
Apache Rat too manually (which many forget).

This must be dome much more often.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

I had a closer look at the code base.

Most important:
1) I have found these files only in the master branch.
2) I checked both release 1.8.2 and 1.7.7 and in these files are NOT
present in any of those releases. (
So we're ok on this part.

I have found exactly 2 files with this problem:
./lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitorAction.java
./lang/java/compiler/src/main/java/org/apache/avro/compiler/schema/SchemaVisitor.java

I have found 1 additional commit that touches these two files:

https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9

commit 9132015450a2ad6f56cd582b393e8f1b8df573c9
> Author: Thiruvalluvan M G <th...@startsmartlabs.com>
> AuthorDate: Sun Apr 30 21:02:02 2017 +0530
> Commit: Thiruvalluvan M G <th...@startsmartlabs.com>
> CommitDate: Sun Apr 30 23:31:29 2017 +0530
> Added more tests and fixed a couple of bugs. Also formatted the code


In both these files the only changes are:
- Removing the author tag
- Whitespace changes.

See:
https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9#diff-d0adffb4097a1e43917fd5c3f2aae1ab
https://github.com/apache/avro/commit/9132015450a2ad6f56cd582b393e8f1b8df573c9#diff-ced3f0d25217ef63c2f2ea09a8b60e92

@Thiru: To be 100% sure: You agree with changing these two files to the
Apache license?

Niels Basjes


On Wed, Dec 13, 2017 at 6:47 PM, Sean Busbey <bus...@apache.org> wrote:

> In addition to Zoltan we'll need to confirm anyone else who has modified
> the files.
>
> On Dec 13, 2017 11:46, "Sean Busbey" <bus...@apache.org> wrote:
>
> > Have these files made it into a release?
> >
> > On Dec 13, 2017 10:18, "Niels Basjes" <ni...@basjes.nl> wrote:
> >
> >> Zoltan,
> >>
> >> Because the copyright notice now says you own it I guess the best way to
> >> approach this is is when you put up a pull request with all those files
> >> files having a new license header.
> >> That way it is clear that you made the license switch. I think this
> should
> >> be a separate jira to document this clearly.
> >>
> >> What do you guys think about this approach?
> >>
> >> @Nandor / Gabor: I'll put up a ticket that we should run rat much more
> >> often (for both 1.8 and master). (i.e. no longer only in separate
> profile
> >> of maven)
> >>
> >> Niels Basjes
> >>
> >>
> >> On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas
> >> <zolyfar...@yahoo.com.invalid
> >> > wrote:
> >>
> >> > Hi Niels, the license is a mistake made by me.
> >> > Those files were based from my work on spf4j-avro which is currently
> >> dual
> >> > licensed with LGPL and Apache .
> >> >
> >> > We should just replace the license headers with the appropriate Apache
> >> > header.
> >> > Let me know if you need me to do anything.
> >> >
> >> > Thank you
> >> >
> >> > --z
> >> >
> >> > > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >> > >
> >> > > Hi all,
> >> > >
> >> > > I was going through the codebase and I found that several files are
> >> not
> >> > > Apache licensed.
> >> > >
> >> > > https://github.com/apache/avro/tree/master/lang/java/
> >> > compiler/src/main/java/org/apache/avro/compiler/schema
> >> > >
> >> > > Some of these files do not have a copyright block (fixable), yet
> some
> >> > have
> >> > > this:
> >> > >
> >> > > /*
> >> > >
> >> > > * Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
> >> > > *
> >> > > * This library is free software; you can redistribute it and/or
> >> > > * modify it under the terms of the GNU Lesser General Public
> >> > > * License as published by the Free Software Foundation; either
> >> > > * version 2.1 of the License, or (at your option) any later version.
> >> > > *
> >> > > * This library is distributed in the hope that it will be useful,
> >> > > * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> > > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >> > > * GNU General Public License for more details.
> >> > > *
> >> > > * You should have received a copy of the GNU Lesser General Public
> >> > > * License along with this program; if not, write to the Free
> Software
> >> > > * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
> >> 02111-1307,
> >> > USA.
> >> > > */
> >> > >
> >> > > And according to https://www.apache.org/legal/r
> >> esolved.html#category-x
> >> > the
> >> > > LGPL is not allowed to be included.
> >> > >
> >> > > How do we fix this problem?
> >> > >
> >> > > --
> >> > > Best regards / Met vriendelijke groeten,
> >> > >
> >> > > Niels Basjes
> >> >
> >> >
> >>
> >>
> >> --
> >> Best regards / Met vriendelijke groeten,
> >>
> >> Niels Basjes
> >>
> >
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Zoltan,

Because the copyright notice now says you own it I guess the best way to
approach this is is when you put up a pull request with all those files
files having a new license header.
That way it is clear that you made the license switch. I think this should
be a separate jira to document this clearly.

What do you guys think about this approach?

@Nandor / Gabor: I'll put up a ticket that we should run rat much more
often (for both 1.8 and master). (i.e. no longer only in separate profile
of maven)

Niels Basjes


On Wed, Dec 13, 2017 at 4:37 PM, Zoltan Farkas <zolyfar...@yahoo.com.invalid
> wrote:

> Hi Niels, the license is a mistake made by me.
> Those files were based from my work on spf4j-avro which is currently dual
> licensed with LGPL and Apache .
>
> We should just replace the license headers with the appropriate Apache
> header.
> Let me know if you need me to do anything.
>
> Thank you
>
> --z
>
> > On Dec 13, 2017, at 8:14 AM, Niels Basjes <ni...@basjes.nl> wrote:
> >
> > Hi all,
> >
> > I was going through the codebase and I found that several files are not
> > Apache licensed.
> >
> > https://github.com/apache/avro/tree/master/lang/java/
> compiler/src/main/java/org/apache/avro/compiler/schema
> >
> > Some of these files do not have a copyright block (fixable), yet some
> have
> > this:
> >
> > /*
> >
> > * Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
> > *
> > * This library is free software; you can redistribute it and/or
> > * modify it under the terms of the GNU Lesser General Public
> > * License as published by the Free Software Foundation; either
> > * version 2.1 of the License, or (at your option) any later version.
> > *
> > * This library is distributed in the hope that it will be useful,
> > * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > * GNU General Public License for more details.
> > *
> > * You should have received a copy of the GNU Lesser General Public
> > * License along with this program; if not, write to the Free Software
> > * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307,
> USA.
> > */
> >
> > And according to https://www.apache.org/legal/resolved.html#category-x
> the
> > LGPL is not allowed to be included.
> >
> > How do we fix this problem?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Updated] (AVRO-2117) Overall cleanup of code

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2117:
---
Summary: Overall cleanup of code  (was: Overall cleanup of java code)

> Overall cleanup of code
> ---
>
> Key: AVRO-2117
> URL: https://issues.apache.org/jira/browse/AVRO-2117
> Project: Avro
>  Issue Type: Improvement
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
>
> When opening Avro in my IDE I see lots of warnings and notifications that are 
> easy to fix.
> I'm going to pick up several types of those issues (only on master / 1.9.0 !)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2117) Overall cleanup of java code

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2117:
---
Summary: Overall cleanup of java code  (was: Overall cleanup of code)

> Overall cleanup of java code
> 
>
> Key: AVRO-2117
> URL: https://issues.apache.org/jira/browse/AVRO-2117
> Project: Avro
>  Issue Type: Improvement
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
>
> When opening Avro in my IDE I see lots of warnings and notifications that are 
> easy to fix.
> I'm going to pick up several types of those issues (only on master / 1.9.0 !)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Licensing problem (LGPL in codebase!!)

[Niels Basjes]

Hi all,

I was going through the codebase and I found that several files are not
Apache licensed.

https://github.com/apache/avro/tree/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/schema

Some of these files do not have a copyright block (fixable), yet some have
this:

/*

* Copyright (c) 2001 - 2016, Zoltan Farkas All Rights Reserved.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
*/

And according to https://www.apache.org/legal/resolved.html#category-x the
LGPL is not allowed to be included.

How do we fix this problem?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Updated] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niels Basjes updated AVRO-2118:
---
Summary: Rat tool fails over several files.  (was: Add missing license 
headers)

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AVRO-2118) Rat tool fails over several files.

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289220#comment-16289220
 ] 

Niels Basjes commented on AVRO-2118:


A {{./build.sh rat}} currently fails.

> Rat tool fails over several files.
> --
>
> Key: AVRO-2118
> URL: https://issues.apache.org/jira/browse/AVRO-2118
> Project: Avro
>  Issue Type: Bug
>        Reporter: Niels Basjes
>    Assignee: Niels Basjes
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AVRO-2118) Add missing license headers

[Niels Basjes (JIRA)]

Niels Basjes created AVRO-2118:
--

 Summary: Add missing license headers
 Key: AVRO-2118
 URL: https://issues.apache.org/jira/browse/AVRO-2118
 Project: Avro
  Issue Type: Bug
Reporter: Niels Basjes
Assignee: Niels Basjes






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)