Re: Volunteering for Avro 1.11.4 and 1.12.0 releases

[Driesprong, Fokko]

Hey JB,

Thanks for taking the lead here. Let me know if there is anything I can
help with.

With regards to the Java version bump, I'm open to going to JDK17 directly.
This is also the case for Spark 4 as well:
https://spark.apache.org/docs/3.5.1/building-spark.html#apache-maven

Kind regards,
Fokko

Op di 16 apr 2024 om 15:56 schreef Jean-Baptiste Onofré :

> Hi,
>
> follow up on this discussion, I created
> https://github.com/apache/avro/pull/2855
>
> I'm working on some new PRs heading to the releases :)
>
> Thanks!
> Regards
> JB
>
> On Tue, Apr 16, 2024 at 10:55 AM Martin Grigorov 
> wrote:
> >
> > On Tue, 16 Apr 2024 at 11:21, Jean-Baptiste Onofré 
> wrote:
> >
> > > Hi Martin,
> > >
> > > (finally I received the email from the list cleanly, w00t ;))
> > >
> > > Thanks for pointing the previous discussion.
> > >
> > > I think 1.12.0 is OK as it's mostly the Java part that gonna be
> > > impacted. I think 2.0.0 would make more sense for a spec change
> > > impacting all languages.
> > >
> > > So, I'm proposing:
> > > - Avro 1.12.0 with Java 11 by default in Java
> > > - Avro 1.11.4 still on Java 8.
> > > It makes sense to maintain both to give users the time to update (and
> > > encourage them to do so :)).
> > >
> > > Thoughts ?
> >
> >
> > +1
> >
> >
> > >
> > > Regards
> > > JB
> > >
> > > On Tue, Apr 16, 2024 at 9:54 AM Martin Grigorov 
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > There were few discussions about this some months ago, e.g.
> > > > https://lists.apache.org/thread/bd39zhk655pgzfctq763vp3z4xrjpx58
> > > > I think everyone agreed that 1.12.0 could be bumped to Java 11+!
> > > > https://lists.apache.org/thread/b8d7cjywz6mpj81bhbx5tz6flbvwzsdk -
> here
> > > > Niels even suggested to bump the version to 2.0.0, but since Avro is
> > > > multi-language repo I don't think changing the version to 2.x for all
> > > langs
> > > > makes sense.
> > > > Users of Java 8 will have to stay with 1.11.x. There was another
> > > discussion
> > > > at dev@ about maintaining both branches for some time.
> > > >
> > > > Martin
> > > >
> > > > On Mon, Apr 15, 2024 at 4:11 PM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > NB: I'm sending this email with my regular email address ;)
> > > > >
> > > > > I'm doing a full pass to prepare Avro 1.11.4 and 1.12.0 releases.
> > > > >
> > > > > I have a question for you: on the main branch (e.g. 1.12.0), I
> propose
> > > > > to update the Java source/release target version to 11 (instead of
> > > > > 1.8). I'm testing it right now, and so far so good.
> > > > >
> > > > > I'm also preparing some updates and cleanup. I will open the PRs
> today.
> > > > >
> > > > > Thanks !
> > > > > Regards
> > > > > JB
> > > > >
> > >
>

Re: [PROPOSAL] Apache Avro 1.11.4 release

[Driesprong, Fokko]

Sorry for the late reply JB.

I'll do a review of #2642 . I
already tested this against Iceberg locally and didn't see any issues
there. I would love to get in #3871
 as well.

Cheers, Fokko

Op di 2 apr 2024 om 16:57 schreef Jean-Baptiste Onofré :

> Hi folks,
>
> As discussed on this thread, I would like to move forward on 1.12.0
> release.
> Thoughts ?
>
> I'm doing a pass on the issues.
>
> PS: I have a weird behavior with the mailing list, I might have to
> re-subscribe.
>
> Regards
> JB
>
> On 2024/03/14 08:53:05 Jean-Baptiste Onofré wrote:
> > Hey Martin
> >
> > Awesome !
> >
> > Happy to help with 1.12.0 if needed :)
> >
> > Thanks !
> > Regards
> > JB
> >
> > On 2024/03/14 08:46:05 Martin Grigorov wrote:
> > > Hi,
> > >
> > > On Wed, Mar 13, 2024 at 7:16 PM Jean-Baptiste Onofré 
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Recently, we upgraded to commons-compress 1.26.0 (on main).
> > > > commons-compress 1.26.0 fixes several CVEs (CVE-2024-25710,
> > > > CVE-2024-26308, CVE-2023-42503).
> > > >
> > > > Would it be possible to release Avro 1.11.4 (that will include
> > > > commons-compress update) ?
> > > >
> > >
> > > Actually a few weeks ago there was a proposal to release 1.12.0!
> > > https://lists.apache.org/thread/b8p8jkxx8f23yrss14q0y0ptdlttnxp4
> > >
> > >
> > > >
> > > > I can help on the release if there are no objections.
> > >
> > >
> > > > Thanks !
> > > > Regards
> > > > JB
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Avro 1.12.0

[Driesprong, Fokko]

Hey Oscar,

Thanks for providing the context of the new schema parser API. It would be
great to get the subset in. Let me know when you have it ready so we can
move forward with the release.

Kind regards,
Fokko

Op wo 14 feb 2024 om 08:58 schreef Oscar Westra van Holthe - Kind <
os...@westravanholthe.nl>:

> Hi,
>
> For 1.12, I would really like to include a change to the (new)
> SchemaParser API that was merged previously. The current version can return
> invalid Schemata (in case of missing references), which can be resolved
> with a special resolve(…) method. I’t like to return a ParseResult instead
> that enforces this, and thus ensures no invalid schemata can be returned.
>
> I’ll make a new PR, a subset of #2642 <
> https://github.com/apache/avro/pull/2642>, to include just that.
>
>
> Kind regards,
> Oscar
>
> --
> Oscar Westra van Holthe - Kind 
>
> > On 12 Feb 2024, at 22:53, Fokko Driesprong  wrote:
> >
> > I compiled the latest snapshot against Iceberg and Spark today, and it
> all
> > succeeded. I think it is good to also extend the
> BlockingDirectBinaryEncode
> > r to support nested
> structures in
> > case we want to use it to write arbitrary Avro as well (next to the
> > metadata which is the first objective). Let me know if there is anything
> > that you would like to get into 1.12.
> >
> > Kind regards,
> > Fokko
> >
> > Op ma 12 feb 2024 om 13:57 schreef Martin Grigorov  >:
> >
> >> On Mon, Feb 12, 2024 at 11:59 AM Fokko Driesprong 
> >> wrote:
> >>
> >>> Hey everyone,
> >>>
> >>> I wanted to bring the release up again. We now have Nanosecond support,
> >>> Fixed bytes encoded UUIDs, and the directBlockingBinaryEncoders that
> >> we're
> >>> on the Iceberg side very much looking forward to.
> >>>
> >>> How about running a 1.12.0 release? After that, we can discuss the 2.0
> >>> release with major new features (dropping Java 8, new schema parser,
> >> etc).
> >>> WDYT?
> >>>
> >>
> >> Agreed!
> >>
> >>
> >>>
> >>> Kind regards,
> >>> Fokko Driesprong
> >>>
> >>> Op do 28 dec 2023 om 14:22 schreef Fokko Driesprong  >:
> >>>
>  Hey everyone,
> 
>  Hope everyone had a great Christmas, and thanks for all the replies
> >> here.
> 
>  Good idea. I would really like to include the changes necessary to
> > complete AVRO-3666 (the new SchemaParser class).
> 
> 
>  I started a review on this one, but haven't finished yet. My main
> >> concern
>  was around API compatibility. I noticed that a public API's changed,
> >> and
>  tried to run the Iceberg tests against the PR to catch any breaking
>  changes. When doing so, I noticed that the checks around logical types
>  became more strict. In Iceberg we extend the Avro specification by
> >>> storing UUIDs
>  as fixed binary . I've raised
> a
>  discussion
>   on
> >>> the
>  mailing-list, and also Christophe created a PR
>   to add this to the Java
> SDK
>  (thank you for that!). I'll finish the PR of the new parser, and I
> >> would
>  also love to get the UUID support in there.
> 
>  Ideally, I'd also like to include a counterpart for formatting
> schemas,
> >>> by that
> > can also wait until 1.12.1.
> 
> 
>  Typically we don't backport features to earlier versions. 1.11 was
>  released in October 2021 which is way back. I'm in favor of doing a
> >> major
>  release more often.
> 
>  We don't have a strict policy for new features in patch releases so
> why
> >>> don't
> > we get out a new 1.11.x release with the latest changes
> >>> (DirectBlockingBinaryEncoder
> > + nanoseconds + others) and wait for the 1.12.0 release for the
> >> plugin +
> > any maintenance policy changes (which versions will be maintained,
> >> etc)
> > and other eventual backward incompatible changes.
> 
> 
>  Adding nanoseconds in a patch version sounds awkward to me, and I
> don't
>  think this aligns with the public consensus around what should be
> >>> included
>  in a patch release. I would just cut master as 1.12.0 and then once we
> >>> get
>  the plugin in we can do a 1.13 release. WDYT?
> 
>  Any volunteer to be the release manager?
> 
> 
>  I'm happy to run the release!
> 
>  It's really irritating that we don't have a "clean" plan for updating
> >> the
> > website with new releases :O   I'm really pleased with the new
> >> website,
> > and I think we have almost all of the elements in place to get this
> > right.
> >
> 
>  Ryan S., can you elaborate on the gaps here?
> 
>  Thanks everyone, and wish y'all a great 2024!
> 
>  Kind regards,
>  Fokko
> 
> 
> 
>  Op do 28 dec 2023 om 01:22 schreef David M. Carr :
> 
> > Just 

Re: [VOTE][LAZY] Support for UUIDs encoded as Fixed[16]

[Driesprong, Fokko]

I went ahead and merged the spec-update. Thanks everyone for the input, and
I'm looking forward to getting this out!

Kind regards, Fokko

Op wo 24 jan 2024 om 10:55 schreef Oscar Westra van Holthe - Kind <
opw...@apache.org>:

> +1 for UUIDs encoded as fixed from me as well.
>
> A already remarked, an automatic conversion from strings would be very
> helpful. However, this has a big influence on schema resolution and should
> be dealt with separately.
>
>
> Kind regards,
> Oscar
>
> --
> Oscar Westra van Holthe - Kind 
>
> Op di 23 jan. 2024 03:27 schreef Jack Klamer :
>
> > +1 for what it's worth.
> >
> > Although, it would be nice to also officially make fixed promotable
> strings
> > as well. Allowing old UUID string schema to read newer UUID encodings.
> >
> > Best,
> > Jack
> >
> > On Mon, Jan 22, 2024 at 1:40 PM Ryan Blue  wrote:
> >
> > > +1 for storing as 16-byte fixed.
> > >
> > > On Mon, Jan 22, 2024 at 1:44 AM Martin Grigorov 
> > > wrote:
> > >
> > > > I just want to add that it would be nice to add an interop test, e.g.
> > an
> > > > Avro file written by one SDK should be readable by the other SDKs.
> > > >
> > > > On Fri, Jan 19, 2024 at 11:58 AM Driesprong, Fokko
> >  > > >
> > > > wrote:
> > > >
> > > > > Hey everyone,
> > > > >
> > > > > Last December I proposed support for encoding UUIDs as fixed[16]
> > > > > <https://lists.apache.org/thread/h4lvc2lk194om6cjm76wzosm22cop2xw
> >,
> > as
> > > > > this
> > > > > has several advantages
> > > > > <
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/16_oSWrEM7AFUCTe0uuraAEHxywezLfoEz5ahzwvhGUk/edit#heading=h.43xuauwfk7ow
> > > > > >.
> > > > > I know that this was over the holidays, so I would like to revisit
> > > this.
> > > > > Let me know if there are any questions or concerns, or feel free to
> > > jump
> > > > in
> > > > > on the PR <https://github.com/apache/avro/pull/2672>. There are
> also
> > > > > implementations already for Java
> > > > > <https://github.com/apache/avro/pull/2652> (thanks
> > > > > Christophe!), and one for Rust (thanks Martin!). If there are no
> > > > concerns,
> > > > > I would like to move this forward next week, and also see if we can
> > > start
> > > > > the release process.
> > > > >
> > > > > Kind regards,
> > > > > Fokko
> > > > >
> > > >
> > >
> > >
> > > --
> > > Ryan Blue
> > > Tabular
> > >
> >
>

[VOTE][LAZY] Support for UUIDs encoded as Fixed[16]

[Driesprong, Fokko]

Hey everyone,

Last December I proposed support for encoding UUIDs as fixed[16]
, as this
has several advantages
.
I know that this was over the holidays, so I would like to revisit this.
Let me know if there are any questions or concerns, or feel free to jump in
on the PR . There are also
implementations already for Java
 (thanks
Christophe!), and one for Rust (thanks Martin!). If there are no concerns,
I would like to move this forward next week, and also see if we can start
the release process.

Kind regards,
Fokko

Re: [DISCUSS] Extending the UUID logical type with Fixed[16]

[Driesprong, Fokko]

Hey everyone,

Happy New Year! Best wishes for 2024 for you and your family.

I went ahead and created a PR for the spec change:
https://github.com/apache/avro/pull/2672 Let me know if there are any
questions or concerns.

Kind regards,
Fokko

Op vr 22 dec 2023 om 14:52 schreef Fokko Driesprong :

> Hi Martin and Scott,
>
> Thanks for the question, and that's a good one. I would suggest:
>
> {
>
>   "type": "fixed",
>
>   "size": 16,
>
>   "logicalType": "uuid"
>
> }
>
> This is in line with the other logicalTypes. For example with date:
>
> {
>   "type": "int",
>   "logicalType": "date"
> }
>
> If you don't support the date, you can still read the int itself (days
> since Epoch).
>
> I've added a schema example to the Google doc and created a PR
>  to clarify the current
> situation.
>
> I am curious about what you guys think of the proposed JSON-type
> representation.
>
> Kind regards,
> Fokko
>
>
> Op vr 22 dec 2023 om 14:25 schreef Scott Belden :
>
>> I think you'd have to go with something like one of the first two options
>> (something in the schema) rather than some flag in a library. The problem
>> with an flag in a library is if someone has an avro file they want to
>> deserialize, they might not know if it was encoded with uuids as bytes or
>> strings and they'd be left with guessing one and trying again with the
>> second if the first failed which would not be a pleasant experience.
>>
>> -Scott
>>
>> On Fri, Dec 22, 2023 at 5:00 AM Martin Grigorov 
>> wrote:
>>
>> > Hi,
>> >
>> > How would the application tell Avro what storage type to use - String or
>> > bytes ?
>> > - new logical type ? e.g. "logicalType": "uuid-bytes"
>> > - extra attribute ? e.g. { ..., "logicalType": "uuid", "storage-type":
>> > "bytes" }
>> > - global switch that tells the library to always use "string" or "bytes"
>> > for all UUIDs ?
>> > - ...
>> >
>> > Martin
>> >
>> > On Fri, Dec 22, 2023 at 10:49 AM Fokko Driesprong 
>> > wrote:
>> >
>> > > Hey everyone,
>> > >
>> > > For Iceberg we're using UUIDs in Avro and we're storing them as
>> binary,
>> > > rather than a string. This has several advantages such as more compact
>> > > storage, more efficient reading, and more efficient skipping. For more
>> > > details, please check out the doc that I've created
>> > > <
>> > >
>> >
>> https://docs.google.com/document/d/16_oSWrEM7AFUCTe0uuraAEHxywezLfoEz5ahzwvhGUk/edit#heading=h.43xuauwfk7ow
>> > > >
>> > > (and feel free to comment). Also created AVRO-3918
>> > >  on Jira to track
>> this.
>> > >
>> > > Looking forward to hearing from y'all!
>> > >
>> > > Kind regards and happy holidays,
>> > >
>> > > Fokko Driesprong
>> > >
>> >
>>
>

Re: [ANNOUNCE] New PMC member: Martin Grigorov

[Driesprong, Fokko]

Congratulations Martin, thanks for all the hard work and well deserved!

Cheers, Fokko

Op wo 14 sep. 2022 om 03:39 schreef Oscar Westra van Holthe - Kind <
os...@westravanholthe.nl>:

> Congratulations Martin!  This is wel deserved.
>
>
>
>
> Oscar Westra van Holthe - Kind 
>
> Op di 13 sep. 2022 19:34 schreef Ryan Skraba :
>
> > Hi folks!
> >
> > On behalf of the Apache Avro PMC, I am pleased to announce that Martin
> > Grigorov has accepted our invitation to become a PMC member.  He has
> > has been active, reliable and responsive to the community and a solid
> > contributor to various SDKs, bringing well-thought out reviews and
> > comments to both old and new PRs and JIRA.  He definitely stepped up
> > for the website refactoring and preparing for the 1.11.1 release!
> >
> > Please join me in welcoming Martin to the Avro PMC!
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@avro.apache.org to
> > let us know.
> >
> > All my best, Ryan
> >
>

Re: Avro Big Data Question from a developer

[Driesprong, Fokko]

Hi Gokay,

That's some CSV file. That will probably be much smaller in Avro.

An Avro file is a so-called Object Container File. This was implemented in
the MapReduce era to make sure that the workload for each of the workers is
roughly the same. Which makes it easier to tune the memory requirements. An
Avro file actually contains one or more containers, which are individually
compressed. The blocks are separated using the synchronization marker. More
info can be found here: https://avro.apache.org/docs/current/spec.html

Also, the C# code gives some good pointers:
https://github.com/apache/avro/blob/42edbd721fedc0ed6cde89ab3b64a9ac606aa74f/lang/csharp/src/apache/main/File/DataFileWriter.cs

You could read (and uncompress) the blocks one by one to keep the memory
constant or use this to parallelize the reading of the file, which might
significantly improve the throughput of the application.

Kind regards,
Fokko Driesprong



Op do 12 mei 2022 om 08:16 schreef Gokay Tosunoglu
:

> Hi there,I have a C# appilcation which deals with big data. For example
> like 1 file is more than 100 GB. This big file is CSV type, I am reading
> this customer data by buffers and checking end of line to read next
> buffer.I want to support avro too, but i couldn't find how can i read a 100
> GB avro file and/or how to divide file into buffers.Can any of you send me
> a way to do it or a sample code maybe?Thanks in advance.Gokay Tosunoglu

Re: [VOTE] Accept donation of Avro Rust implementation

[Driesprong, Fokko]

Cool stuff! Checked the repo, and it looks very good!

+1 (binding) from my side!

Cheers, Fokko

Op ma 24 mei 2021 om 11:34 schreef Ismaël Mejía :

> Dear all,
>
> The developers of
> https://github.com/flavray/avro-rs/
>
> have been in touch with some members of the Apache Avro Project Management
> Committee (PMC). Based on these discussions, it is being proposed to donate
> this
> Avro Rust implementation into the Apache Avro project to continue
> developing it
> together with the Apache Avro community following the Apache Software
> Foundation
> (ASF) guidelines.
>
> This vote is to determine if the Avro PMC is in favor of accepting this
> donation. If the vote passes, the PMC and the authors of the code will work
> together to complete the ASF IP Clearance process
> (http://incubator.apache.org/ip-clearance/) and import the Rust
> implementation
> into the Avro codebase.
>
> +1 : Accept contribution of Rust implementation
>  0 : No opinion
> -1 : Reject contribution because...
>
> Here is my vote: +1
>
> The vote will be open for at least 72 hours.
>
> Thanks,
> Ismaël
>

Re: [DISCUSS] Automatically Merge Dependency Updates

[Driesprong, Fokko]

Hi Ismaël,

Thanks for working on this! I haven't seen many projects doing this, but I
like the idea! I'm all in favor of this!

Cheers, Fokko

Op zo 31 jan. 2021 om 14:06 schreef Ismaël Mejía :

> Since I opened this discussion ~1 month ago I had the time to revisit
> the process and my 'new' conclusion is that self merging is probably
> going too much into automation at this point. I updated the dependency
> updates to run once per week and merged most of the Java PRs, so it
> should be less noise starting from now. So let's better not do this,
> other ecosystems change but it is not as fast as was worried about.
>
> We have however many open PRs for Ruby, C# and JS, so if any of the
> contributors can sit and help with reviews/merges it would be great.
> Notice that the JS case in particular requires probably to sync the
> dependency updates because of conflicts between dependencies.
>
>
> On Mon, Jan 4, 2021 at 11:59 AM Ismaël Mejía  wrote:
> >
> > We enabled recently dependabot to automate dependency upgrades [1].
> Results so
> > far seem good including having new CVEs alerts!
> >
> > Maybe we could automate further by auto merging the PRs given some
> conditions
> > like a whitelist of dependencies that are now stable enough and when
> tests are
> > green we shall not have problems.
> >
> > It seems github now has an option to do this [2] so I was wondering what
> other
> > members of the community thought and if you see any possible
> issue/drawbacks
> > before starting any work on this.
> >
> > [1]
> https://lists.apache.org/thread.html/r2a175f8b96dd7a556cf1b7438a5c8efcacdd4a06080926142734%40%3Cdev.avro.apache.org%3E
> > [2]
> https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/automatically-merging-a-pull-request
>

Re: GitHub Actions

[Driesprong, Fokko]

I prefer to do this in separate PR's, to avoid huge changes. Also, with GA
the CI will be more stable :)

Cheers, Fokko

Op zo 20 dec. 2020 om 20:58 schreef Michael A. Smith :

> On Sun, Dec 20, 2020 at 6:13 AM Driesprong, Fokko 
> wrote:
> >
> > Hi Michael,
> >
> > Thanks for bringing this up. I think it would be a great idea. I don't
> have
> > anything against Travis, but I like GA a lot. For example, their
> container
> > support is much better, and the syntax is cleaner. It also integrates
> > extremely well with Github itself. This can be nice if we want to have
> some
> > flow someday.
> >
> > When it comes to Apache Yetus, I must admit, I've implemented Yetus at
> the
> > time, but I'm not super familiar with the tool. I think the current
> > implementation doesn't get the value out of it that it promises to do.
> > Also, one of the reasons that the implementation is far from optimal
> > because it doesn't fit the project that well. I would suggest to remove
> it.
> >
> > One thing that concerns me a bit is the scattering of the commands in the
> > GA yml files and the build.sh. I would suggest moving everything into one
> > place. In the case of Github Actions, you can also run it easily locally:
> > https://github.com/nektos/act
>
> That sounds great. Is this something we can do iteratively, or did you
> have in mind doing it all in the one PR?
>
> >
> > Cheers, Fokko
> >
> >
> > Op zo 20 dec. 2020 om 06:05 schreef Michael A. Smith <
> mich...@smith-li.com>:
> >
> > > I created a PR to implement our tests in GitHub actions. I'd like to
> > > know if other folks are interested in me pursuing this further and
> > > replacing the Travis/Yetus build system.
> > >
> > > Some data:
> > > - In its current configuration, a Travis build that doesn't fail takes
> > > around 70 minutes.
> > > - Travis usually fails, often for reasons unrelated to a particular PR.
> > > - Understanding why it fails requires spelunking through thousands of
> > > lines of log files.
> > > - Casual contributors are disinclined to set up Travis for their
> > > forks, and can end up triggering multiple travis builds in an Apache
> > > PR to track down a bug.
> > > - The single Docker megafile tightly couples every language toolchain,
> > > so testing multiple language versions is difficult.
> > >
> > > All of these problems can be fixed within the Travis/Yetus build
> > > system (except maybe the "casual contributors" thing), I'm sure. But I
> > > have looked into it before and haven't been able to figure it out.
> > >
> > > Here's what I've done with GitHub actions:
> > > - Jobs are isolated by lang/* and only trigger when a change touches
> > > that language. Even if a problem is causing, say, Ruby tests to fail
> > > in master, PHP contributions can still make it through.
> > > - The tests are run in parallel, both across languages and within,
> > > across multiple language versions and interop and unit tests.
> > > - The slowest jobs (the Java tests) take 15 minutes. The worst case
> > > test run (aside from an outage) will probably be under 20 minutes, if
> > > we are heavily queued.
> > > - This PR tests java 8 and 11, js using node 10, 11 and 12, php 7.3,
> > > 7.4 and 8, python 3.6-3.9 and pypy3.6 and 3.7. Adding and removing
> > > language implementations is trivial.
> > > - If we merge this PR, anyone who forks the repo will get these
> > > actions in their fork.
> > >
> > > One thing I haven't yet implemented is an action for
> > > share/test/interop/bin/test_rpc_interop.sh. I think I can do that,
> > > too, but I want to know if this can go anywhere before I work on it
> > > more.
> > >
> > > WDYT?
> > >
> > > - Michael
> > >
>

Re: [Vote] to Use GitHub Actions and Disable TravisCI [Was Re: GitHub Actions]

[Driesprong, Fokko]

+1 from my side, a huge fan of Github Actions!

Op ma 28 dec. 2020 om 20:13 schreef Ryan Blue :

> +1
>
> Thanks for working on this. I think it will be a big improvement and will
> give better feedback to contributors.
>
> On Mon, Dec 28, 2020 at 8:37 AM Michael A. Smith 
> wrote:
>
> > I would like to call for a vote to merge
> > https://github.com/apache/avro/pull/1043 -- it's a significant enough
> > change that I think a vote is warranted. It disables TravisCI and does
> > testing via GitHub actions. To recap the benefits from earlier in this
> > thread:
> >
> > 1. TravisCI has been slow and unreliable for us.
> > 2. We can skip testing things that haven't changed and help
> > contributors focus on their goals.
> >
> > That said, the GitHub actions aren't perfect, and I am not an expert
> > in every language implementations' best practices for build and test.
> > At a minimum, this PR does invoke the test commands in build.sh for
> > each lang/* and it also runs the interop tests (which themselves need
> > some TLC).
> >
> > The voting process is here:
> > https://www.apache.org/foundation/voting.html. This is a vote on a
> > code modification, with lazy consensus in effect. If nobody objects
> > within three days, I'll merge it sometime in the afternoon or evening
> > Thursday, Eastern Standard Time.
> >
> > (If a PMC member or committer does want to veto this because of some
> > perceived flaw in the PR implementation, please don't leave us hanging
> > with a non-working Travis implementation. Propose a path forward.)
> >
> > Thanks,
> > Michael
> >
> >
> > On Sun, Dec 20, 2020 at 2:58 PM Michael A. Smith 
> > wrote:
> > >
> > > On Sun, Dec 20, 2020 at 6:13 AM Driesprong, Fokko  >
> > wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > Thanks for bringing this up. I think it would be a great idea. I
> don't
> > have
> > > > anything against Travis, but I like GA a lot. For example, their
> > container
> > > > support is much better, and the syntax is cleaner. It also integrates
> > > > extremely well with Github itself. This can be nice if we want to
> have
> > some
> > > > flow someday.
> > > >
> > > > When it comes to Apache Yetus, I must admit, I've implemented Yetus
> at
> > the
> > > > time, but I'm not super familiar with the tool. I think the current
> > > > implementation doesn't get the value out of it that it promises to
> do.
> > > > Also, one of the reasons that the implementation is far from optimal
> > > > because it doesn't fit the project that well. I would suggest to
> > remove it.
> > > >
> > > > One thing that concerns me a bit is the scattering of the commands in
> > the
> > > > GA yml files and the build.sh. I would suggest moving everything into
> > one
> > > > place. In the case of Github Actions, you can also run it easily
> > locally:
> > > > https://github.com/nektos/act
> > >
> > > That sounds great. Is this something we can do iteratively, or did you
> > > have in mind doing it all in the one PR?
> > >
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > > Op zo 20 dec. 2020 om 06:05 schreef Michael A. Smith <
> > mich...@smith-li.com>:
> > > >
> > > > > I created a PR to implement our tests in GitHub actions. I'd like
> to
> > > > > know if other folks are interested in me pursuing this further and
> > > > > replacing the Travis/Yetus build system.
> > > > >
> > > > > Some data:
> > > > > - In its current configuration, a Travis build that doesn't fail
> > takes
> > > > > around 70 minutes.
> > > > > - Travis usually fails, often for reasons unrelated to a particular
> > PR.
> > > > > - Understanding why it fails requires spelunking through thousands
> of
> > > > > lines of log files.
> > > > > - Casual contributors are disinclined to set up Travis for their
> > > > > forks, and can end up triggering multiple travis builds in an
> Apache
> > > > > PR to track down a bug.
> > > > > - The single Docker megafile tightly couples every language
> > toolchain,
> > > > > so testing multiple language versions is difficult.
> > > > >
> > > > > All of t

Re: GitHub Actions

[Driesprong, Fokko]

Hi Michael,

Thanks for bringing this up. I think it would be a great idea. I don't have
anything against Travis, but I like GA a lot. For example, their container
support is much better, and the syntax is cleaner. It also integrates
extremely well with Github itself. This can be nice if we want to have some
flow someday.

When it comes to Apache Yetus, I must admit, I've implemented Yetus at the
time, but I'm not super familiar with the tool. I think the current
implementation doesn't get the value out of it that it promises to do.
Also, one of the reasons that the implementation is far from optimal
because it doesn't fit the project that well. I would suggest to remove it.

One thing that concerns me a bit is the scattering of the commands in the
GA yml files and the build.sh. I would suggest moving everything into one
place. In the case of Github Actions, you can also run it easily locally:
https://github.com/nektos/act

Cheers, Fokko


Op zo 20 dec. 2020 om 06:05 schreef Michael A. Smith :

> I created a PR to implement our tests in GitHub actions. I'd like to
> know if other folks are interested in me pursuing this further and
> replacing the Travis/Yetus build system.
>
> Some data:
> - In its current configuration, a Travis build that doesn't fail takes
> around 70 minutes.
> - Travis usually fails, often for reasons unrelated to a particular PR.
> - Understanding why it fails requires spelunking through thousands of
> lines of log files.
> - Casual contributors are disinclined to set up Travis for their
> forks, and can end up triggering multiple travis builds in an Apache
> PR to track down a bug.
> - The single Docker megafile tightly couples every language toolchain,
> so testing multiple language versions is difficult.
>
> All of these problems can be fixed within the Travis/Yetus build
> system (except maybe the "casual contributors" thing), I'm sure. But I
> have looked into it before and haven't been able to figure it out.
>
> Here's what I've done with GitHub actions:
> - Jobs are isolated by lang/* and only trigger when a change touches
> that language. Even if a problem is causing, say, Ruby tests to fail
> in master, PHP contributions can still make it through.
> - The tests are run in parallel, both across languages and within,
> across multiple language versions and interop and unit tests.
> - The slowest jobs (the Java tests) take 15 minutes. The worst case
> test run (aside from an outage) will probably be under 20 minutes, if
> we are heavily queued.
> - This PR tests java 8 and 11, js using node 10, 11 and 12, php 7.3,
> 7.4 and 8, python 3.6-3.9 and pypy3.6 and 3.7. Adding and removing
> language implementations is trivial.
> - If we merge this PR, anyone who forks the repo will get these
> actions in their fork.
>
> One thing I haven't yet implemented is an action for
> share/test/interop/bin/test_rpc_interop.sh. I think I can do that,
> too, but I want to know if this can go anywhere before I work on it
> more.
>
> WDYT?
>
> - Michael
>

Re: [ANNOUNCE] Apache Avro 1.10.1 released

[Driesprong, Fokko]

Awesome! Thanks for running the release. Great to see some continuity in
the Avro releases.

Cheers, Fokko

Op vr 4 dec. 2020 om 21:19 schreef Ryan Skraba 

> Please note: I mistakenly sent this same message earlier today with the
> wrong subject!  It is, in fact, 1.10.1 that was released.  My apologies!
>
> The Apache Avro community is pleased to announce the release of Avro
> 1.10.1!
>
> All signed release artifacts, signatures and verification instructions can
> be
> found here: https://avro.apache.org/releases.html
>
> This release includes 33 Jira issues, including some interesting features:
>
> C#: AVRO-2750 Support for enum defaults
> C++: AVRO-2891 Expose last sync offset written on DataFileWriter
> Java: AVRO-2924 SpecificCompiler add 'LocalDateTime' logical type
> Java: AVRO-2937 Expose some missing flags in SpecificCompilerTool
> PHP: AVRO-2096 Fixes to missing functions
> Ruby: AVRO-2907 Ruby schema.single_object_schema_fingerprint is reversed
>
> Migration notes:
> Java: AVRO-2817 Turn off validateDefaults when reading legacy Avro files
> Python: AVRO-2656 avro-python package is now the preferred python3 library
> and
>   avro-python3 is prepared to be deprecated
>
> And of course upgraded dependencies to latest versions, CVE fixes and more:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.1
>
> The link to all fixed JIRA issues and a brief summary can be found at:
> https://github.com/apache/avro/releases/tag/release-1.10.1
>
> In addition, language-specific release artifacts are available:
>
> * C#: https://www.nuget.org/packages/Apache.Avro/1.10.1
> * Java: from Maven Central,
> * Javascript: https://www.npmjs.com/package/avro-js/v/1.10.1
> * Python 2: https://pypi.org/project/avro/1.10.1/
> * Python 3: https://pypi.org/project/avro-python3/1.10.1/
> * Ruby: https://rubygems.org/gems/avro/versions/1.10.1
>
> Thanks to everyone for contributing!
>

Re: [Announce] Please welcome Ryan Skraba to the Apache Avro PMC

[Driesprong, Fokko]

Awesome, well deserved Ryan!

Cheers, Fokko

Op do 17 sep. 2020 om 18:06 schreef Micah Kornfield :

> Congratulations!
>
> On Mon, Sep 14, 2020 at 10:07 AM Sean Busbey  wrote:
>
> > Hi folks!
> >
> > On behalf of the Apache Avro PMC I am pleased to announce that Ryan
> > Skraba has accepted our invitation to become a PMC member. We
> > appreciate Ryan stepping up to take more responsibility in the
> > project.
> >
> > Please join me in welcoming Ryan to the Avro PMC!
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@avro.apache.org to
> > let us know.
> >
> > --
> > busbey
> >
>

Re: Type Hints for Python!

[Driesprong, Fokko]

Cool stuff Michael,

I'll pick up some files and start adding type-hints :)

Cheers, Fokko

Op ma 17 aug. 2020 om 14:40 schreef Michael A. Smith :

> We've had mypy, the Python type checker, running in Travis-CI on
> lang/py for a little while. However, mypy doesn't actually check code
> unless it has type hints, so it has been doing very little. Now that
> we are only supporting Python 3.5 and up, we can add real type hints!
>
> I'm excited about this, because I think it will really improve the
> Python implementation. There are a fair amount of Liskov violations
> and other places where type theory rules have been bent or broken in
> the Python implementation, so there will be places where we need to
> decide if we want to make an API breaking change, or ignore an
> inconsistency.
>
> If you want to see what mypy is able to check right now, you can
> install mypy and lxml and run mypy --html-report mypy-html and open
> mypy-html/index.html in your browser.
>
> If you can find the time to open a PR against AVRO-2921 and introduce
> type hints in a single module or test, please give it a shot. If the
> type checker finds a bug as a result, please report the bug.
>
> Happy Hinting!
>

Re: [VOTE] Drop Python 2 support

[Driesprong, Fokko]

Thanks Michael,

Also, Spark is dropping support of 3.5 and below :)

Cheers, Fokko

Op vr 14 aug. 2020 om 17:04 schreef Michael A. Smith :

> Based on these votes and plenty of time passing I will feel
> comfortable making changes that are only compatible with python 3.5
> and up.
>
> Thank you all for your responses!
>
> On Tue, Jul 28, 2020 at 11:49 PM Micah Kornfield 
> wrote:
> >
> > +1 (non-binding)
> >
> > On Tue, Jul 28, 2020 at 9:30 AM Cris Ewing 
> > wrote:
> >
> > > +1
> > >
> > > (to be clear, I'm not certain my vote counts for anything other than
> > > advisory)
> > >
> > > yours
> > >
> > > c
> > >
> > > On Tue, Jul 28, 2020 at 6:15 AM Michael A. Smith  >
> > > wrote:
> > >
> > > > Python 2 has been officially obsolete since January 1. I propose we
> drop
> > > > support for it.
> > > >
> > > > If we decide to continue supporting it, note that pip, the (main)
> Python
> > > > package management tool, will itself only support python3 in its next
> > > > release. So installing avro with python 2 will have to be done more
> > > > manually after that point.
> > > >
> > > > Here are the Apache Voting Rules:
> > > > http://www.apache.org/foundation/voting.html
> > > >
> > > > This vote will remain open for at least 72 hours, more probably until
> > > next
> > > > week.
> > > >
> > > > [ ] +1 Drop support for Python 2
> > > > [ ] +0
> > > > [ ] -1 Continue supporting Python 2 because/until…
> > > >
> > >
>

Re: [REPORT] Board report for Apache Avro June 2020

[Driesprong, Fokko]

Looks good! Also a valid point on the Community Health.

Cheers, Fokko

Op wo 8 jul. 2020 om 08:54 schreef Sean Busbey :

> Hi folks!
>
> Here's the report for the quarter I submitted. Let me know if there are any
> last minute changes y'all would like to see.
>
> 
>
> ## Description:
> Apache Avro is a data serialization system with a compact binary format.
> It is
> used for storing and transporting schema driven serialized data. The unique
> features of Avro include automatic schema resolution - when the reader's
> expected schema is different from the actual schema with which the data was
> serialized the data is automatically adapted to meet reader's requirements.
>
> ## Issues:
> The project currently has no issues that require board attention.
>
> ## Membership Data:
> Apache Avro was founded 2010-04-20 (10 years ago)
> There are currently 34 committers and 23 PMC members in this project.
> The Committer-to-PMC ratio is roughly 3:2.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Nándor Kollár on 2019-08-29.
> - Kengo Seki was added as committer on 2020-07-08.
>
> ## Project Activity:
> Apache Avro 1.10.0 was released on 2020-06-29. This is a new major release
> from the project and most of our recent community work has been focused on
> it.
> The release updates many dependencies, addressing multiple CVEs, as well as
> introducing new features. For more details see the announcement:
>
> https://s.apache.org/avro-1.10.0-announce
>
> We added a new mailing list to get notifications about issue updates from
> github.
>
> ## Numbers
> For those who prefer metrics:
>
> Mailing Lists:
>  - dev@avro.apache.org had 1320 emails (28% increase)
>  - u...@avro.apache.org had 41 emails (55% decrease)
>  - iss...@avro.apache.org had 75 emails (new list)
>
> JIRA:
>  - 87 issues opened (24% decrease)
>  - 112 issues closed (47% increase)
>
> GitHub:
>  - 73 PRs open (17% decrease)
>  - 108 PRs closed (59% increase)
>
> Code Repository:
>  - 157 commits in the past quarter (29% increase)
>  - 45 code contributors in the past quarter (80% increase)
>
> ## Community Health:
> Community health is doing well at drawing in contributions. The PMC still
> needs to work to encourage repeat contributions and to recognize
> contributors
> through committership.
>

Re: [VOTE] Release Apache Avro 1.10.0 RC2

[Driesprong, Fokko]

+1 (binding)

Checked against Iceberg, and it looks good! Thanks for running the release
Ismaël!

Cheers, Fokko

Op vr 26 jun. 2020 om 08:09 schreef Sean Busbey :

> +1 binding
>
> I think the nexus link was wrong.  It was just the avro:avro jar for
> 1.10. looking at nexus I believe the staged repo for just his RC is
> this:
>
> https://repository.apache.org/content/repositories/orgapacheavro-1028/
>
> I based my voting on the provided dist RC and this staged repo.
>
> * signature and sums look good.
> * source looks like it matches to tag as of ref
> 6b55656b25cacf0d88cf44d9d802ce46dfaadc83
> * rat checks out on the source files (running the plugin at the top of
> the source tree has some false positives)
> * confirmed that the avro-maven-plugin pom has an up to date
> plexus-utils dependency.
> * A brief inspection of the docs looks good.
>
> a compatibility report for the java libraries would be nice to have,
> but I do not have the free time to come up with one currently.
>
>
> On Mon, Jun 22, 2020 at 10:43 AM Ismaël Mejía  wrote:
> >
> > Hi everyone,
> >
> > I'd like to propose the following RC2 to be released as the official
> Apache
> > Avro 1.10.0 release.
> >
> > The commit id is 6b55656b25cacf0d88cf44d9d802ce46dfaadc83
> >
> > * This corresponds to the tag: release-1.10.0-rc2
> > * https://github.com/apache/avro/releases/tag/release-1.10.0-rc2
> >
> > The release tarball, signature, and checksums are here (revision 40003):
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.10.0-rc2/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.10.0/
> >
> > This release includes 187 Jira issues (Thanks Ryan Skraba for the
> highlights)
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.0
> >
> > * Some interesting highlights (Thanks Ryan Skraba for the list):
> >
> > C#
> > - [AVRO-2389] Add Avro serialization for POCO (Reflection)
> >
> > Java
> > - [AVRO-2723] Automatically find defaults on POJO when using reflection.
> > - [AVRO-2278] Now throws an exception when getting a non-existent
> > field from a record
> > - [AVRO-2335] Remove Joda Time library.
> > - [AVRO-2438] Better support for URI and URL types.
> >
> > Perl
> > - [AVRO-1461] Distribute as a CPAN module.
> >
> > PHP
> > - [AVRO-2527] Update to PHP 7.x
> >
> > Python
> > - [AVRO-2656] avro-python package is now the preferred python3 library
> > and avro-python3 is prepared to be deprecated
> > - [AVRO-2387] type checking added to python
> >
> > Ruby
> > - [AVRO-1740] Support fingerprinting.
> > - [AVRO-2535] Support enum defaults.
> > - [AVRO-2545] Support aliases.
> >
> > * Upgrade dependencies to latest versions, including CVE fixes.
> > * Multiple fixes and more...
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close after noon UTC
> > Friday, June 12th, 2020
> >
> > [ ] +1 Release this as Apache Avro 1.10.0
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Best regards,
> > Ismaël Mejía
>
>
>
> --
> Sean
>

Re: [VOTE] Release Apache Avro 1.10.0 RC1

[Driesprong, Fokko]

Thanks Ismaël for running the release. I've just tested an all looks good.

+1 (binding)

Cheers, Fokko

Op za 13 jun. 2020 om 13:23 schreef Ryan Skraba :

> The only change necessary to compile Parquet with the 1.10.0 RC1
> artifacts was to add `public` to
> the parquet-avro pom.xml so that field members are generated with
> public access (the default is now private).   Or we could rewrite the
> test classes to use getters!
>
> (Just for info, if you want to reproduce my build, I also changed the
> repositories and pluginRepositories to temporarily include the
> https://repository.apache.org/content/groups/staging for these release
> candidates.  Something in my settings makes protoc-jar-maven-plugin
> unhappy with this setup, and the workaround is to build to a brand new
> local m2: `-Dmaven.repo.local=/tmp/m2` )
>
> I'm interested in your Parquet/Avro/Spark experience!  I've gotten
> them to work together using Apache Beam on Spark and extensive shading
> :/  Ismael has done some excellent (and tricky!) work trying to get
> the communities to move forward together!
>
> All my best, Ryan
>
> On Fri, Jun 12, 2020 at 7:23 PM Michael Heuer  wrote:
> >
> > Hello Ryan,
> >
> > If you don't mind me asking, what were the changes you found necessary
> for Parquet to work with Avro 1.10?
> >
> > The only reason I lurk on this mailing list is in the vain hope that
> Parquet and Avro and Spark will work together nicely at some point.
> >
> >michael
> >
> >
> > > On Jun 12, 2020, at 3:44 AM, Ryan Skraba  wrote:
> > >
> > > Hello!
> > >
> > > I checked the signature and SHA512 sums of the artifacts.
> > >
> > > Internally we have a large library of internal code that uses Avro
> > > 1.9, and it all recompiled and tests succeeded without modification.
> > >
> > > We have some other code that definitely breaks with 1.10 because of
> > > AVRO-2278 (getting an unknown field throws an exception).  In
> > > practice, it's not going to impact us in the short term, since it's
> > > all run on a cluster where the avro version is provided (and usually
> > > 1.8.x).  Our code will need to be rewritten to handle these exceptions
> > > gracefully for Avro 1.10+
> > >
> > > Parquet needed some minor massaging because of the change to the
> > > avro-maven-plugin (now generates private field members).  This should
> > > definitely be in the release notes:
> > >
> > > AVRO-2581: Maven plugin generates specific records with private fields.
> > >
> > > The arrow Avro adapter builds and tests with the  1.10.0 RC artifacts.
> > >
> > > I did some minor tests with a toy python project (both py and py3).
> > >
> > > +1 for me (mostly Java, some python).
> > >
> > > If there is a blocker or another release candidate, AVRO-2849 (spec
> > > clarification) would be nice to include.
> > >
> > > All my best and thanks for the work, Ismael!
> > >
> > > Ryan
> > >
> > >
> > > On Thu, Jun 11, 2020 at 2:25 PM Daniel Kulp  wrote:
> > >>
> > >> +1
> > >>
> > >> Been playing around with it a bit and it looks OK.
> > >>
> > >> Dan
> > >>
> > >>
> > >>> On Jun 11, 2020, at 4:53 AM, Ismaël Mejía  wrote:
> > >>>
> > >>> Kind reminder two days have passed, please check and vote!
> > >>>
> > >>> On Thu, Jun 11, 2020 at 2:29 AM Andy Le  wrote:
> > 
> >  Hey Ryan,
> > 
> >  Thank you for your clear explanation.
> > 
> >  +1 for me :D
> > 
> >  Thank you.
> > 
> > 
> >  On 2020/06/10 14:39:10, Ryan Skraba  wrote:
> > > Hey Anh, a release requires 3 PMC votes, but "unofficial" opinions
> are
> > > welcome and taken into account. My opinion is unofficial too!
> > >
> > > Just reply with a +1 if you think the artifacts are good to go or
> -1
> > > if you've found a bug, regression, security issue, or something
> that
> > > should block these artifacts from being the 1.10.0 release.
> > >
> > > For example, at my company, we have a lot of internal code and
> > > libraries that work with Avro 1.9.2, especially the generic Java
> APIs,
> > > so I'm going to compile with 1.10.0 and see if all the unit tests
> > > pass.
> > >
> > > If you're familiar with building any open source projects that use
> > > Avro, try bumping there and reporting back.  You can go through the
> > > open JIRA and open PR and make recommendations if we missed
> something
> > > important.
> > >
> > > For example, I've gone through a lot of the JIRA and it looks like
> > > AVRO-2799 (a protobuf bug) is blocking a user.  I'm looking to see
> if
> > > this is a regression or if it's an easy fix before I vote.
> > >
> > > I've verified the signatures and SHA512 of the artifacts, so
> that's good!
> > >
> > > All my best, Ryan
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jun 10, 2020 at 8:53 AM Anh Le  wrote:
> > >>
> > >> Hi,
> > >>
> > >> How can I make a vote on this?
> > >>
> > >> Best Regards.
> 

Re: Avro 1.10.0 release / branch cut

[Driesprong, Fokko]

Sounds good Ismaël, thanks for the update.

We need to update the jetty which contains a CVE before cutting 1.10

Cheers, Fokko

Op wo 13 mei 2020 om 23:42 schreef Ismaël Mejía :

> Previously in this thread I proposed to cut the release branch for Avro
> 1.10
> on friday 15 May and let a period of 2 weeks after that for stabilization
> and
> then start the vote for the release.
>
> However I (as other people in the project) at this moment have less time
> than
> expected so I would like to propose to postpone the branch cut and release
> two
> weeks to give us some short extra time to finish pending reviews and clean
> up
> stuff. New cut date May 29th.
>
> I hope this does not change many people plans, please let me know if you
> have
> any issues because of this and remember don't hesitate to ping me
> (hopefully) in
> advance if you want to discuss any release specific issue.
>
> Regards,
> Ismaël
>
> On Tue, Apr 28, 2020 at 10:02 AM Ismaël Mejía  wrote:
> >
> > Hello everyone,
> >
> > As discussed previously [1] we intend to do the Avro 1.10.0 release on
> > May so I wanted to propose to cut the branch the next May 15th. The
> > idea is that in the following weeks we work hard on reviews and issue
> > triage and maybe the latest dependency updates. Once the branch is cut
> > we let then some days to stabilize the release branch (hopefully 1-2
> > weeks) and then go with the vote.
> >
> > If you have issues you want to mark as critical to be part of the
> > release (e.g. security related or data corruption) mark them with the
> > Fix Version: 1.10.0 field in JIRA and more important if you have
> > issues that you think won’t be able to make it in time please
> > unmark/untag them or mark them as version 1.11.0
> >
> > This is the list of open issues for the release, so please go ahead
> > and help us finish/triage them.
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%201.10.0
> >
> > Thanks, and good triage, don’t hesitate to email me or ping me in your
> > PRs./tickets to bring my attention on critical things for the release.
> >
> > Best,
> > Ismaël
>

Re: [DISCUSS] Create a new mailing list for github notifications

[Driesprong, Fokko]

Yes, I'd be in favor of having a separate ML. If you want to keep track of
the action at the repository, you can also get notified using Github.

Cheers, Fokko

Op do 7 mei 2020 om 14:37 schreef Ismaël Mejía :

> Let's wait for others opinions and if we agree I will proceed to
> create the ML next week.
>
> On Thu, May 7, 2020 at 12:13 PM Ryan Skraba  wrote:
> >
> > I think it's safe to create issues@ and merge the PR right away -- it
> > would pretty much restore the behaviour the dev@ list had at the
> > beginning of the year.
> >
> > I have all my mailing lists dumped into the same folder, but if
> > issues@ existed for JIRA and Github notifications, I would be tempted
> > to subscribe to the daily digest instead.
> >
> > I'm curious about how other highly-productive Apache developers have
> > their email filters set up for various projects!  It's easy to get
> > overwhelmed by the volume (especially if you don't keep up daily, I
> > was off most of March/April).
> >
> > In any case, https://github.com/apache/avro/pull/873 LGTM!
> >
> > Ryan
> >
> >
> > Ryan
> >
> > On Wed, May 6, 2020 at 11:51 PM Ismaël Mejía  wrote:
> > >
> > > Other alternative (probably the cleanest is to let dev@ only for
> > > discussion and issues for all the JIRA/github action. The tradeoff
> > > there is that people subscribed to dev@ only may miss part of the
> > > action (+ it is not backwards compatible with our archives) but it is
> > > clearly an option too.
> > >
> > > On Wed, May 6, 2020 at 11:47 PM Ismaël Mejía 
> wrote:
> > > >
> > > > The issues in the yaml file are every comment done on every issue or
> > > > pull requests on github (notice that the issues part probably won't
> > > > apply to us since we use JIRA to track issues).
> > > >
> > > > Currently we receive ALL JIRA notifications into dev@ we can change
> > > > that us well since it probably makes sense.,However this is a
> > > > different process of the .asf.yaml file.
> > > >
> > > > The proposed notification schema will still notify new PRs and closed
> > > > PRs into dev@ which is valuable for awareness on new contributions,
> > > > maybe we can do that for JIRA too, just notify on created and closed
> > > > JIRAs (assuming INFRA can map such rule). then people interested in
> > > > tracking the middle steps and issue/PR discussions could go and
> > > > subscribe into issues@ where all the detailed information will be.
> > > >
> > > > Other opinions? Suggestions?
> > > >
> > > >
> > > > So many options
> > > >
> > > > On Wed, May 6, 2020 at 4:18 PM Ryan Skraba  wrote:
> > > > >
> > > > > This sounds good to me!  The "issues" in the yaml are the GitHub
> > > > > issues, which are disabled in github right?
> > > > >
> > > > > Any opinions on moving JIRA notifications from j...@apache.org to
> > > > > issues@ as well?  That's what I would expect!
> > > > >
> > > > > I guess this would make little difference to me in practice since I
> > > > > would subscribe to both, but it would make it a lot easier to
> browse
> > > > > and link to the dev@ mailing list on the web!
> > > > >
> > > > > All my best, Ryan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 6, 2020 at 1:15 AM Andy Le  wrote:
> > > > > >
> > > > > > Hi Ismaël,
> > > > > >
> > > > > > Yep, I think it's a little bit noisy for our mailing list with
> such notifications. But it's fun to see what other people are working on.
> > > > > >
> > > > > > IMHO, in addition to a new mailing list, we should have a better
> label mechanism for easily routing/filtering messages.
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > On 2020/05/05 22:59:59, Ismaël Mejía  wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > Some of you have probably noticed that we are receiving a LOT
> of extra
> > > > > > > notifications that were not active in the past.  This is
> happening because
> > > > > > > Apache INFRA installed a new way to map notifications into
> mailing lists for its
> > > > > > > projects and the default configuration right now is sending
> all the github
> > > > > > > notifications to dev@
> > > > > > >
> > > > > > > https://gitbox.apache.org/schemes.cgi?avro
> > > > > > >
> > > > > > > So I would like to propose that we create a new mailing list
> called maybe
> > > > > > > iss...@avro.apache.org To send all the git individual PR
> comments/reviews there
> > > > > > > and only the new/close PRs here.
> > > > > > >
> > > > > > > I suppose that issues@ could be a list of interest for some
> people and it will
> > > > > > > help us have a more readable dev@ mailing list.
> > > > > > >
> > > > > > > Technically all we need to do is create the new ML on
> selfserve and a new
> > > > > > > .asf.yaml file following this format.
> > > > > > >
> > > > > > > https://s.apache.org/asfyaml-notify
> > > > > > >
> > > > > > > WDYT? Other preferences or other name for the mailing list
> github@ maybe?
> > > > > > > More configuration ideas?
> > > > > > >
> > > > 

Re: Community governance

[Driesprong, Fokko]

For the releases, the information is on Github:
https://github.com/apache/avro/releases. For the fine details, Jira is the
way to go. You can select which tickets are merged in which version. This
is also how we generate the changelog, we condense it and put it on Github.
Maybe we should add a link from the docs website to Github.

There is some traction around .Net, Ruby, Python, and especially Java.
However, implementation such as Perl, C(++) is not happening a lot. Also,
the original contributors aren't that active anymore. I like the idea of
splitting the different languages to different repositories. For Parquet,
the each repository contains also one language. This is also what Ryan
suggested in the other thread. For more fundamental changes, there is this
concept of Avro Enhancement Proposals:
https://cwiki.apache.org/confluence/display/AVRO/Avro+Enhancement+Proposals
This
is a confluence section where you can create a proposal, to get consensus
in the community, and then implement it.

Hope this helps!

Cheers, Fokko



Op di 5 mei 2020 om 10:56 schreef Andy Le :

> I've checked Avro release notes [1] and found them containing vague
> information about each release. There's also a lack of documentation for
> other programming languages.
>
> My question: should we carefully lay out what to do, opening dedicated
> issues to call for contributions?
>
> Thanks & best regards.
>
> [1]
> https://avro.apache.org/releases.html#12+February+2020%3A+Avro+1.9.2+Released
>
> On 2020/05/03 07:07:46, Andy Le  wrote:
> > Hi guys,
> >
> > I'm starting this thread to discuss about my issue about our community:
> >
> > > Recently I've seen so many Jira issues and Github PRs having no proper
> responses from committers.
> >
> > I think responsive answers from members will create a better Avro
> community.
> >
> > How can we resolve the issue? Glad that I can help any thing.
> >
> > Thanks.
> >
> >
> >
> >
> >
>

Re: When and how should we remove lang/py3?

[Driesprong, Fokko]

Hi Mchael,

Thank you for bringing this up. I agree, but I'm not sure how we're going
to migrate the users to the avro package.

Rijprojectnum_downloads
1
avro
885838
2
avro-python3
147313

A quick query on bigquery, that contains all the pypi downloads shows that
the python3 package is more popular. Maybe we should add a deprecation
warning in 1.10.0?

Cheers,
Fokko

The query:
SELECT file.project, COUNT(*) AS num_downloads
FROM `the-psf.pypi.downloads*`
WHERE file.project IN ('avro-python3', 'avro')
  -- Only query the last 30 days of history
  AND _TABLE_SUFFIX
BETWEEN FORMAT_DATE(
  '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY file.project





Op do 30 apr. 2020 om 14:20 schreef Michael Smith :

>  Previous emails on this list have discussed the "why" questions. But when
> and how should we go about it? Could 1.10 be the last avro release to
> include avro-python3? This would allow us to play a little less ticket
> tennis with folks reporting bugs (can you reproduce this in "the other"
> python library?) and focus all our effort into one python implementation.
>
> What do you think?
>

Re: Proposal: RFCs for Avro 2.x

[Driesprong, Fokko]

Yes, I'd love to see the AEPs being revived. This would enable it to make
more structural/fundamental changes, such as the one mentioned above. This
also enables others to have a complete overview of changes and the impact.

I also agree with Michael when it comes to the project structure. Maybe we
can look at other projects, such as Thrift and Protobuf, how they handle
this.

Cheers, Fokko

Op wo 29 apr. 2020 om 05:01 schreef Michael A. Smith :

> Definitely in favor of the AEPs.
>
> I have mixed feelings about the notion of a single language binding. These
> types of packages are commonplace enough in Python, but installing a Python
> package that requires a build time clib is often troublesome, and I'm often
> grateful to find a pure Python alternative.
>
> But I also think the project suffers from the current language structure –
> it encourages implementations to be similar to one another over being
> idiomatic in their own language, and we can't do a release for just one
> language. If boiling down the project into a single core lib for multiple
> languages is what it takes to fix that, then it's probably for the best.
>
> If we do go down that road, I'd like to see us make the build process
> extremely smooth.
>
> All the best,
> Michael A. Smith
>
> On Tue, Apr 28, 2020 at 04:01 Ismaël Mejía  wrote:
>
> > Huge +1 to recover the Avro Enhancement Proposals (AEP)
> >
> > The experimental features Ryan mentioned definitely merit(ed) to be
> > part of it, and in particular the procedure to decide when they will
> > become ‘stable’ or default, for example for fastread. Also other
> > proposals/discussions like the split release or semantic versioning
> > should be part of it.
> >
> > About Avro 2.0.0 I think breaking binary compatibility of the format
> > is going to prove to be a hard sell (are named unions valuable enough
> > to break backwards compatibility?), if we can extend the binary format
> > in a compatible way there is no reason to have 2.0.0 so I agree that
> > there is a delicate balance we should avoid because strict stability
> > could let us also ostracized.
> >
> > What I personally would like is to make Avro as lean and efficient as
> > possible and focus mostly in the binary format part and tools probably
> > removing the less used parts (IPC/RPC/trevni) so it is good to see
> > that other people are starting to agree on that.
> >
> > One more radical idea I would like is to try is to unify a bit the
> > implementations probably having a robust low level one in one systems
> > language (C or Rust) and bindings for all the languages that rely on
> > it but this is probably more because of my frustration of seeing
> > projects that take this approach becoming slowly the standard and
> > Apacho Avro relegated (this is already happening on the python front).
> >
> > In general the critical issue with Avro are the downstream
> > consequences of our actions, and of course we will always have
> > incomplete information, but we can investigate and see if changes are
> > worth.
> >
> > Regards,
> > Ismaël
> >
> > On Mon, Apr 27, 2020 at 6:51 PM Ryan Skraba  wrote:
> > >
> > > Hello!
> > >
> > > You bring up some good points -- I'm glad Avro is so widely used, but
> > > it does make me nervous to see any changes that might break other
> > > projects, or change any behaviour.
> > >
> > > Currently, we've talked about managing developer expectations with
> > > semantic versioning (especially with the necessary Jackson API cleanup
> > > that happened in 1.9.x), or versioning artifacts separately.
> > >
> > > We also have a couple of experimental/feature flags for some behaviour
> > > changes:
> >
> https://cwiki.apache.org/confluence/display/AVRO/Experimental+features+in+Avro
> > >
> > > And there is already a page for Avro Enhancement Proposals that look
> > > largely out of date:
> > >
> >
> https://cwiki.apache.org/confluence/display/AVRO/Avro+Enhancement+Proposals
> > >
> > > Moving some of the extras to a separate repo brings many of the same
> > > problems as versioning artifacts separately (nobody wants to deal with
> > > a compatibility matrix).  I'm definitely not against it, but I'm not
> > > sure how it would improve the situation.
> > >
> > > There's a fine line between being extremely stable and being
> > > paralyzed! I would be enthusiastic about any process changes that
> > > would help us encourage and adopt new features (and fixes) more
> > > quickly.

Re: Proposal: RFCs for Avro 2.x

[Driesprong, Fokko]

Hi Andy,

Thanks for reaching out. Sorry for not being so active in the community
lately.

Since Avro 1.8.2 there has been some activity on the repository again,
fixing stuff like security issues and migrating to later versions of Java.
Avro has been around for 10 years now, and I would like to keep (some)
backward compatibility to make sure that people are still going to use it
for another 10 years :) In the past, the idea was to keep the format
backward compatibility, this excludes the Java API to. So we did some
changes to the API, such as removing Jackson from the public API and
aggressively migrating from Joda Time to Java JSR-310. This caused a lot of
issues because Avro is deeply nested in a lot of projects. For example, it
is a huge task to update Avro in Hive or Hadoop. Therefore we believe that
backward compatibility is very important.

And I agree that we should mainly focus on the Avro spec itself, and not
too much on File I/O and Network etc :) However, if we decide to break an
API, we should do it for a good reason.

Cheers, Fokko

Op wo 22 apr. 2020 om 16:09 schreef Andy Le :

> Hi guys,
>
> I'm new to this vibrant open source community. My story with Avro can be
> found here [1]
>
> While implementing the feature, I got stuck and had various discussions
> with Dough Cutting, Fokko Driesprong You may see here [2]
>
> Here my (bias) observations about our current Avro 1.9.x:
>
> - Some improvements can't be made due to fear of backward
> incompatibilities. For example: specifications about named Union.
>
> - If `Apache Avro™ is a data serialization system.` then the repository
> `apache/avro` should solely focus on (de)serialization, right? Currently
> our repository contains many nice-to-have-but-not-critical things like:
> File I/O, Network I/O
>
> IMHO, I think:
>
> - We should publicly gather RFCs for Avro 2.x
>
> - We should move such nice things out of Avro 2.x (may be to other
> dedicated repositories)
>
> What do you think about my suggestions. Pls kindly let me know.
>
> Thank you & be strong.
>
> [1] My fork: https://github.com/anhldbk/avro-fork#why-this-fork
> [2] My opened issue:
> https://issues.apache.org/jira/browse/AVRO-2808?jql=reporter%3Danhldbk%20AND%20resolution%20is%20EMPTY
>
>
>

Re: Append to existing file in C#

[Driesprong, Fokko]

Hi Vladimir,

Please feel free to open a ticket (https://jira.apache.org/jira) and a PR
if you want feedback on the implementation. Try to make the PR's small to
make the reviewing easier :)

Cheers, Fokko

Op zo 29 mrt. 2020 om 14:51 schreef Volodymyr Aleksandrov :

> Hello,
>
> Want to propose to add a possibility to append to existing files in C# in a
> similar way how it is implemented in Java.
>
> I haven't found such tasks in GitHub and Jira and unfortunately haven't
> found a way to access the search mailing list archive.
>
> I created a working draft implementation and can finish writing unit tests
> in a couple days. You can find the code here:
> https://github.com/valxv/avro.
>
> My proposal is to add two new public static methods OpenAppendWriter (one
> receiving string path and other - streams) to DataFileWriter class to be
> consistent with C# implementation.
>
> --
> Vladimir
>

Re: [DISCUSS] version numbers and where changes should land

[Driesprong, Fokko]

So, as I understand it. The whole 1.x version should be binary compatible.
So anything is written with Java 1.x should be readable with Python 1.x.
We've been working on extending the integration tests as well.

Not all languages support all the features, for example, many languages
lack support for logical types. In this case, a datetime would be just read
as an integer, so there is a fallback scenario.

For 1.9 we broke some of the API's because, at the time, the decision was
that removing Jackson from the public API was required to move from
Codehaus jackson (1.x) to the Fasterxml one (2.x). The public API shouldn't
have exposed these methods in the first place.

I wouldn't be in favor of switching to 10.x (dropping the 1. in front of
it). What's the added value in this? I'm just afraid of changing this,
would confuse a lot of downstream users.

Also, a similar discussion was on the Spark devlist, I think Michael has
some valid points here:
https://mail-archives.apache.org/mod_mbox/spark-dev/202002.mbox/browser

Maybe it is good to formalize our policy, and put it on the website.

Cheers, Fokko Driesprong

Op vr 28 feb. 2020 om 17:53 schreef Sean Busbey :

> Counterpoint on independently versioning the various languages. Do we
> know if Python Avro X works with Java Avro Y as it is? It seems like
> we already get surprised pretty often when they don't.
>
> If we stop including the "data compatibility version" or whatever
> we're calling the first number, we'll need to get more formal on
> versioning the specification and having libraries plainly label which
> specification(s) they comport to.
>
> At the very least it seems like we'd make the _easy_ path easier for
> the languages that are well maintained. Sure it'll be burden on those
> languages that aren't well maintained, but it seems like those are
> already in that position?
>
> On Thu, Feb 27, 2020 at 9:13 AM Ismaël Mejía  wrote:
> >
> > Bringing my comment from the JIRA ticket here for discussion:
> >
> > > "One argument against semantic versioning is the fact that Avro
> supports
> > 9 language APIs, so if let's say C++ breaks its backwards compatibility
> > should we move the version number up for every single language? Sounds
> like
> > a burden and in particular a not easy task to track since we do not have
> > proper validation of breaking changes in place for every language at this
> > point.
> > > ... (even if we separate release numbers per language) that seems like
> a
> > lot of work for probably a similar output because then users will doubt,
> > wait is Python Avro 3.1.0 compatible with Java Avro 5.2.0? and they will
> > probably be for the binary format."
> >
> > Also there is the case of interop tests, how will those act in this case.
> > We will need a compatibility matrix, again I am not sure if it is the
> best
> > approach, looks like lots of work for not much in return.
> >
> >
> >
> > On Thu, Feb 27, 2020 at 12:21 PM Ryan Skraba  wrote:
> >
> > > Hello!  Resurrecting -- I think this was the last thread bringing up
> this
> > > issue!
> > >
> > > Since we've talked about releasing 1.10.x in May, and it's a nice
> > > round number... what do you think about
> > >
> > > 1) finally dropping the prefix for the "specification version" and
> > > calling it Avro 10.x
> > >
> > > 2) committing to semantic versioning for future releases
> > >
> > > I can see this being a hugely positive move for aligning with the
> > > expectations of developers and projects... but it leads to a lot of
> > > questions about releasing all the artifacts together.
> > >
> > > There's already a JIRA:
> https://issues.apache.org/jira/browse/AVRO-2687
> > >
> > > Ryan
> > >
> > > On Fri, Sep 13, 2019 at 12:00 PM Driesprong, Fokko
> 
> > > wrote:
> > > >
> > > > Thanks Sean for bringing this up.
> > > >
> > > > For the 1.9 branch there were some incompatible changes in the API
> with
> > > > respect to 1.8.2. We've removed Jackson
> > > > <https://github.com/apache/avro/pull/135> and Netty from the public
> API.
> > > > This is actually breaking some of the builds
> > > > <https://github.com/apache/incubator-iceberg/pull/297>, so,
> > > unfortunately,
> > > > it isn't compatible, and therefore the major version bump.
> > > >
> > > > The 1.9.x branch still has support for the Joda time library, but
> > > defaults
> > > > to jsr310, but is still compatible (I believe). 

Re: Documenting optional, noncompliant behavior for compatibility

[Driesprong, Fokko]

Hi Michael,

A logical place for me would be the docs. In this case, I would document
the options in the PythonDoc and then use a tool to generate the docs. We
can then make this part of the website: https://avro.apache.org/docs/1.9.2/

This is similar to what we've done in the past for Java/C/C# etc.

Cheers, Fokko

Op za 22 feb. 2020 om 19:00 schreef Michael A. Smith :

> In https://github.com/apache/avro/pull/830 I propose that we offer
> users the option to maintain Python's current behavior of allowing
> spaces in enum symbols. In this implementation, the default behavior
> is more spec-compliant, but users can choose to go back to the way it
> was. Since Python has never implemented the spec change in
> https://issues.apache.org/jira/browse/AVRO-1725 to this point, it's
> certain that many python-only Avro users will find this new behavior a
> sad surprise. (I'm certain of this because I'm one of those users who
> was dismayed when I found my avro incompatible with the Java avro
> implementation.) So I wanted to offer them an out.
>
> I'm not exactly sure how to document this other than in the Python
> code itself, because it's not part of the spec. Any suggestions?
>

Re: Upgrade Avro dependency in Spark to 1.9.2

[Driesprong, Fokko]

Awesome work Ismaël, let me know if I can help somewhere.

Chees, Fokko

Op ma 17 feb. 2020 om 16:00 schreef Ismaël Mejía :

> I had not tried to do the upgrade Spark since I assumed it will fail
> because of the transitive dependencies of Hive.
>
> But I decided to give it a shot today. Luckily the Spark code base is quite
> Avro friendly so codewise it was 'easy'.
>
> Of course it is still failing, but you can use that to refer on the other
> PR.
> And if you can find any fixes to the pending things that would be great.
>
> https://github.com/apache/spark/pull/27609
>
> Regards,
> Ismaël
>
>
> On Fri, Feb 14, 2020 at 5:42 PM Michael Heuer  wrote:
>
> > Hello Ismaël,
> >
> > Might you be able to share a link to your patch for Spark?  I would like
> > to try to apply it on top of
> >
> > https://github.com/apache/spark/pull/26804 <
> > https://github.com/apache/spark/pull/26804>
> >
> > which attempts to upgrade the Parquet dependency for Spark to 1.11.0.
> >
> > Thank you,
> >
> >michael
> >
> >
> > > On Feb 14, 2020, at 10:30 AM, Ismaël Mejía  wrote:
> > >
> > > Ah lovely question.
> > >
> > > tldr; version
> > > Spark depends on Hive so Hive should be upgraded first
> > > Spark depends on two versions of Hive a fork by Spark of 1.x and
> upstream
> > > Hive 2.x
> > > Upgrading the first is not even discussed at the moment, for the
> second I
> > > added a patch that passes all tests if you run it against Spark
> > 2.4/master,
> > > but Hive uses a forked version of Spark 2.3 to run its tests (YES
> > CIRCULAR
> > > DEPENDENCY!!!)
> > >
> > > One extra point that is pushing things in the right direction is that
> > > Parquet and Iceberg already moved to Avro 1.9.x so pressure is growing
> > for
> > > things to move, but it is still is a mess, but we want to give the
> fight,
> > > one thing is sure it won't be for Spark 3.0.0, best case 3.1.x and that
> > > also depends on the good will of the Hive contributors that have
> ignored
> > my
> > > emails + patches for some time.
> > >
> >
> https://lists.apache.org/thread.html/rc6c672ad4a5e255957d54d80ff83bf48eacece2828a86bc6cedd9c4c%40%3Cdev.hive.apache.org%3E
> > >
> > > For the detailed details on the saga:
> > > https://issues.apache.org/jira/browse/SPARK-27733
> > > https://issues.apache.org/jira/browse/HIVE-21737
> > >
> > >
> > > On Fri, Feb 14, 2020 at 5:04 PM Michael Heuer 
> wrote:
> > >
> > >> Hello,
> > >>
> > >> I wonder if any Avro devs might be willing to help push a PR for
> Apache
> > >> Spark to update the Avro dependency from 1.8.2 to 1.9.2?
> > >>
> > >> I foresee some trouble with binary incompatible code changes and
> > >> dependency version conflicts, and could use some additional support.
> > >>
> > >> Thank you in advance,
> > >>
> > >>   michael
> >
> >
>

Re: Viewing Test Results

[Driesprong, Fokko]

Thanks for brining this up Michael,

This was on my plate if I'm not mistaken. I've just commented on the INFRA
ticket. I'll monitor the ticket, and see what comes out of it.

Cheers, Fokko

Op zo 16 feb. 2020 om 20:06 schreef Michael A. Smith :

> A while back I opened
> https://issues.apache.org/jira/browse/INFRA-19615 to see if Apache
> Infrastructure can help us do something constructive with test results
> in the form of something standard, like a nice result viewer for
> junit.xml artifacts. One question that was asked in the ticket was
>
> > The fact that the build happens on non ASF hardware might make logistics
> hard, maybe.
>
> > What if the build was to happen on our jenkins instance or buildbot?
> what are we lacking in that forces you to use Travis?
>
> I don't know the whole story, so I can't answer it. Is there someone
> else who can shed some light on this?
>
> Thanks,
> Michael
>

Re: [ANNOUNCE] Apache Avro 1.9.2 released

[Driesprong, Fokko]

Awesome, thanks Ryan for running the release!

Cheers, Fokko

Op do 13 feb. 2020 om 16:27 schreef Ryan Skraba :

> The Apache Avro community is pleased to announce the release of Avro 1.9.2!
>
> The link to all fixed JIRA issues and a brief summary can be found at:
> https://github.com/apache/avro/releases/tag/release-1.9.2
>
> This release includes 73 Jira issues:
>
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.2
>
> Some bug fixes:
> * C#: AVRO-2606 handle multidimensional arrays of custom types
> * Java: AVRO-2592 Avro decimal fails on some conditions
> * Java: AVRO-2641 Generated code results in java.lang.ClassCastException
> * Java: AVRO-2663 Projection on nested records does not work
> * Python: AVRO-2429 unknown logical types should fall back
> Improvements:
> * Java: AVRO-2247 Improve Java reading performance with a new reader
> * Python: AVRO-2104 Schema normalisation and fingerprint support for
> Python 3
> Work to unify Python2 and Python3 APIs in preparation for sunset.
> Improved tests
> Improved, more reliable builds.
> Improved readability
> Upgraded dependencies to latest versions, including CVE fixes.
> And more...
>
> This release can be downloaded from:
> https://www.apache.org/dyn/closer.cgi/avro/
>
> The released artifacts are available:
> * C#: https://www.nuget.org/packages/Apache.Avro/1.9.2
> * Java: from Maven Central,
> * Javascript: https://www.npmjs.com/package/avro-js/v/1.9.2
> * Python 2: https://pypi.org/project/avro/1.9.2/
> * Python 3: https://pypi.org/project/avro-python3/1.9.2.1/
>   - See https://issues.apache.org/jira/browse/AVRO-2737
> * Ruby: https://rubygems.org/gems/avro/versions/1.9.2
>
> Thanks to everyone for contributing!
>
> Ryan Skraba
>

Re: [VOTE] Release Apache Avro 1.9.2 RC2

[Driesprong, Fokko]

Thanks again Ryan for running the release.

+1 (binding)

Did the following:

   - Verified the keys
   - Verified the hashes
   - Ran the tests
   - Ran the testsuite of Iceberg against Avro 1.9.2

Cheers, Fokko



Op vr 7 feb. 2020 om 11:47 schreef Ryan Skraba :

> Hi everyone,
>
> I'd like to propose the following RC2 to be released as the official Apache
> Avro 1.9.2 release.
>
> The commit id is bf20128ca6138a830b2ea13e0490f3df6b035639
> * This corresponds to the tag: release-1.9.2-rc2
> * https://github.com/apache/avro/releases/tag/release-1.9.2-rc2
>
> The release tarball, signature, and checksums are here (revision 37932):
> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.2-rc2/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/avro/KEYS
>
> Binary artifacts for Java are staged in Nexus here:
> *
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.2/
>
> This release includes 71 Jira issues (Thanks Ismael for the Fix
> Version clean-up):
>
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.2
>
> * Some bug fixes:
>   - C#: AVRO-2606 handle multidimensional arrays of custom types
>   - Java: AVRO-2592 Avro decimal fails on some conditions
>   - Java: AVRO-2641 Generated code results in java.lang.ClassCastException
>   - Java: AVRO-2663 Projection on nested records does not work
>   - Python: AVRO-2429 unknown logical types should fall back
> * Improvements:
>   - Java: AVRO-2247 Improve Java reading performance with a new reader
>   - Python: AVRO-2104 Schema normalisation and fingerprint support for
> Python 3
> * Work to unify Python2 and Python3 APIs in preparation for sunset.
> * Improved tests
> * Improved, more reliable builds.
> * Improved readability
> * Upgrade dependencies to latest versions, including CVE fixes.
> * And more...
>
> Since RC1, five commits have been added, fixing the following JIRA:
> * https://jira.apache.org/jira/browse/AVRO-2663 (to complete the fix in
> RC1)
> * https://jira.apache.org/jira/browse/AVRO-2729 Upgrade Velocity to
> version 2.2
> * https://jira.apache.org/jira/browse/AVRO-2727 Upgrade hadoop3 to
> version 3.2.1
> * https://jira.apache.org/jira/browse/AVRO-2726 Upgrade Jackson to
> version 2.10.2
>
> Please download, verify, and test. This vote will remain open for at least
> 72 hours. Given sufficient votes, I would like to close after noon UTC
> Wednesday, February
>  12th
>
> [ ] +1 Release this as Apache Avro 1.9.2
> [ ] +0
> [ ] -1 Do not release this because...
>
> Best regards, Ryan Skraba
>

Re: [VOTE] Release Apache Avro 1.9.2 RC1

[Driesprong, Fokko]

Thanks Ryan for taking care of this.

Unfortunately, I have to give a -1 (binding) to the RC. It turns out that
AVRO-2663 wasn't properly cherry-picked. I've tried to run Avro against
Iceberg, and the tests failed. It turned out that only the first commit of
the PR was being cherry-picked onto the 1.9 branch. I've tested it, and
after the patch, the tests in Iceberg pass as well.

I've created a PR for patching this: https://github.com/apache/avro/pull/801

Cheers, Fokko

Op do 6 feb. 2020 om 14:58 schreef Ryan Skraba :

> Hello!  Two points about the release announcement!
>
> 1. I originally signed the artifacts in
> https://dist.apache.org/repos/dist/dev/avro/avro-1.9.2-rc1/ with the wrong
> key.
>
> I've just updated all of the *.asc files with my code signing key, so they
> should be correct now (SVN revision 37897).  My apologies.  The nexus
> artifacts were already signed with the correct key fortunately.
>
> 2. I missed a version change from my copy/paste in the vote text; if you
> copy it from the announcement, make sure you vote for/against releasing
> 1.9.2!
>
> All my best, Ryan
>
>
> On Wed, Feb 5, 2020 at 5:59 PM Ryan Skraba  wrote:
>
> > Hi everyone,
> >
> > I'd like to propose the following RC1 to be released as the official
> Apache
> > Avro 1.9.2 release.
> >
> > The commit id is 39dd5c6bb33ab6634b4ed69f591a0676be62563a
> > * This corresponds to the tag: release-1.9.2-rc1
> > * https://github.com/apache/avro/releases/tag/release-1.9.2-rc1
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.2-rc1/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.2/
> >
> > This release includes 57 Jira issues:
> >
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.2
> >
> > * Some bug fixes:
> >   - C#: AVRO-2606 handle multidimensional arrays of custom types
> >   - Java: AVRO-2592 Avro decimal fails on some conditions
> >   - Java: AVRO-2641 Generated code results in
> java.lang.ClassCastException
> >   - Java: AVRO-2663 Projection on nested records does not work
> >   - Python: AVRO-2429 unknown logical types should fall back
> > * Improvements:
> >   - Java: AVRO-2247 Improve Java reading performance with a new reader
> >   - Python: AVRO-2104 Schema normalisation and fingerprint support for
> > Python 3
> > * Work to unify Python2 and Python3 APIs in preparation for sunset.
> > * Improved tests
> > * Improved, more reliable builds.
> > * Improved readability
> > * Upgrade dependencies to latest versions, including CVE fixes.
> > * And more...
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close after noon UTC
> > Monday, February
> >  10th
> >
> > [ ] +1 Release this as Apache Avro 1.9.1
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Cheers, Ryan Skraba
> >
>

Re: Including "fastavro" in release 1.9.2

[Driesprong, Fokko]

Sounds like a plan to me. I don't have any objections.

In the far future, I'd like to merge these readers, otherwise, you'll end
up with many configuration permutations. This makes it challenging to test
each permutation. But first having it as an option in 1.9 is a good idea,
so we can battle test the new reader.

Cheers, Fokko


Op za 1 feb. 2020 om 03:06 schreef Sean Busbey :

> unless the RM has an objection based on where they are in the RC
> creation process, I say go for it. (and if there is such an issue then
> let's get it ready for 1.9.3)
>
> On Fri, Jan 31, 2020 at 10:29 AM Ryan Skraba  wrote:
> >
> > Hello!  There's a long standing PR[1][2] that provides an impressive
> > performance boost to deserializing Avro.
> >
> > The author has been patient and reactive for a LONG time (thanks
> Martin!),
> > and it's inspired some really interesting discussion.
> >
> > I'd like to merge and cherry-pick into release 1.9.2 -- as long as it's
> > turned off there shouldn't be any visible effect at all, and having it
> in a
> > release would be a great way to find regressions (if any) and encourage
> > some of the ideas brought up in the PR discussions.
> >
> > Any objections?  Is this too big a change for a minor release?
> >
> > I noted in the PR that it would be pretty cool if we had a "better" way
> of
> > noting experimental features (and that the Yetus annotations would be a
> > good start for 1.10!)
> >
> > All my best, Ryan
> >
> > [1]: https://issues.apache.org/jira/browse/AVRO-2247
> > [2]: https://github.com/apache/avro/pull/391
>

Re: Avro 1.9.2

[Driesprong, Fokko]

Thanks for all the work Ryan.

We just create an RC tag on the 1.9 branch, and build and publish the
artifact to the mailing list.  Today I'll check if we want to cherry-pick
any minor dependency updates for Java.

Cheers, Fokko

Op wo 22 jan. 2020 om 18:28 schreef Ryan Skraba :

> Hello!  Slow but certain -- I pushed the rest of the identified commits for
> python, updated the tickets and am doing some confirmation!
>
> Notably, there was a nice cleanup/refactor of the Dockerfile that was
> cherry-picked.  I kept some but not all of the tool bumps that it included
> (as a separate commit).
>
> There's a couple of JIRA that are marked as fixed in 1.9.1, but don't seem
> to actually be present (For info: AVRO-2377, AVRO-2298, AVRO-2426), and I'm
> continuing down the list!
>
> Anybody have a good deadline to think about cutting a branch?  Any last
> requests for fixes to be cherry-picked?
>
> I'll continue doing some basic clean-up as I can, but if the python work is
> satisfactory https://github.com/apache/avro/pull/777, I'd be pretty happy
> with the state of the branch.
>
> All my best, Ryan
>
>
>
>
> On Sat, Jan 18, 2020 at 1:07 AM Michael A. Smith 
> wrote:
>
> > Ryan, thanks for agreeing to take a shot. I created a tracking ticket
> > for the effort: https://issues.apache.org/jira/browse/AVRO-2697
> >
> > I attached a text file to that ticket with the PRs against master that
> > I think we want in 1.9. The ones checked off are what I already
> > managed to do against my own fork's branch-1.9, in
> > https://github.com/apache/avro/pull/777
> >
> > That PR is set to allow edits from maintainers, so you can update it
> > directly. If you find working that way cumbersome, feel free to just
> > open your own PR and I'll close mine.
> >
> > On Fri, Jan 17, 2020 at 9:52 AM Ryan Skraba  wrote:
> > >
> > > Hello!
> > >
> > > For python I'd be happy to go through the build changes, especially if
> > > you can list (or create a branch) for the lang/py cherry-picks that
> > > are already known to be necessary!
> > >
> > > I've been going through the list of ALL the commits in master that
> > > have no equivalent in release-1.9 using:
> > >
> > > git co master && git cherry branch-1.9
> > >
> > > I've identified a few minor issues, such as AVRO-2377, which is marked
> > > as fixed 1.9.0 in JIRA but isn't in the release-1.9 branch.  I'm
> > > making a list (and checking it twice), but it's very, very slow
> > > going... I'm hoping I pick up speed as I move along, any tips+tricks
> > > would be appreciated!  (I'm always on ASF slack, and I'm willing to
> > > put in the effort to document what I've learned in the wiki!)
> > >
> > > What do you think?  One PR with all the cherry-picks I think are
> > > missing, or one PR per?
> > >
> > > In the meantime, I've got my key
> > > (http://people.apache.org/keys/committer/rskraba.asc) but I'll need
> > > some PMC help to get all the permissions set up for a release.
> > >
> > > 1. My key should be added at svn co --depth=files
> > > https://dist.apache.org/repos/dist/release/avro/
> > > 2.3.4. I have accounts at https://www.npmjs.com/~ryanskraba,
> > > https://pypi.org/user/RyanSkraba/, and
> > > https://rubygems.org/profiles/RyanSkraba and need permissions to
> > > (eventually) publish artifacts.
> > > 5. Can I get edit permissions on the Avro confluence wiki?
> > >
> > > Is 2,3,4 necessary or does someone already with permissions do the
> > release?
> > >
> > > All my best!  Ryan
> > >
> > >
> > > On Fri, Jan 17, 2020 at 1:47 AM Michael A. Smith  >
> > wrote:
> > > >
> > > > I may have bitten off more than I can chew here. I've been unable to
> > cherry
> > > > pick all the changes from master into 1.9. The python changes
> > themselves
> > > > are not the problem-- I think the problem is that there have been
> > several
> > > > changes to Dockerfile and the build system, some related to python
> and
> > some
> > > > not. Those changes are wide-ranging and not easy to cherry pick. They
> > don't
> > > > affect functionality directly, but if I skip them I don't think the
> > tests
> > > > can pass.
> > > >
> > > > Does anyone with a better understanding of the 1.9 chronology want to
> > give
> > > > it a shot?
> > > >
> > > > On Mon, Jan 1

Re: Avro 1.9.2

[Driesprong, Fokko]

Hi Mike,

Do you know if this introduces any breaking changes to the API? Since this
is a minor update, we should keep the API compatible.

Cheers, Fokko

Op za 11 jan. 2020 om 13:51 schreef Michael A. Smith :

> So far none of my python2/3 changes have been targeting 1.9. They're
> currently just in master. It would be great if someone could cherry pick
> them. I'm away this weekend, but happy to help via phone or more directly
> next week if it turns complex.
>
> Best regards,
> Mike
>
> On Sat, Jan 11, 2020 at 07:39 Driesprong, Fokko 
> wrote:
>
> > Thanks for the review Ryan, appreciate it.
> >
> > I'm happy to help you with the release, if you could pick that up, that
> > would be great. Before starting the release process, I'd like to check if
> > there are commits that are targeted for 1.10 but can be cherry-picked
> back
> > to 1.9.2.
> >
> > For doing the release, there are two important things:
> > - Having the gpg keys in place:
> > https://www.apache.org/dev/release-signing.html
> > - Run the build chain for building the artifacts for the different
> > platforms:
> https://cwiki.apache.org/confluence/display/AVRO/How+To+Release
> >
> > Luckily we have Docker for the latter :-)
> >
> > Cheers, Fokko
> >
> > Op vr 10 jan. 2020 om 17:41 schreef Ryan Skraba :
> >
> > > Hello!  I read and reviewed the PR -- it looks OK to me, is there
> > > something more to do to the current fix?
> > >
> > > I'd be happy to do or help out with the release ... I suspect I might
> > > need a bit of hand-holding for this first time, but the doc looks
> > > pretty complete.  Alternatively, if it's easier, I could shadow
> > > whoever is doing this one and be ready for the next.  It's pretty
> > > exciting to see the cadence pick up for Avro!
> > >
> > > All my best, Ryan
> > >
> > > On Thu, Jan 9, 2020 at 9:48 AM Driesprong, Fokko  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'm working on bumping Apache Avro on the Apache Iceberg project:
> > > > https://github.com/apache/incubator-iceberg/pull/297, I've
> discovered
> > a
> > > > regression bug. I've managed to create a unit test that catches the
> > > issue:
> > > > https://github.com/apache/avro/pull/752. The current fix isn't the
> > > correct
> > > > one. I hope to fix it somewhere this weekend, and afterward, start
> the
> > > > release process for 1.9.2.
> > > >
> > > > If there is anything that you need to have cherry-picked onto the 1.9
> > > > branch, please let me know. Also, if there is anyone who likes to
> > > shepherd
> > > > the release, or wants to help out, please feel free to let me know
> :-)
> > > >
> > > > Thanks all,
> > > >
> > > > Cheers, Fokko
> > >
> >
>

Re: Avro 1.9.2

[Driesprong, Fokko]

Thanks for the review Ryan, appreciate it.

I'm happy to help you with the release, if you could pick that up, that
would be great. Before starting the release process, I'd like to check if
there are commits that are targeted for 1.10 but can be cherry-picked back
to 1.9.2.

For doing the release, there are two important things:
- Having the gpg keys in place:
https://www.apache.org/dev/release-signing.html
- Run the build chain for building the artifacts for the different
platforms: https://cwiki.apache.org/confluence/display/AVRO/How+To+Release

Luckily we have Docker for the latter :-)

Cheers, Fokko

Op vr 10 jan. 2020 om 17:41 schreef Ryan Skraba :

> Hello!  I read and reviewed the PR -- it looks OK to me, is there
> something more to do to the current fix?
>
> I'd be happy to do or help out with the release ... I suspect I might
> need a bit of hand-holding for this first time, but the doc looks
> pretty complete.  Alternatively, if it's easier, I could shadow
> whoever is doing this one and be ready for the next.  It's pretty
> exciting to see the cadence pick up for Avro!
>
> All my best, Ryan
>
> On Thu, Jan 9, 2020 at 9:48 AM Driesprong, Fokko 
> wrote:
> >
> > Hi all,
> >
> > I'm working on bumping Apache Avro on the Apache Iceberg project:
> > https://github.com/apache/incubator-iceberg/pull/297, I've discovered a
> > regression bug. I've managed to create a unit test that catches the
> issue:
> > https://github.com/apache/avro/pull/752. The current fix isn't the
> correct
> > one. I hope to fix it somewhere this weekend, and afterward, start the
> > release process for 1.9.2.
> >
> > If there is anything that you need to have cherry-picked onto the 1.9
> > branch, please let me know. Also, if there is anyone who likes to
> shepherd
> > the release, or wants to help out, please feel free to let me know :-)
> >
> > Thanks all,
> >
> > Cheers, Fokko
>

Avro 1.9.2

[Driesprong, Fokko]

Hi all,

I'm working on bumping Apache Avro on the Apache Iceberg project:
https://github.com/apache/incubator-iceberg/pull/297, I've discovered a
regression bug. I've managed to create a unit test that catches the issue:
https://github.com/apache/avro/pull/752. The current fix isn't the correct
one. I hope to fix it somewhere this weekend, and afterward, start the
release process for 1.9.2.

If there is anything that you need to have cherry-picked onto the 1.9
branch, please let me know. Also, if there is anyone who likes to shepherd
the release, or wants to help out, please feel free to let me know :-)

Thanks all,

Cheers, Fokko

Re: Unified Python Support

[Driesprong, Fokko]

Makes sense Michael.

I'm still working on this one:
https://issues.apache.org/jira/browse/AVRO-2663

The current fix that I came up with does not fix the root cause, but this
is very complex. I'd like to fix this, and then start the release for
1.9.2. Hopefully, I have some time this weekend. Would that work for you?

One more thing, for Airflow we had to release the package name from airflow
<https://pypi.org/project/airflow/> to apache-airflow
<https://pypi.org/project/apache-airflow/>. We've dropped the old one by
throwing an Error when you try to import the package. Would this be
something that we would like to do for Avro in the future? For example,
releasing avro-python3 version 1.10.0 with the sole message of having to
import the avro package? Would like to get your opinion on this.

Cheers, Fokko

Op ma 6 jan. 2020 om 13:32 schreef Michael A. Smith :

> I'd suggest that we do at least one release that has support for both
> python 2 and 3 in the same codebase. This may open doors for folks trying
> to transition from both avro-python3 (lang/py3) to avro (lang/py) as well
> as those trying to go from python 2 to 3 with lang/py.
>
> After that we should officially close out support for python 2.
>
> Please let me know how I can help with the release process. Should we have
> a release soon?
>
> On Sun, Jan 5, 2020 at 13:31 Driesprong, Fokko 
> wrote:
>
> > Thanks for bringing this up Michael and an awesome job on the Python
> part.
> >
> > I'd suggest stopping releasing the avro-python3, and continue only
> > releasing the avro package itself: https://pypi.org/project/avro/
> >
> > This will stop the releases of avro-python3, and in time we can also
> remove
> > it from the git repository. The big question is, are we still going to
> > support Python2 for a while, it is still part of the CI. Supporting only
> > higher versions of Python, such as 3.6, allows us to use new features,
> such
> > as type annotations.
> >
> > Cheers, Fokko
> >
> > Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith <
> mich...@smith-li.com
> > >:
> >
> > > Hi! Given that Python has ended support for python 2 as of the first, I
> > > went ahead and merged the PR. Test coverage is pretty good, so I'm
> fairly
> > > confident; however this is a big change, involving nearly every module
> in
> > > the python part of the project.
> > >
> > > So I'm wondering how this works when it comes to releasing. There
> aren't
> > > any API changes in the literal implementation. So in that light there
> > isn't
> > > any need to treat this version specially. But Python itself is markedly
> > > different between 2 and 3 in some relevant areas.
> > >
> > > Do we need to do anything different for the next release of the lang/py
> > > codebase?
> > >
> > > Thanks for your guidance!
> > >
> > > On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
> > >
> > > > Hello!  I wanted to make sure to thank you for doing all this
> > > > python2/3 work!  I've learned a lot by watching and reading the
> Python
> > > > PRs coming through.
> > > >
> > > > I did a rough pass through the types of changes and cleanup, and I'm
> > > > pretty happy :D  I'll try to get more thorough pass done, but
> (indeed)
> > > > I probably won't have a lot of time between now and the new year.
> > > >
> > > > All my best, Ryan
> > > >
> > > > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith <
> > mich...@smith-li.com>
> > > > wrote:
> > > > >
> > > > > Hi, I've finished building out a unified python approach in
> lang/py.
> > > > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also
> tested
> > > > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > > > both 2.7 and 3.6.
> > > > >
> > > > > The pull request is here: https://github.com/apache/avro/pull/744
> > > > >
> > > > > I know many people are on holiday or unavailable in the near
> future,
> > > > > but I would really appreciate some eyes on this if you can find the
> > > > > time. The tests give me some confidence, but the change was a
> > > > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > > > strings in substantially different ways.
> > > > >
> > > > > This gives us a path forward to unifying our python support (I
> mean,
> > > > > dropping lang/py3 and focusing on one API in one place) as well as
> > > > > managing the sunset of python 2 support altogether.
> > > > >
> > > > > Thank you for your help with this project, either way!
> > > > >
> > > > > - Michael
> > > >
> > >
> >
>

Re: Unified Python Support

[Driesprong, Fokko]

Thanks for bringing this up Michael and an awesome job on the Python part.

I'd suggest stopping releasing the avro-python3, and continue only
releasing the avro package itself: https://pypi.org/project/avro/

This will stop the releases of avro-python3, and in time we can also remove
it from the git repository. The big question is, are we still going to
support Python2 for a while, it is still part of the CI. Supporting only
higher versions of Python, such as 3.6, allows us to use new features, such
as type annotations.

Cheers, Fokko

Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith :

> Hi! Given that Python has ended support for python 2 as of the first, I
> went ahead and merged the PR. Test coverage is pretty good, so I'm fairly
> confident; however this is a big change, involving nearly every module in
> the python part of the project.
>
> So I'm wondering how this works when it comes to releasing. There aren't
> any API changes in the literal implementation. So in that light there isn't
> any need to treat this version specially. But Python itself is markedly
> different between 2 and 3 in some relevant areas.
>
> Do we need to do anything different for the next release of the lang/py
> codebase?
>
> Thanks for your guidance!
>
> On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
>
> > Hello!  I wanted to make sure to thank you for doing all this
> > python2/3 work!  I've learned a lot by watching and reading the Python
> > PRs coming through.
> >
> > I did a rough pass through the types of changes and cleanup, and I'm
> > pretty happy :D  I'll try to get more thorough pass done, but (indeed)
> > I probably won't have a lot of time between now and the new year.
> >
> > All my best, Ryan
> >
> > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith 
> > wrote:
> > >
> > > Hi, I've finished building out a unified python approach in lang/py.
> > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
> > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > both 2.7 and 3.6.
> > >
> > > The pull request is here: https://github.com/apache/avro/pull/744
> > >
> > > I know many people are on holiday or unavailable in the near future,
> > > but I would really appreciate some eyes on this if you can find the
> > > time. The tests give me some confidence, but the change was a
> > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > strings in substantially different ways.
> > >
> > > This gives us a path forward to unifying our python support (I mean,
> > > dropping lang/py3 and focusing on one API in one place) as well as
> > > managing the sunset of python 2 support altogether.
> > >
> > > Thank you for your help with this project, either way!
> > >
> > > - Michael
> >
>

New Committer: Ryan Skraba

[Driesprong, Fokko]

Folks,

The Project Management Committee (PMC) for Apache Avro has invited Ryan
Skraba to become a committer and we are pleased to announce that he has
accepted. Ryan is actively fixing bugs by providing patches and reviewing
pull requests by others. We're very happy to have him on board.

Being a committer enables easier contribution to the project since there is
no need to go via the patch submission process. This should enable better
productivity.

Please join me in congratulating Ryan on his recognition of great work thus
far in our community.

Cheers, Fokko

Re: Consider Python < 2.7 obsolete

[Driesprong, Fokko]

It makes sense to me to remove Py3, having ant as the build tool is awkward
indeed. Thanks for sharing your vision.

Cheers, Fokko

Op vr 22 nov. 2019 om 16:38 schreef Ivan Greene :

> One more task for this subject, the python 3 implementation does not yet
> have support for Avro logical types as far as I’ve been able to tell. So a
> decent amount of code would need to be ported there.
>
> —Ivan
>
> > On Nov 22, 2019, at 7:40 AM, Ryan Skraba  wrote:
> >
> > Thanks for the info; it sounds reasonable to me!  (A big +1 to getting
> > rid of ant, of course).
> >
> > On Thu, Nov 21, 2019 at 12:56 PM Michael A. Smith 
> wrote:
> >>
> >> i would like to update and maintain the py colleges and deprecate and
> >> eventually remove the py3 one.
> >>
> >> 1. Despite being less modern, the py codebase has been kept somewhat
> more
> >> pythonic. Capitalizing `schema.Parse` and the literal translation of the
> >> java parsing normal form implementation are two oddities we could
> address.
> >> There are several issues and pull requests inquiring why the two python
> >> implementations aren't API compatible.
> >> 2. Several modules in py3 were never completed. I called out txipc as
> >> broken, but the tether stuff is missing entirely.
> >>
> >> Things we need to do to make this possible:
> >>
> >> 1. Make the py codebase compatible with py3.5. I've been working on
> this,
> >> while still trying to maintain 2.7 compatibility for now.
> >> 2. I want to port py3's setup approach, making it possible to package
> and
> >> test py without ant. There are lots of benefits, but the only thing on
> >> topic here is to be able to be able to use multiple python versions at
> the
> >> same time. (We should look at tox soon.)
> >>
> >> What do you think?
> >>
> >> On Thu, Nov 21, 2019 at 04:23 Ryan Skraba  wrote:
> >>
> >>> Tick-tock... just bumping this up as the year end approaches!  Any
> >>> interest in making a statement or plan for python2 support for future
> >>> releases of Avro?
> >>>
> >>> There should be one more maintenance release of python 2.7 in 2020
> >>> (after sunset) for the accumulated fixes.
> >>>
> >>> I'm in the context of looking at the docker+build scripts: keeping or
> >>> dropping the python2 runtime has little significant impact.
> >>>
> >>> Ryan
> >>>
> >>>
> >>>
> >>> On Tue, Jun 25, 2019 at 1:22 PM Michael A. Smith  >
> >>> wrote:
> >>>>
> >>>> Inline…
> >>>>
> >>>> On Tue, Jun 25, 2019 at 05:03 Ismaël Mejía  wrote:
> >>>>
> >>>>> Probably is a good idea that we publish our policy around python
> >>>>> support [1] as other projects have done [2].
> >>>>> I think supporting python 2 makes sense at least for our latest
> >>>>> release of this year so probably 1.9.x or eventually 1.10.x.
> >>>>
> >>>>
> >>>> i agree wholeheartedly, but only python 2.7.
> >>>>
> >>>> I am not at all familiar with our python3 codebase, are we feature
> >>>>> equivalent? otherwise maybe worth to create JIRAs and work on those.
> >>>>
> >>>>
> >>>> Not perfectly, and there is work on that, but the biggest gap is that
> >>>> lang/py is much more extensively tested, but its tests use pyant,
> which I
> >>>> have not yet figured out how to port.
> >>>>
> >>>> [1] https://pythonclock.org/
> >>>>> [2] https://python3statement.org/
> >>>>>
> >>>>> On Tue, Jun 25, 2019 at 9:38 AM Driesprong, Fokko
>  >>>>
> >>>>> wrote:
> >>>>>>
> >>>>>> I'm not sure how much effort we should put into Python2.7 in
> general,
> >>>>> since
> >>>>>> this version is EOL after this year.
> >>>>>>
> >>>>>> Cheers, Fokko
> >>>>>>
> >>>>>> Op ma 24 jun. 2019 om 03:20 schreef Michael A. Smith <
> >>>>> mich...@smith-li.com>:
> >>>>>>
> >>>>>>> There's some not-insignificant complexity in the lang/py codebase
> >>> to
> >>>>>>> support derelict versions of Python. There are polyfills for json,
> >>>>> structs,
> >>>>>>> a whole "StoppableHTTPServer" in avro.tool.
> >>>>>>>
> >>>>>>> I created AVRO-2445 and will start removing this stuff now, but
> >>> wanted
> >>>>> to
> >>>>>>> bounce the idea around the list in case there's some obscure
> >>> reason to
> >>>>> keep
> >>>>>>> these things around.
> >>>>>>>
> >>>>>
> >>>
>
>

Re: Preserving defaults in canonical form schema record fields

[Driesprong, Fokko]

Thanks for bringing this up Michael,

At my current project, I've bumped into this one as well. We're trying to
build a schema registry and take the fingerprint from the canonical schema
in order to check if something changed. The issue here is that the
canonical schema takes the minimal schema that ensures binary
compatibility. The default values are only considered when reading the file
and do not impact the binary. This was the original idea that I could
derive from https://issues.apache.org/jira/browse/AVRO-2299

For me, the first step would be to clarify the Avro spec on the different
schemas, such as plain, canonical, etc. So we are sure that every
implementation has the same notion of a canonical schema.

There is a PR that addresses this issue in the Java implementation and
updates the spec: https://github.com/apache/avro/pull/452 But it looks like
something is off with the PR.

Cheers, Fokko

Op zo 3 nov. 2019 om 22:01 schreef Michael A. Smith :

> I'm picking up the languishing ticket AVRO-1938. One issue with the PR
> (https://github.com/apache/avro/pull/143) is that it strips the
> default value from a field:
>
> input:
> '{"name":"example","type":"record","fields":[{"name":"def","type":"bytes","default":"abc"}]}'
> output:
> '{"name":"example","type":"record","fields":[{"name":"def","type":"bytes"}]}'
>
> The specification
> (
> https://avro.apache.org/docs/1.9.1/spec.html#Parsing+Canonical+Form+for+Schemas
> )
> doesn't address record fields at all. If we are to assume that the
> same rules apply to fields as to other schema parts, then the [STRIP]
> rule says we should drop "default" and "order" from fields. But that
> can't be right -- default is crucial for readers, and two schema
> differing only on by a default are certainly different schema and
> ought to have different fingerprints.
>
> Did I miss something in reading the spec, or is this a gap? How should
> I interpret the spec in implementing parsing canonical form for record
> fields. Specifically, should the canonical form of a record field
> preserve its order and default values in the [STRIP] rule, and if so,
> where do those things go in the [ORDER] rule?
>
> Thanks,
> Michael A. Smith
>

Re: Autolink to JIRA from GitHub

[Driesprong, Fokko]

Hi Michael,

Thanks for pointing that out. Looks great to have this on the project.
After looking in the INFRA Jira, I found the following:
https://jira.apache.org/jira/browse/INFRA-19271

It looks like they are trying to configure the autolink. My
suggestion would be to follow that ticket, and see what comes out of it.

Cheers, Fokko

Op di 22 okt. 2019 om 01:59 schreef Michael A. Smith :

> The webpage at https://avro.apache.org/version_control.html says
>
> > The Avro community welcomes pull requests, but please create an issue in
> Avro's issue tracker for your bug report or contribution as well.
>
> Then in our PR template we request that folks link back to Jira.
>
> GitHub recently added a feature, custom autolink references
> (https://github.blog/2019-10-14-introducing-autolink-references/) that
> will make that much easier for contributors by automatically creating
> the link given a simple ticket prefix. Is this something an Avro
> committer can accomplish, or would it need to go through Apache's
> infrastructure team first?
>
> Thanks,
> Michael
>

Re: [ANNOUNCE] new committer Michael Smith

[Driesprong, Fokko]

Welcome Michael, great to have you as a committer. You've done some awesome
work on the Python part of Avro!

Cheers, Fokko

Op zo 20 okt. 2019 om 10:17 schreef Nándor Kollár :

> Congrats, welcome Michael on board!
>
> On 2019/10/20 03:20:14, Sean Busbey  wrote:
> > Hi folks!
> >
> > I'm very pleased to announce that Michael Smith has accepted the PMC's
> > invitation to become a committer on the Apache Avro project.
> >
> > Please join me in congratulating Michael on this recognition of their
> > great work thus far in our community.
> >
>

Re: DRAFT ASF Board Report - Apache Avro October 2019

[Driesprong, Fokko]

Looks great Sean, thanks for taking care of this.

Great to see that Avro is getting more traction again.

Cheers, Fokko

Op wo 9 okt. 2019 om 17:07 schreef Niels Basjes :

> LGTM.
> +1
>
> On Wed, 9 Oct 2019, 16:06 Sean Busbey,  wrote:
>
> > I went ahead and posted the report, with one minor change noted below. I
> > can still edit it over the next few days if needed.
> >
> > On Wed, Oct 9, 2019 at 12:00 AM Sean Busbey  wrote:
> >
> > >
> > > ## Activity:
> > > The Avro project slowly is improving on our gains from the last year.
> > >
> > > We've gotten a fair bit of feedback on the 1.9.0 release and are
> working
> > > towards establishing a regular cadence for getting the result of that
> > > feedback
> > > into our downstream communities. As noted in the summary, we've gotten
> > the
> > > follow-up 1.9.1 release out after a period of approximately 4 months.
> > >
> > >
> > I added a parenthetical here to make clear that 1.9.1 is a minor release
> > (rather than a maintenance release).
> >
>

Re: [DISCUSS] version numbers and where changes should land

[Driesprong, Fokko]

Thanks Sean for bringing this up.

For the 1.9 branch there were some incompatible changes in the API with
respect to 1.8.2. We've removed Jackson
 and Netty from the public API.
This is actually breaking some of the builds
, so, unfortunately,
it isn't compatible, and therefore the major version bump.

The 1.9.x branch still has support for the Joda time library, but defaults
to jsr310, but is still compatible (I believe). For 1.10 the plan is to
completely remove Joda from the codebase since it is officially deprecated
in favor of Java8 time (jsr310). A lot of this stuff is just changes to the
Java API of Avro, which mostly involves changes to the LogicalTypes, so the
actual format is still compatible (as it should).

I agree with you Sean, that a lot of the changes that are targeted for 1.10
could be cherry-picked back to the 1.9 branch. If someone is willing to do
this, I would be grateful. However, maintaining a lot of different branches
is quite time-consuming in terms of release management of the different
versions. For Apache Avro 1.9.0 we actually had some regression bugs which
were blocking, therefore the 1.9.1 release.

Personally I don't have big objection on bumping the major version if there
are breaking changes to one of the API's. But a big +1 on having a
standardized approach on the versioning, this also includes a more clear
approach on documenting the upgrade process and a better changelog. I've
added summaries of the releases a the Github releases:
https://github.com/apache/avro/releases but I think having this on the Avro
website might be more appropriate.

Cheers, Fokko Driesprong



Op wo 11 sep. 2019 om 18:17 schreef Ryan Blue :

> > What would it look like if we *did* have to make an incompatible data
> format change after adopting "conventional" library version strings?
>
> Let's call these format v1 and v2. The library must produce v1 by default,
> so it's a matter of having support for writing v2. When the default
> changes to v2, then that behavior change would require a major version
> increase to signal changes to compatibility. I think we would also want
> clear documentation for each version that shows what versions of the format
> it can read, write, and what it will use by default. A table on the site
> would work.
>
> On Tue, Sep 10, 2019 at 2:51 PM Sean Busbey  wrote:
>
> > What would it look like if we *did* have to make an incompatible data
> > format change after adopting "conventional" library version strings?
> >
> > What if we version the specification independent from the libraries
> > and then have the docs for the libraries claim spec version
> > compatibility?
> >
> > On Tue, Sep 10, 2019 at 3:55 PM Ryan Blue 
> > wrote:
> > >
> > > +1 for changing the version strings to follow a more standard
> convention.
> > > We don't have any breaking format changes, so I think it is expected
> that
> > > the format compatibility version won't change.
> > >
> > > On Tue, Sep 10, 2019 at 7:28 AM Sean Busbey  wrote:
> > >
> > > > Hi folks!
> > > >
> > > > historically, Avro version numbers have had the form:
> > > >
> > > >  .  .  > version>
> > > >
> > > > That is, the first number says wether or not we expect data
> > > > serialization to be compatible, and the second to say wether we
> expect
> > > > some library will be backwards incompatible however that's defined
> for
> > > > the library's language. For example, in the Java library when we make
> > > > changes to public method signatures such that folks can't just swap
> > > > out jar files of our implementation.
> > > >
> > > > While getting myself up to speed on the state of our release lines, I
> > > > noticed we already have the 1.9 release line in a branch, with master
> > > > set up for the next major library version. JIRA shows ~46 issues that
> > > > are in 1.10 but not in a 1.9 release[1].
> > > >
> > > > I haven't looked at all of them yet, but the few I sampled don't see
> > > > to require a major version increment.
> > > >
> > > > I looked around our site and I also can't find anywhere that we've
> > > > documented our version strings. I know I've been in discussions in
> > > > other communities where our version strings have been surprising.
> e.g.
> > > > folks had assumed they can do a low-effort upgrade from 1.7 to 1.8
> > > > only to find that there were documented incompatibilities and
> behavior
> > > > changes.
> > > >
> > > > Are we actively planning on rolling out 1.10? (like, do we have a
> goal
> > > > date?)
> > > >
> > > > I know that when 1.9 went out we EOLed 1.7 and 1.8 in part due to the
> > > > overhead of trying to maintain multiple release lines (especially
> once
> > > > that had so much baggage) while we're trying to reestablish good
> > > > habits on release cadence. How many major version are we planning to
> > > > keep going once 1.10 is ready?
> > > >
> > > > What do folks think about 

Re: [VOTE] Release Apache Avro RC3

[Driesprong, Fokko]

I've released the artifacts for Python2, Python3, Ruby, Javascript and
Java.
I'm not experienced with Perl or C(++) anyone who's comfortable doing this?
Brian, could you publish the C# artifacts? Thanks all!

Cheers, Fokko

Op ma 2 sep. 2019 om 09:27 schreef Driesprong, Fokko :

> I'm happy to announce that the vote has been passed. I'll start publishing
> the artifacts soon.
>
> Cheers, Fokko
>
> Op do 29 aug. 2019 om 13:52 schreef Daniel Kulp :
>
>> +1
>>
>> Did some java testing as well as checked parts of the build in docker.
>> Looks good.
>>
>> Dan
>>
>>
>> > On Aug 28, 2019, at 6:05 AM, Driesprong, Fokko 
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > I'm delighted to propose the following RC to be released as official
>> Apache
>> > Avro 1.9.1 release.
>> >
>> > The commit id is 89218262cde62e98fcb3778b86cd3f03056c54f3
>> > * This corresponds to the tag: release-1.9.1-rc3
>> > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc3
>> >
>> > The release tarball, signature, and checksums are here:
>> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc3/
>> >
>> > You can find the KEYS file here:
>> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
>> >
>> > Binary artifacts for Java are staged in Nexus here:
>> > *
>> >
>> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
>> >
>> > This release includes 31 Jira issues:
>> >
>> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
>> > * Most important, fix regression issues:
>> >  * Java: decoding schema's:
>> https://jira.apache.org/jira/browse/AVRO-2400
>> >  * .Net: Performance issue:
>> https://jira.apache.org/jira/browse/AVRO-2396
>> > * Java: Make org.apache.avro.Schema serializable
>> > * Java: Ability to add custom objects to Velocity templating
>> > * Improved interoperability testing
>> > * Remove possible NPE's
>> > * Upgrade dependencies to latest to the latest version
>> > * And many more :-)
>> >
>> > Since RC1:
>> > - Expand access of ProtobufData methods:
>> >
>> https://github.com/apache/avro/commit/9f5209caeadd45d7a7787005ec1a106540f93c88
>> > - Fix std::string(NULL) in DataFile.cc:
>> >
>> https://github.com/apache/avro/commit/2dfb04d9114270b55b079617066f52b602a703d2
>> > - Add Automatic-Module-Name headers for Avro modules:
>> >
>> https://github.com/apache/avro/commit/e189fe82a71822bede60b60fc74e11f250039f6d
>> > - Move Perf module classes into its own package:
>> >
>> https://github.com/apache/avro/commit/4efbf589732f615400636d78cd02389bf3d3bef2
>> > - Exclude test resources from license check:
>> >
>> https://github.com/apache/avro/commit/aa9abbabd0efca5d86d33a1db74dcbb36203f607
>> >
>> > Since RC2:
>> > - AVRO-2531: Update commons-compress to version 1.19:
>> > https://github.com/apache/avro/pull/625
>> >
>> > Please download, verify, and test. This vote will remain open for at
>> least
>> > 72 hours. Given sufficient votes, I would like to close it on or about
>> > midnight
>> > on Saturday, 31 Augusts 2019.
>> >
>> > [ ] +1 Release this as Apache Avro 1.9.1
>> > [ ] -1 Do not release this because...
>> >
>> > Consider this a +1 (binding) from my side:
>> > * Compiled against Divolte collector and Iceberg
>> > * Checked the licenses using `mvn verify`
>> >
>> > Cheers, Fokko Driesprong
>>
>> --
>> Daniel Kulp
>> dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
>> http://dankulp.com/blog>
>> Talend Community Coder - http://talend.com <http://coders.talend.com/>
>>
>

Re: [VOTE] Release Apache Avro RC3

[Driesprong, Fokko]

I'm happy to announce that the vote has been passed. I'll start publishing
the artifacts soon.

Cheers, Fokko

Op do 29 aug. 2019 om 13:52 schreef Daniel Kulp :

> +1
>
> Did some java testing as well as checked parts of the build in docker.
> Looks good.
>
> Dan
>
>
> > On Aug 28, 2019, at 6:05 AM, Driesprong, Fokko 
> wrote:
> >
> > Hi everyone,
> >
> > I'm delighted to propose the following RC to be released as official
> Apache
> > Avro 1.9.1 release.
> >
> > The commit id is 89218262cde62e98fcb3778b86cd3f03056c54f3
> > * This corresponds to the tag: release-1.9.1-rc3
> > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc3
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc3/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> >
> > This release includes 31 Jira issues:
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > * Most important, fix regression issues:
> >  * Java: decoding schema's:
> https://jira.apache.org/jira/browse/AVRO-2400
> >  * .Net: Performance issue:
> https://jira.apache.org/jira/browse/AVRO-2396
> > * Java: Make org.apache.avro.Schema serializable
> > * Java: Ability to add custom objects to Velocity templating
> > * Improved interoperability testing
> > * Remove possible NPE's
> > * Upgrade dependencies to latest to the latest version
> > * And many more :-)
> >
> > Since RC1:
> > - Expand access of ProtobufData methods:
> >
> https://github.com/apache/avro/commit/9f5209caeadd45d7a7787005ec1a106540f93c88
> > - Fix std::string(NULL) in DataFile.cc:
> >
> https://github.com/apache/avro/commit/2dfb04d9114270b55b079617066f52b602a703d2
> > - Add Automatic-Module-Name headers for Avro modules:
> >
> https://github.com/apache/avro/commit/e189fe82a71822bede60b60fc74e11f250039f6d
> > - Move Perf module classes into its own package:
> >
> https://github.com/apache/avro/commit/4efbf589732f615400636d78cd02389bf3d3bef2
> > - Exclude test resources from license check:
> >
> https://github.com/apache/avro/commit/aa9abbabd0efca5d86d33a1db74dcbb36203f607
> >
> > Since RC2:
> > - AVRO-2531: Update commons-compress to version 1.19:
> > https://github.com/apache/avro/pull/625
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close it on or about
> > midnight
> > on Saturday, 31 Augusts 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.1
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (binding) from my side:
> > * Compiled against Divolte collector and Iceberg
> > * Checked the licenses using `mvn verify`
> >
> > Cheers, Fokko Driesprong
>
> --
> Daniel Kulp
> dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com <http://coders.talend.com/>
>

Re: [Announce] Please welcome Nándor Kollár to the Apache Avro PMC

[Driesprong, Fokko]

Welcome Nándor, great to have you on the PMC! 

Op za 31 aug. 2019 om 09:20 schreef Niels Basjes :

> Welcome!
>
> On Fri, Aug 30, 2019, 23:39 Brian Lachniet  wrote:
>
> > Congratulations, Nándor!
> >
> > On Fri, Aug 30, 2019, 5:37 PM Sean Busbey  wrote:
> >
> >> Hi folks!
> >>
> >> On behalf of the Apache Avro PMC I am pleased to announce that Nándor
> >> Kollár has accepted our invitation to become a PMC member. We
> >> appreciate Nándor stepping up to take more responsibility in the
> >> project.
> >>
> >> Please join me in welcoming Nándor to the Avro PMC!
> >>
> >> As a reminder, if anyone would like to nominate another person as a
> >> committer or PMC member, even if you are not currently a committer or
> >> PMC member, you can always drop a note to priv...@avro.apache.org to
> >> let us know.
> >>
> >
>

[VOTE] Release Apache Avro RC3

[Driesprong, Fokko]

Hi everyone,

I'm delighted to propose the following RC to be released as official Apache
Avro 1.9.1 release.

The commit id is 89218262cde62e98fcb3778b86cd3f03056c54f3
* This corresponds to the tag: release-1.9.1-rc3
* https://github.com/apache/avro/releases/tag/release-1.9.1-rc3

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc3/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/

This release includes 31 Jira issues:
https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
* Most important, fix regression issues:
  * Java: decoding schema's: https://jira.apache.org/jira/browse/AVRO-2400
  * .Net: Performance issue: https://jira.apache.org/jira/browse/AVRO-2396
* Java: Make org.apache.avro.Schema serializable
* Java: Ability to add custom objects to Velocity templating
* Improved interoperability testing
* Remove possible NPE's
* Upgrade dependencies to latest to the latest version
* And many more :-)

Since RC1:
- Expand access of ProtobufData methods:
https://github.com/apache/avro/commit/9f5209caeadd45d7a7787005ec1a106540f93c88
- Fix std::string(NULL) in DataFile.cc:
https://github.com/apache/avro/commit/2dfb04d9114270b55b079617066f52b602a703d2
- Add Automatic-Module-Name headers for Avro modules:
https://github.com/apache/avro/commit/e189fe82a71822bede60b60fc74e11f250039f6d
- Move Perf module classes into its own package:
https://github.com/apache/avro/commit/4efbf589732f615400636d78cd02389bf3d3bef2
- Exclude test resources from license check:
https://github.com/apache/avro/commit/aa9abbabd0efca5d86d33a1db74dcbb36203f607

Since RC2:
- AVRO-2531: Update commons-compress to version 1.19:
https://github.com/apache/avro/pull/625

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 31 Augusts 2019.

[ ] +1 Release this as Apache Avro 1.9.1
[ ] -1 Do not release this because...

Consider this a +1 (binding) from my side:
* Compiled against Divolte collector and Iceberg
* Checked the licenses using `mvn verify`

Cheers, Fokko Driesprong

Re: [VOTE] Release Apache Avro 1.9.1 RC2

[Driesprong, Fokko]

Thanks for bringing this to my attention. I would like to include this into
1.9.1 as well.

Canceling the RC2, and I'll come up with RC3 in the afternoon.

Cheers, Fokko

Op wo 28 aug. 2019 om 09:46 schreef Ismaël Mejía :

> Hello,
>
> I would like to bring the attention to
> https://issues.apache.org/jira/browse/AVRO-2531
> They discovered a CVE that allows to do a DoS via commons-compress.
> Due that this is a 'core' dependency for us. I think we should
> probably do an extra RC to include this one.
>
> WDYT?
>
>
> On Wed, Aug 28, 2019 at 4:08 AM Brian Lachniet 
> wrote:
> >
> > +1
> >
> >- Verified all checksums & signatures
> >- Built and tested C# bindings
> >- Validated NuGet packages for C# bindings
> >
> >
> > On Tue, Aug 27, 2019 at 5:53 PM Daniel Kulp  wrote:
> >
> > > +1
> > >
> > > Did a bunch of testing with the java stuff on my Mac and some minor
> > > testing with the other languages in Docker.   Looks OK.
> > >
> > > Dan
> > >
> > >
> > >
> > >
> > > > On Aug 26, 2019, at 3:34 PM, Driesprong, Fokko  >
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'm delighted to propose the following RC to be released as official
> > > Apache
> > > > Avro 1.9.1 release.
> > > >
> > > > The commit id is aa9abbabd0efca5d86d33a1db74dcbb36203f607
> > > > * This corresponds to the tag: release-1.9.1-rc2
> > > > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc2
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc2/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> > > >
> > > > This release includes 31 Jira issues:
> > > >
> > >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > > > * Most important, fix regression issues:
> > > >  * Java: decoding schema's:
> > > https://jira.apache.org/jira/browse/AVRO-2400
> > > >  * .Net: Performance issue:
> > > https://jira.apache.org/jira/browse/AVRO-2396
> > > > * Java: Make org.apache.avro.Schema serializable
> > > > * Java: Ability to add custom object to Velocity templating
> > > > * Improved interoperability testing
> > > > * Removed NPE's
> > > > * Upgrade dependencies to latest to the latest version
> > > > * And many more :-)
> > > >
> > > > Since RC1:
> > > > - Expand access of ProtobufData methods:
> > > >
> > >
> https://github.com/apache/avro/commit/9f5209caeadd45d7a7787005ec1a106540f93c88
> > > > - Fix std::string(NULL) in DataFile.cc:
> > > >
> > >
> https://github.com/apache/avro/commit/2dfb04d9114270b55b079617066f52b602a703d2
> > > > - Add Automatic-Module-Name headers for Avro modules:
> > > >
> > >
> https://github.com/apache/avro/commit/e189fe82a71822bede60b60fc74e11f250039f6d
> > > > - Move Perf module classes into its own package:
> > > >
> > >
> https://github.com/apache/avro/commit/4efbf589732f615400636d78cd02389bf3d3bef2
> > > > - Exclude test resources from license check:
> > > >
> > >
> https://github.com/apache/avro/commit/aa9abbabd0efca5d86d33a1db74dcbb36203f607
> > > >
> > > > Please download, verify, and test. This vote will remain open for at
> > > least
> > > > 72 hours. Given sufficient votes, I would like to close it on or
> about
> > > > midnight
> > > > on Thursday, 29 August 2019.
> > > >
> > > > [ ] +1 Release this as Apache Avro 1.9.1
> > > > [ ] -1 Do not release this because...
> > > >
> > > > Consider this a +1 (binding) from my side:
> > > > * Compiled against Divolte collector and Iceberg
> > > > * Checked the licenses using `mvn verify`
> > > >
> > > > Cheers, Fokko Driesprong
> > >
> > > --
> > > Daniel Kulp
> > > dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> > > http://dankulp.com/blog>
> > > Talend Community Coder - http://talend.com <http://coders.talend.com/>
> > >
> >
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachn...@gmail.com | blachniet.com <http://www.blachniet.com>
> >
> > <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>

[VOTE] Release Apache Avro 1.9.1 RC2

[Driesprong, Fokko]

Hi everyone,

I'm delighted to propose the following RC to be released as official Apache
Avro 1.9.1 release.

The commit id is aa9abbabd0efca5d86d33a1db74dcbb36203f607
* This corresponds to the tag: release-1.9.1-rc2
* https://github.com/apache/avro/releases/tag/release-1.9.1-rc2

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc2/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/

This release includes 31 Jira issues:
https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
* Most important, fix regression issues:
  * Java: decoding schema's: https://jira.apache.org/jira/browse/AVRO-2400
  * .Net: Performance issue: https://jira.apache.org/jira/browse/AVRO-2396
* Java: Make org.apache.avro.Schema serializable
* Java: Ability to add custom object to Velocity templating
* Improved interoperability testing
* Removed NPE's
* Upgrade dependencies to latest to the latest version
* And many more :-)

Since RC1:
- Expand access of ProtobufData methods:
https://github.com/apache/avro/commit/9f5209caeadd45d7a7787005ec1a106540f93c88
- Fix std::string(NULL) in DataFile.cc:
https://github.com/apache/avro/commit/2dfb04d9114270b55b079617066f52b602a703d2
- Add Automatic-Module-Name headers for Avro modules:
https://github.com/apache/avro/commit/e189fe82a71822bede60b60fc74e11f250039f6d
- Move Perf module classes into its own package:
https://github.com/apache/avro/commit/4efbf589732f615400636d78cd02389bf3d3bef2
- Exclude test resources from license check:
https://github.com/apache/avro/commit/aa9abbabd0efca5d86d33a1db74dcbb36203f607

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Thursday, 29 August 2019.

[ ] +1 Release this as Apache Avro 1.9.1
[ ] -1 Do not release this because...

Consider this a +1 (binding) from my side:
* Compiled against Divolte collector and Iceberg
* Checked the licenses using `mvn verify`

Cheers, Fokko Driesprong

Re: [ANNOUNCE] Please welcome Brian Lachniet to the Apache Avro PMC

[Driesprong, Fokko]

Congrats Brian! Well deserved!

Cheers, Fokko

Op za 24 aug. 2019 om 18:57 schreef Niels Basjes 

> Welcome aboard
>
> On Sat, Aug 24, 2019, 17:40 Sean Busbey  wrote:
>
> > Hi folks!
> >
> > On behalf of the Apache Avro PMC I am pleased to announce that Brian
> > Lachniet has accepted our invitation to become a PMC member. We
> > appreciate Brian stepping up to take more responsibility in the
> > project.
> >
> > Please join me in welcoming Brian to the Avro PMC!
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@avro.apache.org to
> > let us know.
> >
> > -busbey
> >
>

Re: [VOTE] Release Apache Avro 1.9.1 RC1

[Driesprong, Fokko]

I'm fine with merging it into 1.9.1 since it is only the performance tests
and no core components of Avro. The impact will be minimal and I would like
to keep it Java11 compatible.

It would be nice to introduce some check to capture this in the future.

Cheers, Fokko

Op wo 21 aug. 2019 om 16:52 schreef Ismaël Mejía :

> Hello,
>
> I have a question, while checking some Java 11 improvements I
> discovered that the Perf module that we introduced in 1.9.0 has a
> split package issue that can cause conflicts with Java 11. I was
> wondering if we could fix this (the fix is just to isolate the module
> classes into its own package), however this is not backwards
> compatible with Avro 1.9.0. So should we get this into this release?
> or be strict and let it for 1.10.x?
>
> I just opened the PR for the solution for master (1.10.0) [1] but was
> thinking that it is probably worth to cherry pick it and include it in
> 1.9.1 even if breaking the previous version compatiblity, because it
> is quite new and I expect it has low impact.
>
> Any opinions?
>
> [1] https://github.com/apache/avro/pull/615
>
>
> https://issues.apache.org/jira/browse/AVRO-2517
>
> On Wed, Aug 21, 2019 at 11:22 AM Driesprong, Fokko 
> wrote:
> >
> > Thank you Brian,
> >
> > I'm canceling the RC1 and I'm doing some checks on the 1.9 branch. Seeing
> > some funky stuff with the RAT checker. I'll prepare for RC2 this evening.
> >
> > Cheers, Fokko
> >
> >
> >
> >
> > Op wo 21 aug. 2019 om 04:17 schreef Brian Lachniet  >:
> >
> > > It sounds like an rc2 is imminent, but I went ahead and ran some basic
> > > checks on the C# bindings. I didn't find anything unusual. I published
> > > Apache.Avro
> > > v1.9.1-rc1 <https://www.nuget.org/packages/Apache.Avro/1.9.1-rc1> and
> > > Apache.Avro.Tools
> > > v1.9.1-rc1 <https://www.nuget.org/packages/Apache.Avro.Tools/1.9.1-rc1>
> to
> > > nuget.org.
> > >
> > > On Tue, Aug 20, 2019 at 3:44 PM Driesprong, Fokko  >
> > > wrote:
> > >
> > > > Thanks, Dan for giving it a try. My mistake, this file comes from the
> > > > test-suite. I'll clean up the repo and publish a new package.
> > > >
> > > > Cheers, Fokko
> > > >
> > > > Op di 20 aug. 2019 om 21:41 schreef Daniel Kulp :
> > > >
> > > > > Java doesn’t build.There is an empty file:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc
> > > > >
> > > > > In the tar that doesn’t seem to be in the git repo.  It’s causing a
> > > > > problem with RAT since there isn’t a license header or anything.
>  I’m
> > > > not
> > > > > sure where that file came from.
> > > > >
> > > > >
> > > > > Dan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko
>  > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'm delighted to propose the following RC to be released as
> official
> > > > > Apache
> > > > > > Avro 1.9.1 release.
> > > > > >
> > > > > > The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> > > > > > * This corresponds to the tag: release-1.9.1-rc1
> > > > > > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> > > > > >
> > > > > > The release tarball, signature, and checksums are here:
> > > > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> > > > > >
> > > > > > You can find the KEYS file here:
> > > > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > > > >
> > > > > > Binary artifacts for Java are staged in Nexus here:
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> > > > > >
> > > > > > This release includes 31 Jira issues:
> > > > > >
> > > > >
> > > >
> > >
> https://jira.apache.org/jira/issues/?jql=project%20%3D

Re: [VOTE] Release Apache Avro 1.9.1 RC1

[Driesprong, Fokko]

Thank you Brian,

I'm canceling the RC1 and I'm doing some checks on the 1.9 branch. Seeing
some funky stuff with the RAT checker. I'll prepare for RC2 this evening.

Cheers, Fokko




Op wo 21 aug. 2019 om 04:17 schreef Brian Lachniet :

> It sounds like an rc2 is imminent, but I went ahead and ran some basic
> checks on the C# bindings. I didn't find anything unusual. I published
> Apache.Avro
> v1.9.1-rc1 <https://www.nuget.org/packages/Apache.Avro/1.9.1-rc1> and
> Apache.Avro.Tools
> v1.9.1-rc1 <https://www.nuget.org/packages/Apache.Avro.Tools/1.9.1-rc1> to
> nuget.org.
>
> On Tue, Aug 20, 2019 at 3:44 PM Driesprong, Fokko 
> wrote:
>
> > Thanks, Dan for giving it a try. My mistake, this file comes from the
> > test-suite. I'll clean up the repo and publish a new package.
> >
> > Cheers, Fokko
> >
> > Op di 20 aug. 2019 om 21:41 schreef Daniel Kulp :
> >
> > > Java doesn’t build.There is an empty file:
> > >
> > >
> > >
> >
> lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc
> > >
> > > In the tar that doesn’t seem to be in the git repo.  It’s causing a
> > > problem with RAT since there isn’t a license header or anything.   I’m
> > not
> > > sure where that file came from.
> > >
> > >
> > > Dan
> > >
> > >
> > >
> > >
> > > > On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko  >
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'm delighted to propose the following RC to be released as official
> > > Apache
> > > > Avro 1.9.1 release.
> > > >
> > > > The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> > > > * This corresponds to the tag: release-1.9.1-rc1
> > > > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> > > >
> > > > This release includes 31 Jira issues:
> > > >
> > >
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > > > * Most important, fix regression issues:
> > > >  * Java: decoding schema's:
> > > https://jira.apache.org/jira/browse/AVRO-2400
> > > >  * .Net: Performance issue:
> > > https://jira.apache.org/jira/browse/AVRO-2396
> > > > * Java: Make org.apache.avro.Schema serializable
> > > > * Java: Ability to add custom object to Velocity templating
> > > > * Improved interoperability testing
> > > > * Removed NPE's
> > > > * Upgrade dependencies to latest to the latest version
> > > > * And many more :-)
> > > >
> > > > Please download, verify, and test. This vote will remain open for at
> > > least
> > > > 72 hours. Given sufficient votes, I would like to close it on or
> about
> > > > midnight
> > > > on Saturday, 24 August 2019.
> > > >
> > > > [ ] +1 Release this as Apache Avro 1.9.1
> > > > [ ] +0
> > > > [ ] -1 Do not release this because...
> > > >
> > > > Consider this a +1 (binding) from my side:
> > > > * Compiled against Divolte collector and Iceberg
> > > >
> > > > Cheers, Fokko Driesprong
> > >
> > > --
> > > Daniel Kulp
> > > dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> > > http://dankulp.com/blog>
> > > Talend Community Coder - http://talend.com <http://coders.talend.com/>
> > >
> >
>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>

Re: [VOTE] Release Apache Avro 1.9.1 RC1

[Driesprong, Fokko]

Thanks, Dan for giving it a try. My mistake, this file comes from the
test-suite. I'll clean up the repo and publish a new package.

Cheers, Fokko

Op di 20 aug. 2019 om 21:41 schreef Daniel Kulp :

> Java doesn’t build.There is an empty file:
>
>
> lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc
>
> In the tar that doesn’t seem to be in the git repo.  It’s causing a
> problem with RAT since there isn’t a license header or anything.   I’m not
> sure where that file came from.
>
>
> Dan
>
>
>
>
> > On Aug 20, 2019, at 2:04 PM, Driesprong, Fokko 
> wrote:
> >
> > Hi everyone,
> >
> > I'm delighted to propose the following RC to be released as official
> Apache
> > Avro 1.9.1 release.
> >
> > The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
> > * This corresponds to the tag: release-1.9.1-rc1
> > * https://github.com/apache/avro/releases/tag/release-1.9.1-rc1
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/
> >
> > This release includes 31 Jira issues:
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
> > * Most important, fix regression issues:
> >  * Java: decoding schema's:
> https://jira.apache.org/jira/browse/AVRO-2400
> >  * .Net: Performance issue:
> https://jira.apache.org/jira/browse/AVRO-2396
> > * Java: Make org.apache.avro.Schema serializable
> > * Java: Ability to add custom object to Velocity templating
> > * Improved interoperability testing
> > * Removed NPE's
> > * Upgrade dependencies to latest to the latest version
> > * And many more :-)
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close it on or about
> > midnight
> > on Saturday, 24 August 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.1
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (binding) from my side:
> > * Compiled against Divolte collector and Iceberg
> >
> > Cheers, Fokko Driesprong
>
> --
> Daniel Kulp
> dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com <http://coders.talend.com/>
>

[VOTE] Release Apache Avro 1.9.1 RC1

[Driesprong, Fokko]

Hi everyone,

I'm delighted to propose the following RC to be released as official Apache
Avro 1.9.1 release.

The commit id is aad028bf84d43cc3481ac8b527f30debbdf213d2
* This corresponds to the tag: release-1.9.1-rc1
* https://github.com/apache/avro/releases/tag/release-1.9.1-rc1

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.1-rc1/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.1/

This release includes 31 Jira issues:
https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.1
* Most important, fix regression issues:
  * Java: decoding schema's: https://jira.apache.org/jira/browse/AVRO-2400
  * .Net: Performance issue: https://jira.apache.org/jira/browse/AVRO-2396
* Java: Make org.apache.avro.Schema serializable
* Java: Ability to add custom object to Velocity templating
* Improved interoperability testing
* Removed NPE's
* Upgrade dependencies to latest to the latest version
* And many more :-)

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 24 August 2019.

[ ] +1 Release this as Apache Avro 1.9.1
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (binding) from my side:
* Compiled against Divolte collector and Iceberg

Cheers, Fokko Driesprong

Re: Run C# Tests on Windows in Travis

[Driesprong, Fokko]

Thanks for contributing Brian,

I have very little knowledge of Windows, but as far as I can tell it looks
good. I do see a lot of warnings in the output, but this can be fixed in
another PR.

Cheers, Fokko

Op za 20 jul. 2019 om 17:17 schreef Brian Lachniet :

> There have been a few recent "close calls" on changes that would have
> broken the C# bindings on Windows. To help avoid this, I'd like to start
> running the C# tests on a Travis Windows agent.
>
> I've submitted AVRO-2467 
> and
> PR #579  to address this new
> feature.
>
> In my PR, the two original build environments work the same as before.
> However, they are launched from new Travis scripts rather than directly
> from the .travis.yml. The Windows environment only runs the C# tests at
> this time.
>
> While I've only got this specialized for the Windows C# build right now, I
> imagine this could be
> further expanded in the future to test other environments easily.
>
> I appreciate any feedback that anyone is willing to share.
>
> Thanks,
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com 
>
>  
>

Release 1.9.1

[Driesprong, Fokko]

Hi all,

I would like to start on the release of Apache Avro 1.9.1. With the 1.9.0
we've released regression which blocks some projects from upgrading to the
new 1.9 branch.

Where AVRO-2400  is the
major issue. This same issue also appears while updating Avro at the
Iceberg project .
Therefore I think it is important to start releasing 1.9.1.

I've created a Jira ticket to keep track of the release:
https://jira.apache.org/jira/browse/AVRO-2476

Please let me know if there are any PR's/commits that you want to have in
this release so we can get an overview of what needs to be done.

Thank you and cheers,
Fokko Driesprong

Re: Should a Schema be serializable in Java?

[Driesprong, Fokko]

Correct me if I'm wrong here. But as far as I understood the way of
serializing the schema is using Avro, as it is part of the file. To avoid
confusion there should be one way of serializing.

However, I'm not sure if this is worth the hassle of not simply
implementing serializable. Also Flink there is a rather far from optimal
implementation:
https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.java#L72
This converts it to JSON and back while distributing the schema to the
executors.

Cheers, Fokko

Op ma 15 jul. 2019 om 23:03 schreef Doug Cutting :

> I can't think of a reason Schema should not implement Serializable.
>
> There's actually already an issue & patch for this:
>
> https://issues.apache.org/jira/browse/AVRO-1852
>
> Doug
>
> On Mon, Jul 15, 2019 at 6:49 AM Ismaël Mejía  wrote:
>
> > +dev@avro.apache.org
> >
> > On Mon, Jul 15, 2019 at 3:30 PM Ryan Skraba  wrote:
> > >
> > > Hello!
> > >
> > > I'm looking for any discussion or reference why the Schema object isn't
> > serializable -- I'm pretty sure this must have already been discussed
> (but
> > the keywords +avro +serializable +schema have MANY results in all the
> > searches I did: JIRA, stack overflow, mailing list, web)
> > >
> > > In particular, I was at a demo today where we were asked why Schemas
> > needed to be passed as strings to run in distributed tasks.  I remember
> > running into this problem years ago with MapReduce, and again in Spark,
> and
> > again in Beam...
> > >
> > > Is there any downside to making a Schema implement
> > java.lang.Serializable?  The only thing I can think of is that the schema
> > _should not_ be serialized with the data, and making it non-serializable
> > loosely enforces this (at the cost of continually writing different
> > flavours of "Avro holders" for when you really do want to serialize it).
> > >
> > > Willing to create a JIRA and work on the implementation, of course!
> > >
> > > All my best, Ryan
> >
>

Re: sharp edges on moving to 1.9.0

[Driesprong, Fokko]

Hi Sean,

Thanks for picking this up right away. I think we should see if we can fix
the snappy issue. What kind of throwable are you seeing?

A few things have been updated.


*Deprecate Joda-Time in favor of Java8 JSR310*
Right now Avro uses JSR310 (Java8) Time by default and replaces the Joda
time library. You can still use the Joda library using by a setting. If
you're using logical times this might impact your application.


*Move from Jackson 1.x to 2.9*
The schema of Avro is encoded in JSON. The handling of reading the JSON is
being done by Java's popular library called Jackson. The old Codehaus
Jackson 1.x has been replaced by FasterXML's Jackson 2.9. It is always
important to keep Jackson up to date since it is prone to security issues.
Also, the Jackson library has been removed from the public API. So:
import org.codehaus.jackson.node.NullNode;
new Schema.Field(name, schema, doc, NullNode.getInstance());

Will become:
import org.apache.avro.JsonProperties;
new Schema.Field(name, schema, doc, JsonProperties.NULL_VALUE);

This is also important if you shade Jackson away, to make sure to switch
from codehaus to fasterxml.

One issue https://jira.apache.org/jira/browse/AVRO-2400 which is blocking
for Spark.

In general, there are some rough edges, but most of the time it is just API
and binary compatible, with the few exceptions above. Right now we're
trying to find these issues, and fix them in 1.9.1.

Cheers, Fokko



Op di 25 jun. 2019 om 17:54 schreef Sean Busbey :

> today I tried to move a java project from 1.7.7 to 1.9.0 and the first
> issue I hit was an unexpected NPE when using CodecFactory.
>
> Specifically I was trying to get the string name of the SnappyCodec as
> a default cli option. This was fine on 1.7 and 1.8 and in 1.9 fails.
> This was surprising because I didn't recall anything about snappy
> changes that would make me expect to do work on this part of things.
>
> I have chased it down to this commit:
>
>
> https://github.com/apache/avro/commit/21098529e659743e551f2c35652d1a93c1528082
>
> Unfortunately the commit does not have a JIRA key, and I haven't been
> able to reverse engineer where it came in. I'd like to work it out so
> I can add a release note recommending how to work around the issue.
>
> This also led me to a related problem: looking at release notes so I
> can figure out what to expect when upgrading. Where should I be
> looking?
>
> On Tue, Jun 25, 2019 at 10:51 AM Sean Busbey  wrote:
> >
> > What difficulties are folks running into when trying to move to 1.9.0 so
> far?
> >
> > --
> > busbey
>
>
>
> --
> busbey
>

Re: Consider Python < 2.7 obsolete

[Driesprong, Fokko]

I'm not sure how much effort we should put into Python2.7 in general, since
this version is EOL after this year.

Cheers, Fokko

Op ma 24 jun. 2019 om 03:20 schreef Michael A. Smith :

> There's some not-insignificant complexity in the lang/py codebase to
> support derelict versions of Python. There are polyfills for json, structs,
> a whole "StoppableHTTPServer" in avro.tool.
>
> I created AVRO-2445 and will start removing this stuff now, but wanted to
> bounce the idea around the list in case there's some obscure reason to keep
> these things around.
>

Re: Avro C#: Target .NET Standard Only

[Driesprong, Fokko]

Hi Brian,

Do we have something like the number of downloads? To measure how much this
is still being used. I haven't touched a Windows machine in a couple of
years, so for me, it isn't super relevant.

There are some different opinions on if this is breaking or not. At the
Parquet project, we had a similar discussion:
https://lists.apache.org/thread.html/0df375609a2faf384cebc9f0423eead7ca358b73953b07652d7d6a6e@%3Cdev.parquet.apache.org%3E
Parquet wants to drop some old modules that aren't used that much anymore,
you could also argue that you stop releasing the .Net standard.

Cheers, Fokko Driesprong


Op vr 14 jun. 2019 om 07:02 schreef Patrick Farry :

> Hi Brian,
>
> This is good for us at Pitney Bowes, but we don't do anything on Windows so
> its all upside as far as we are concerned.
>
> I'd also add that the current release doesnt build out of the box on macos
> with VS code.
> It would be great if we could just build with 'dotnet build' etc. so this
> could be another reason to make this change.
>
> patrick
>
> On Thu, Jun 13, 2019, 7:08 PM Brian Lachniet  wrote:
>
> > I would like to propose that we update the main Avro C# project to target
> > .NET Standard only. I will lay out a couple reasons below, but first
> let's
> > start with a little context.
> >
> > You can see the different frameworks that the C# project currently
> targets
> > here in the README
> > <
> https://github.com/apache/avro/tree/master/lang/csharp#target-frameworks
> > >.
> > We updated the main Avro library to target .NET Standard with the release
> > of v1.9.0. However, we continue to target the .NET Framework v4.0 as
> well.
> > This allows users that are targeting .NET Framework versions between 4.0
> > and 4.6.1 to still use the library. As you can see in this table on .NET
> > Standard
> > <
> >
> https://docs.microsoft.com/en-us/dotnet/standard/net-standard#net-implementation-support
> > >compatibility,
> > as long as you are targeting .NET Framework 4.6.1 or later OR .NET Core
> 2.0
> > or later, you can use a library that targets .NET Standard.
> >
> > To be clear, we are not dropping support for all of .NET Framework. We
> > would drop support for for any version of .NET Framework before v4.6.1.
> >
> > This change will simplify our release process. At this time, we can only
> > build a NuGet package that contains both .NET Framework and .NET Standard
> > binaries on Windows. This means that after the "official" release is
> > created and deployed, I have to rebuild the project on my Windows machine
> > before publishing the package to nuget.org. With this change, we would
> > only
> > create .NET Standard binaries, which means that we could build the NuGet
> > package on a Linux machine. Then, we could publish the NuGet directly
> from
> > the "official" build to nuget.org. No more side channel builds.
> >
> > This change will also simplify development, particularly when developing
> on
> > non-Windows platforms. I think it's still a good idea to run our unit
> tests
> > on .NET Framework in addition to Core, but we could make it so that those
> > are only run when run on Windows.
> >
> > This is a breaking change for the C# library, so we would need to save
> this
> > for the 1.10 release.
> >
> > Does anyone have any objections, questions or concerns?
> >
> > Thanks,
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachn...@gmail.com | blachniet.com 
> >
> >  
> >
>

Re: [ANNOUNCE] Please welcome Ismaël Mejía to the Apache Avro PMC

[Driesprong, Fokko]

Congrats, welcome Ismaël!

Op di 11 jun. 2019 om 07:02 schreef Niels Basjes 

> Welcome!
>
> On Tue, 11 Jun 2019, 00:01 Brian Lachniet,  wrote:
>
>> Congratulations, Ismaël!
>>
>> On Mon, Jun 10, 2019 at 5:48 PM Jesse Anderson 
>> wrote:
>>
>>> Congrats!
>>>
>>> On Mon, Jun 10, 2019, 4:41 PM Sean Busbey  wrote:
>>>
 Hi folks!

 On behalf of the Apache Avro PMC I am pleased to announce that Ismaël
 Mejía has accepted our invitation to become a PMC member. We
 appreciate Ismaël stepping up to take more responsibility in the
 project.

 Please join me in welcoming Ismaël to the Avro PMC!

 As a reminder, if anyone would like to nominate another person as a
 committer or PMC member, even if you are not currently a committer or
 PMC member, you can always drop a note to priv...@avro.apache.org to
 let us know.

 -busbey

>>>
>>
>> --
>>
>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>
>> Brian Lachniet
>>
>> Software Engineer
>>
>> E: blachn...@gmail.com | blachniet.com 
>>
>>  
>>
>

Re: Automate python formatting

[Driesprong, Fokko]

Why change to pycodestyle? We can still use black for this, right? The
command:
black --check --diff .
will throw a non-zero exit code when there is something to format.

Cheers, Fokko



Op wo 29 mei 2019 om 04:04 schreef Michael A. Smith :

> If all we want is the diff/exit code, then we should use pycodestyle
> instead of black. I'll change course to work on that.
>
> On Tue, May 28, 2019 at 5:24 AM Driesprong, Fokko 
> wrote:
>
> > Fully agree with you Ismaël.
> >
> >
> > The issue is that it CAN write directly. For example here:
> > https://github.com/Fokko/avro/pull/35 If you ask dependabot to merge it,
> > it
> > will be merged into master. And therefore it will ask for write access to
> > the repository. Recently dependabot has been acquired
> > <https://dependabot.com/blog/hello-github/> by Github, so I guess we can
> > ask Github to sign a CLA :-) It is worth a shot of discussing this
> upstream
> > in the ASF. Dependabot letting us know when there is a library with known
> > CVE's will make the software much saver.
> >
> > Cheers, Fokko
> >
> > Op di 28 mei 2019 om 10:46 schreef Ismaël Mejía :
> >
> > > +1 for Black, great idea and in line we the changes we did with
> > > spotless for the Java code base.
> > >
> > > About dependabot I think this is an interesting case to discuss in
> > > upstream apache lists. So far ASF allows bots that do not touch the
> > > code, for example for metadata, as we do with the autolabeler bot. If
> > > I undertand correctly dependabot does not change the code, it opens
> > > Pull Requests and it is up to a committer to decide or not if the code
> > > is good. Or does it write directly?
> > >
> > > I suppose that even if technically is possible, there could be an
> > > authorship issue to be discussed. Can bots sign an ICLA :D living in
> > > the future mates!
> > >
> > > On Tue, May 28, 2019 at 10:01 AM Driesprong, Fokko
>  > >
> > > wrote:
> > > >
> > > > Thanks, Michael for working on this. I think having an auto formatter
> > for
> > > > Python is valuable since it will decrease the conflicts in the
> future.
> > > For
> > > > now, we need to do a big PR to get all the files in the correct
> format.
> > > >
> > > > We need to add to the CI: black --check --diff . This will throw a
> > > non-zero
> > > > exit code if there is something to format. So that the author of the
> PR
> > > > needs to apply black to let the CI pass. We should make this part of
> > the
> > > > build.sh of the python3 project.
> > > >
> > > > I don't think that such a CI service exists, and I also think it is a
> > bad
> > > > idea. The author should format his code on forehand.
> > > >
> > > > I've also looked in setting up Dependabot for Avro, but this is
> against
> > > the
> > > > Apache rules because the Dependabot integration requires write
> > > permissions
> > > > on the repository which isn't allowed. Hope this helps.
> > > >
> > > > Cheers, Fokko
> > > >
> > > > Op di 28 mei 2019 om 03:39 schreef Michael A. Smith <
> > > mich...@smith-li.com>:
> > > >
> > > > > I am working on making all the py and py3 code consistent with
> > > > > https://github.com/python/black, but once done it’d be great if we
> > > could
> > > > > keep it consistent. I will look into adding hooks and stuff for
> > > > > yetus/TravisCI, but is there a way to have an automation that can
> > > > > periodically do all the formatting for us, and open pull requests
> > with
> > > any
> > > > > changes required?
> > > > >
> > > > > I mean like dependabot, but instead of opening a pr to update
> > > dependencies,
> > > > > it opens a pr that does isort, black, docformatter, or whatever we
> > > want.
> > > > >
> > > > > This way, we get consistent style without it being an “enforcement
> > > > > priority” in prs by humans. If someone has a valuable contribution,
> > we
> > > > > don’t have to do a back-and-forth with them about style and
> > formatting.
> > > > >
> > > > > If there’s interest I could look into implementing something with
> > > existing
> > > > > CI tools, or using github actions.
> > > > >
> > > > > What do y’all think?
> > > > >
> > >
> >
>

Re: Automate python formatting

[Driesprong, Fokko]

Fully agree with you Ismaël.


The issue is that it CAN write directly. For example here:
https://github.com/Fokko/avro/pull/35 If you ask dependabot to merge it, it
will be merged into master. And therefore it will ask for write access to
the repository. Recently dependabot has been acquired
<https://dependabot.com/blog/hello-github/> by Github, so I guess we can
ask Github to sign a CLA :-) It is worth a shot of discussing this upstream
in the ASF. Dependabot letting us know when there is a library with known
CVE's will make the software much saver.

Cheers, Fokko

Op di 28 mei 2019 om 10:46 schreef Ismaël Mejía :

> +1 for Black, great idea and in line we the changes we did with
> spotless for the Java code base.
>
> About dependabot I think this is an interesting case to discuss in
> upstream apache lists. So far ASF allows bots that do not touch the
> code, for example for metadata, as we do with the autolabeler bot. If
> I undertand correctly dependabot does not change the code, it opens
> Pull Requests and it is up to a committer to decide or not if the code
> is good. Or does it write directly?
>
> I suppose that even if technically is possible, there could be an
> authorship issue to be discussed. Can bots sign an ICLA :D living in
> the future mates!
>
> On Tue, May 28, 2019 at 10:01 AM Driesprong, Fokko 
> wrote:
> >
> > Thanks, Michael for working on this. I think having an auto formatter for
> > Python is valuable since it will decrease the conflicts in the future.
> For
> > now, we need to do a big PR to get all the files in the correct format.
> >
> > We need to add to the CI: black --check --diff . This will throw a
> non-zero
> > exit code if there is something to format. So that the author of the PR
> > needs to apply black to let the CI pass. We should make this part of the
> > build.sh of the python3 project.
> >
> > I don't think that such a CI service exists, and I also think it is a bad
> > idea. The author should format his code on forehand.
> >
> > I've also looked in setting up Dependabot for Avro, but this is against
> the
> > Apache rules because the Dependabot integration requires write
> permissions
> > on the repository which isn't allowed. Hope this helps.
> >
> > Cheers, Fokko
> >
> > Op di 28 mei 2019 om 03:39 schreef Michael A. Smith <
> mich...@smith-li.com>:
> >
> > > I am working on making all the py and py3 code consistent with
> > > https://github.com/python/black, but once done it’d be great if we
> could
> > > keep it consistent. I will look into adding hooks and stuff for
> > > yetus/TravisCI, but is there a way to have an automation that can
> > > periodically do all the formatting for us, and open pull requests with
> any
> > > changes required?
> > >
> > > I mean like dependabot, but instead of opening a pr to update
> dependencies,
> > > it opens a pr that does isort, black, docformatter, or whatever we
> want.
> > >
> > > This way, we get consistent style without it being an “enforcement
> > > priority” in prs by humans. If someone has a valuable contribution, we
> > > don’t have to do a back-and-forth with them about style and formatting.
> > >
> > > If there’s interest I could look into implementing something with
> existing
> > > CI tools, or using github actions.
> > >
> > > What do y’all think?
> > >
>

Re: Automate python formatting

[Driesprong, Fokko]

Thanks, Michael for working on this. I think having an auto formatter for
Python is valuable since it will decrease the conflicts in the future. For
now, we need to do a big PR to get all the files in the correct format.

We need to add to the CI: black --check --diff . This will throw a non-zero
exit code if there is something to format. So that the author of the PR
needs to apply black to let the CI pass. We should make this part of the
build.sh of the python3 project.

I don't think that such a CI service exists, and I also think it is a bad
idea. The author should format his code on forehand.

I've also looked in setting up Dependabot for Avro, but this is against the
Apache rules because the Dependabot integration requires write permissions
on the repository which isn't allowed. Hope this helps.

Cheers, Fokko

Op di 28 mei 2019 om 03:39 schreef Michael A. Smith :

> I am working on making all the py and py3 code consistent with
> https://github.com/python/black, but once done it’d be great if we could
> keep it consistent. I will look into adding hooks and stuff for
> yetus/TravisCI, but is there a way to have an automation that can
> periodically do all the formatting for us, and open pull requests with any
> changes required?
>
> I mean like dependabot, but instead of opening a pr to update dependencies,
> it opens a pr that does isort, black, docformatter, or whatever we want.
>
> This way, we get consistent style without it being an “enforcement
> priority” in prs by humans. If someone has a valuable contribution, we
> don’t have to do a back-and-forth with them about style and formatting.
>
> If there’s interest I could look into implementing something with existing
> CI tools, or using github actions.
>
> What do y’all think?
>

[Announce] Release of Apache Avro 1.9.0

[Driesprong, Fokko]

Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later, I'm thrilled to announce the release of Avro 1.9.0!

Changes are listed at https://s.apache.org/avro190 A list of highlights of
the new version: https://blog.godatadriven.com/apache-avro-1-9-release

This release can be downloaded from:
https://www.apache.org/dyn/closer.cgi/avro/

Java artifacts are available from Maven Central, Ruby artifacts are
at RubyGems, Python is at PyPI, JS at NPM.

Thanks to everyone for contributing and helping out.

Fokko Driesprong

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Great to see that Maven Repository caught up. I also released the Python3
version: https://pypi.org/project/avro-python3/
<https://pypi.org/project/avro-python3/#history>

I'll get the announce out! Thanks all.

Cheers, Fokko

Op do 23 mei 2019 om 08:41 schreef RumeshKrishnan Mohan <
rumeshkr...@gmail.com>:

> Nice work Fokko and Ismael.  
>
> By - Rumeshkrishnan
>
>
> On Thu, May 23, 2019 at 8:35 AM Ismaël Mejía  wrote:
>
> > For info it seems that mvnrepository has finally catched up.
> > https://mvnrepository.com/artifact/org.apache.avro/avro/1.9.0
> >
> > On Tue, May 21, 2019 at 5:51 PM Ismaël Mejía  wrote:
> > >
> > > Thanks for fixing that Fokko.
> > > Maven Repository always takes 'ages' to catch up but since they are in
> > > Maven Central [1] I think we are good to announce.
> > >
> > > [1]
> https://search.maven.org/artifact/org.apache.avro/avro/1.9.0/bundle
> > >
> > > On Tue, May 21, 2019 at 2:47 PM Driesprong, Fokko  >
> > wrote:
> > > >
> > > > My mistake. I've removed the 1.8.2 version from the mirror. In the
> > > > meantime, I'm still waiting for it to appear on Maven Repository:
> > > > https://mvnrepository.com/artifact/org.apache.avro/avro
> > > >
> > > > It would expect to have it been indexed already. Anyone any idea?
> > > >
> > > > Cheers, Fokko Driesprong
> > > >
> > > > Op di 21 mei 2019 om 10:06 schreef Ismaël Mejía :
> > > >
> > > > > Seems the jars are now everywhere. Are we going to do an
> Announcement
> > > > > (specially to announcement mailing list)?
> > > > > Also the download link points still to both 1.8.2 and 1.9.0 is this
> > > > > wished or we just forgot to remove 1.8.2 ?
> > > > > https://www.apache.org/dyn/closer.cgi/avro/
> > > > >
> > > > > On Sun, May 19, 2019 at 6:59 PM Driesprong, Fokko
> > 
> > > > > wrote:
> > > > > >
> > > > > > Can you elaborate Raman? How did you confirm that it isn't the
> > latest
> > > > > > version?
> > > > > >
> > > > > > Thanks for the heads-up Rumeshkrishnan, I'll check why 1.9.0
> isn't
> > > > > showing
> > > > > > up in Maven Central.
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > > Op vr 17 mei 2019 om 22:28 schreef Raman Gupta <
> > rocketra...@gmail.com>:
> > > > > >
> > > > > > > That's just mvnrepository not showing the latest information.
> > See:
> > > > > > >
> > https://search.maven.org/artifact/org.apache.avro/avro/1.9.0/bundle
> > > > > > >
> > > > > > > On Fri, May 17, 2019 at 4:16 PM RumeshKrishnan Mohan
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I am unable to see 1.9.0 in the artifact in maven,
> > > > > > > > https://mvnrepository.com/artifact/org.apache.avro/avro
> > > > > > > >
> > > > > > > > By -  Rumeshkrishnan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 17, 2019 at 10:09 PM rocketra...@gmail.com <
> > > > > > > > rocketra...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Fokko, did you have a chance to look into this? I just
> built
> > a
> > > > > project
> > > > > > > > > with 1.9.0, and am running into several issues that should
> > have
> > > > > been
> > > > > > > fixed
> > > > > > > > > during the RCs:
> > > > > > > > > https://issues.apache.org/jira/browse/AVRO-2360,
> > > > > > > > > https://issues.apache.org/jira/browse/AVRO-2386, and
> > > > > > > > > https://issues.apache.org/jira/browse/AVRO-2383 which
> > should all
> > > > > have
> > > > > > > > > been fixed.
> > > > > > > > >
> > > > > > > > > Looking at the avro-compiler.jar, all the class files are
> > dated
> > > > > Apr 26
> > > > > > > > > 9:32 E

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

My mistake. I've removed the 1.8.2 version from the mirror. In the
meantime, I'm still waiting for it to appear on Maven Repository:
https://mvnrepository.com/artifact/org.apache.avro/avro

It would expect to have it been indexed already. Anyone any idea?

Cheers, Fokko Driesprong

Op di 21 mei 2019 om 10:06 schreef Ismaël Mejía :

> Seems the jars are now everywhere. Are we going to do an Announcement
> (specially to announcement mailing list)?
> Also the download link points still to both 1.8.2 and 1.9.0 is this
> wished or we just forgot to remove 1.8.2 ?
> https://www.apache.org/dyn/closer.cgi/avro/
>
> On Sun, May 19, 2019 at 6:59 PM Driesprong, Fokko 
> wrote:
> >
> > Can you elaborate Raman? How did you confirm that it isn't the latest
> > version?
> >
> > Thanks for the heads-up Rumeshkrishnan, I'll check why 1.9.0 isn't
> showing
> > up in Maven Central.
> >
> > Cheers, Fokko
> >
> > Op vr 17 mei 2019 om 22:28 schreef Raman Gupta :
> >
> > > That's just mvnrepository not showing the latest information. See:
> > > https://search.maven.org/artifact/org.apache.avro/avro/1.9.0/bundle
> > >
> > > On Fri, May 17, 2019 at 4:16 PM RumeshKrishnan Mohan
> > >  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I am unable to see 1.9.0 in the artifact in maven,
> > > > https://mvnrepository.com/artifact/org.apache.avro/avro
> > > >
> > > > By -  Rumeshkrishnan
> > > >
> > > >
> > > > On Fri, May 17, 2019 at 10:09 PM rocketra...@gmail.com <
> > > > rocketra...@gmail.com> wrote:
> > > >
> > > > > Fokko, did you have a chance to look into this? I just built a
> project
> > > > > with 1.9.0, and am running into several issues that should have
> been
> > > fixed
> > > > > during the RCs:
> > > > > https://issues.apache.org/jira/browse/AVRO-2360,
> > > > > https://issues.apache.org/jira/browse/AVRO-2386, and
> > > > > https://issues.apache.org/jira/browse/AVRO-2383 which should all
> have
> > > > > been fixed.
> > > > >
> > > > > Looking at the avro-compiler.jar, all the class files are dated
> Apr 26
> > > > > 9:32 EDT, which is before some of those issues were even reported.
> > > > >
> > > > > Regards,
> > > > > Raman
> > > > >
> > > > > On 2019/05/15 18:46:03, "Driesprong, Fokko" 
> > > wrote:
> > > > > > Hi Jacob,
> > > > > >
> > > > > > This looks off, I'll dive into it. I think it got released for
> some
> > > > > reason.
> > > > > > While releasing it today, I also noticed that the artifact wasn't
> > > staged
> > > > > in
> > > > > > Nexus anymore. However, RC4 is the same as the release.
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar
> > > > > > :
> > > > > >
> > > > > > > Great news! We have been looking forward to the 1.9 release.
> > > > > > >
> > > > > > > I see 1.9.0 published on Maven Central. It appears to have been
> > > > > published
> > > > > > > 5/8
> > > > > > > (the same day as RC4 was announced but ~1week before vote
> passed).
> > > > > > >
> > > > > > > Is this the release build (presumably the same as RC4) or an
> > > earlier
> > > > > RC?
> > > > > > >
> > > > > > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/
> > > > > > >
> > > > > > > Would it be helpful to have maven include the git sha
> somewhere in
> > > > > > > MANIFEST.MF?
> > > > > > >
> > > > > > > jacob
> > > > > > >
> > > > > > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko
> > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thank you all,
> > > > > > > >
> > > > > > > > I'm happy to announce, the vote for Avro 1.9.0 has passed.
> > > > > > > >
> > > > > > > > Binding

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Can you elaborate Raman? How did you confirm that it isn't the latest
version?

Thanks for the heads-up Rumeshkrishnan, I'll check why 1.9.0 isn't showing
up in Maven Central.

Cheers, Fokko

Op vr 17 mei 2019 om 22:28 schreef Raman Gupta :

> That's just mvnrepository not showing the latest information. See:
> https://search.maven.org/artifact/org.apache.avro/avro/1.9.0/bundle
>
> On Fri, May 17, 2019 at 4:16 PM RumeshKrishnan Mohan
>  wrote:
> >
> > Hi everyone,
> >
> > I am unable to see 1.9.0 in the artifact in maven,
> > https://mvnrepository.com/artifact/org.apache.avro/avro
> >
> > By -  Rumeshkrishnan
> >
> >
> > On Fri, May 17, 2019 at 10:09 PM rocketra...@gmail.com <
> > rocketra...@gmail.com> wrote:
> >
> > > Fokko, did you have a chance to look into this? I just built a project
> > > with 1.9.0, and am running into several issues that should have been
> fixed
> > > during the RCs:
> > > https://issues.apache.org/jira/browse/AVRO-2360,
> > > https://issues.apache.org/jira/browse/AVRO-2386, and
> > > https://issues.apache.org/jira/browse/AVRO-2383 which should all have
> > > been fixed.
> > >
> > > Looking at the avro-compiler.jar, all the class files are dated Apr 26
> > > 9:32 EDT, which is before some of those issues were even reported.
> > >
> > > Regards,
> > > Raman
> > >
> > > On 2019/05/15 18:46:03, "Driesprong, Fokko" 
> wrote:
> > > > Hi Jacob,
> > > >
> > > > This looks off, I'll dive into it. I think it got released for some
> > > reason.
> > > > While releasing it today, I also noticed that the artifact wasn't
> staged
> > > in
> > > > Nexus anymore. However, RC4 is the same as the release.
> > > >
> > > > Cheers, Fokko
> > > >
> > > > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar
> > > > :
> > > >
> > > > > Great news! We have been looking forward to the 1.9 release.
> > > > >
> > > > > I see 1.9.0 published on Maven Central. It appears to have been
> > > published
> > > > > 5/8
> > > > > (the same day as RC4 was announced but ~1week before vote passed).
> > > > >
> > > > > Is this the release build (presumably the same as RC4) or an
> earlier
> > > RC?
> > > > >
> > > > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/
> > > > >
> > > > > Would it be helpful to have maven include the git sha somewhere in
> > > > > MANIFEST.MF?
> > > > >
> > > > > jacob
> > > > >
> > > > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko
>  > > >
> > > > > wrote:
> > > > >
> > > > > > Thank you all,
> > > > > >
> > > > > > I'm happy to announce, the vote for Avro 1.9.0 has passed.
> > > > > >
> > > > > > Binding:
> > > > > > +1 Daniel Kulp
> > > > > > +1 Sean Busbey
> > > > > > +1 Fokko Driesprong
> > > > > >
> > > > > > Non-binding:
> > > > > > +1 Lunjie Jin
> > > > > > +1 Michael A. Smith
> > > > > > +1 Ismaël Mejía
> > > > > > +1 Brian Lachniet
> > > > > >
> > > > > > I would like to thank the community from participating in the
> > > release.
> > > > > Both
> > > > > > in providing patches for Apache Avro and giving the RC's a try.
> > > > > >
> > > > > > Yesterday I did some additional testing against Apache Spark
> > > (similar to
> > > > > > Nandor :-) and Apache Flink and did not find any blocking issues.
> > > There
> > > > > are
> > > > > > some incompatibilities, but most of them are; revoking Jackson
> from
> > > the
> > > > > > public API and deprecating Joda.
> > > > > >
> > > > > > I'll publish the artifacts to the different repositories soon. On
> > > this, I
> > > > > > consider myself a polyglot programmer, but my knowledge is
> limited on
> > > > > .Net,
> > > > > > Perl, and Javascript, so I might ask some help from the community
> > > here as
> > > > > > well.
> > > > 

Re: Getting Avro 1.9 into downstream Apache projects

[Driesprong, Fokko]

Thanks Doug,

My username for both PyPi and Rubygems is Fokko. Also, feel free to push
yourself :-)

Cheers, Fokko

Op do 16 mei 2019 om 19:50 schreef Doug Cutting :

> I'm happy to push these myself, or better yet, permit others.
>
> Please send me PyPi and rubygems.org usernames to add.
>
> Thanks,
>
> Doug
>
> On Thu, May 16, 2019 at 10:13 AM Daniel Kulp  wrote:
>
> >
> >
> > >
> > > Also I'm blocked on verifying the python/ruby code due to the pip/gem
> > > packages not yet having been pushed. Is there an ETA for those?
> >
> > The issue here is that none of the “currently active” Avro committers
> have
> > access to the repositories for those.   We’ve asked to have our accounts
> > added to the Avro “groups” on those repositories, but that hasn’t
> happened
> > yet.   We need for some of the Avro old timers that do have access to
> > either do the push or grant us access so we can do the push.   We’re
> > working on it, just trying to get ahold of them.
> >
> > Dan
> >
> >
> >
> >
> > >
> > > Patrick
> > >
> > > On Thu, May 16, 2019 at 7:43 AM Ismaël Mejía 
> wrote:
> > >
> > >> This is a fork of the discussion started in the release vote thread
> [1].
> > >>
> > >> Just to keep the discussion going and follow up on issues and projects
> > >> that deserve attention.
> > >> So far open issues are:
> > >>
> > >> BEAM-7328 Update Avro to version 1.9.0 in Java SDK
> > >> HIVE-21737 Upgrade Avro to version 1.9.0
> > >> FLINK-12532 Upgrade Avro to version 1.9.0
> > >> PARQUET-1576 Upgrade to Avro 1.9.0
> > >> SPARK-27733 Upgrade to Avro 1.9.x
> > >>
> > >> Don't hesitate to create new issues if required and bring info in this
> > >> thread that could be useful for others working on this.
> > >>
> > >> [1]
> > >>
> >
> https://lists.apache.org/thread.html/ab598c36fd823a3fc563f7c621f957f0c2ef09513d4e8428391ddb69@%3Cdev.avro.apache.org%3E
> > >>
> >
> > --
> > Daniel Kulp
> > dk...@apache.org  - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com 
> >
>

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Thanks Nandor,

Upgrading Hive 2.x (Hadoop 2) is going to be painful since this is still on
Java 1.7. It is still relying on Avro 1.7.7. This is an issue because this
is also blocking upgrading Parquet, this implies that Hive 2.x does not
support reading lz4 and zstd compression.

>From Hive 3.x the Java version is 1.8 and we could upgrade Avro. The
current version is 1.8.2, so that should be doable.

Cheers, Fokko

Op wo 15 mei 2019 om 23:51 schreef Nandor Kollar
:

> If Spark is upgraded, then I'm afraid first Hive should be upgraded to
> 1.9.0. When I did a quick release verification, I remember I saw several
> failures due to Hive using deprecated and removed Avro methods (related to
> Jackson removal from public API).
>
> On Wed, May 15, 2019 at 11:04 PM Ismaël Mejía  wrote:
>
> > Getting a new release of Avro feels amazing, thanks a lot Fokko for
> > all your wokr to get this out with the release.
> >
> > An interesting next step is downstreaming. I saw Fokko created a
> > ticket for Parquet, is anyone working on the Spark one? It will be
> > really nice to get this into Spark 3. Anything I can do to help there
> > or that we can sync work?
> > I filled https://issues.apache.org/jira/browse/SPARK-27733 for the
> moment.
> >
> > I am tackling the upgrade on Beam (the other project I work mostly on
> too.
> > https://issues.apache.org/jira/browse/BEAM-7328
> >
> > Any other project worth the sync/work?
> >
> >
> > On Wed, May 15, 2019 at 8:54 PM Driesprong, Fokko 
> > wrote:
> > >
> > > Hi Jacob,
> > >
> > > This looks off, I'll dive into it. I think it got released for some
> > reason.
> > > While releasing it today, I also noticed that the artifact wasn't
> staged
> > in
> > > Nexus anymore. However, RC4 is the same as the release.
> > >
> > > Cheers, Fokko
> > >
> > > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar
> > > :
> > >
> > > > Great news! We have been looking forward to the 1.9 release.
> > > >
> > > > I see 1.9.0 published on Maven Central. It appears to have been
> > published
> > > > 5/8
> > > > (the same day as RC4 was announced but ~1week before vote passed).
> > > >
> > > > Is this the release build (presumably the same as RC4) or an earlier
> > RC?
> > > >
> > > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/
> > > >
> > > > Would it be helpful to have maven include the git sha somewhere in
> > > > MANIFEST.MF?
> > > >
> > > > jacob
> > > >
> > > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko
>  > >
> > > > wrote:
> > > >
> > > > > Thank you all,
> > > > >
> > > > > I'm happy to announce, the vote for Avro 1.9.0 has passed.
> > > > >
> > > > > Binding:
> > > > > +1 Daniel Kulp
> > > > > +1 Sean Busbey
> > > > > +1 Fokko Driesprong
> > > > >
> > > > > Non-binding:
> > > > > +1 Lunjie Jin
> > > > > +1 Michael A. Smith
> > > > > +1 Ismaël Mejía
> > > > > +1 Brian Lachniet
> > > > >
> > > > > I would like to thank the community from participating in the
> > release.
> > > > Both
> > > > > in providing patches for Apache Avro and giving the RC's a try.
> > > > >
> > > > > Yesterday I did some additional testing against Apache Spark
> > (similar to
> > > > > Nandor :-) and Apache Flink and did not find any blocking issues.
> > There
> > > > are
> > > > > some incompatibilities, but most of them are; revoking Jackson from
> > the
> > > > > public API and deprecating Joda.
> > > > >
> > > > > I'll publish the artifacts to the different repositories soon. On
> > this, I
> > > > > consider myself a polyglot programmer, but my knowledge is limited
> on
> > > > .Net,
> > > > > Perl, and Javascript, so I might ask some help from the community
> > here as
> > > > > well.
> > > > >
> > > > > Thanks all, and keep up the good work.
> > > > >
> > > > > Cheers, Fokko Driesprong
> > > > >
> > > > > Op di 14 mei 2019 om 14:48 schreef Nandor Kollar
> > > > > :
> > > > >
> > > > > > +1 (non-bindin

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

I just updated the docs:
https://avro.apache.org/docs/current/

Still waiting on some permissions to fix some other languages.

For the Apache Parquet one I ended up fixing more regression stuff than I
expected. Especially that the NullNode wasn't working anymore:
https://github.com/apache/parquet-mr/pull/638/files#diff-254da5a3fbfdb2d20acea16d54b9dc62L51

This now throws an exception. Don't know why I didn't saw this before.

I've started on updating Flink as well, but that one still needs a bit more
love. Replacing the Joda stuff with JSR310 requires some more attention. If
someone else feels picking this up, this is fine from my side as well.
I do would like to do the Incubating Druid one if no one objects.

Cheers, Fokko

Op wo 15 mei 2019 om 23:04 schreef Ismaël Mejía 

> Getting a new release of Avro feels amazing, thanks a lot Fokko for
> all your wokr to get this out with the release.
>
> An interesting next step is downstreaming. I saw Fokko created a
> ticket for Parquet, is anyone working on the Spark one? It will be
> really nice to get this into Spark 3. Anything I can do to help there
> or that we can sync work?
> I filled https://issues.apache.org/jira/browse/SPARK-27733 for the moment.
>
> I am tackling the upgrade on Beam (the other project I work mostly on too.
> https://issues.apache.org/jira/browse/BEAM-7328
>
> Any other project worth the sync/work?
>
>
> On Wed, May 15, 2019 at 8:54 PM Driesprong, Fokko 
> wrote:
> >
> > Hi Jacob,
> >
> > This looks off, I'll dive into it. I think it got released for some
> reason.
> > While releasing it today, I also noticed that the artifact wasn't staged
> in
> > Nexus anymore. However, RC4 is the same as the release.
> >
> > Cheers, Fokko
> >
> > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar
> > :
> >
> > > Great news! We have been looking forward to the 1.9 release.
> > >
> > > I see 1.9.0 published on Maven Central. It appears to have been
> published
> > > 5/8
> > > (the same day as RC4 was announced but ~1week before vote passed).
> > >
> > > Is this the release build (presumably the same as RC4) or an earlier
> RC?
> > >
> > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/
> > >
> > > Would it be helpful to have maven include the git sha somewhere in
> > > MANIFEST.MF?
> > >
> > > jacob
> > >
> > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko  >
> > > wrote:
> > >
> > > > Thank you all,
> > > >
> > > > I'm happy to announce, the vote for Avro 1.9.0 has passed.
> > > >
> > > > Binding:
> > > > +1 Daniel Kulp
> > > > +1 Sean Busbey
> > > > +1 Fokko Driesprong
> > > >
> > > > Non-binding:
> > > > +1 Lunjie Jin
> > > > +1 Michael A. Smith
> > > > +1 Ismaël Mejía
> > > > +1 Brian Lachniet
> > > >
> > > > I would like to thank the community from participating in the
> release.
> > > Both
> > > > in providing patches for Apache Avro and giving the RC's a try.
> > > >
> > > > Yesterday I did some additional testing against Apache Spark
> (similar to
> > > > Nandor :-) and Apache Flink and did not find any blocking issues.
> There
> > > are
> > > > some incompatibilities, but most of them are; revoking Jackson from
> the
> > > > public API and deprecating Joda.
> > > >
> > > > I'll publish the artifacts to the different repositories soon. On
> this, I
> > > > consider myself a polyglot programmer, but my knowledge is limited on
> > > .Net,
> > > > Perl, and Javascript, so I might ask some help from the community
> here as
> > > > well.
> > > >
> > > > Thanks all, and keep up the good work.
> > > >
> > > > Cheers, Fokko Driesprong
> > > >
> > > > Op di 14 mei 2019 om 14:48 schreef Nandor Kollar
> > > > :
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * verified signatures and checksums
> > > > > * ran unit tests, passed
> > > > > * did a quick sanity test against Spark master: upgraded Spark Avro
> > > > > dependency to this release candidate, ran spark-avro tests, all
> passed
> > > > (for
> > > > > the record, two minor Maven changes are required due to removal of
> > > > Hadoop 1
> > > > > support and making org.tukaan

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Hi Jacob,

This looks off, I'll dive into it. I think it got released for some reason.
While releasing it today, I also noticed that the artifact wasn't staged in
Nexus anymore. However, RC4 is the same as the release.

Cheers, Fokko

Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar
:

> Great news! We have been looking forward to the 1.9 release.
>
> I see 1.9.0 published on Maven Central. It appears to have been published
> 5/8
> (the same day as RC4 was announced but ~1week before vote passed).
>
> Is this the release build (presumably the same as RC4) or an earlier RC?
>
> https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/
>
> Would it be helpful to have maven include the git sha somewhere in
> MANIFEST.MF?
>
> jacob
>
> On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko 
> wrote:
>
> > Thank you all,
> >
> > I'm happy to announce, the vote for Avro 1.9.0 has passed.
> >
> > Binding:
> > +1 Daniel Kulp
> > +1 Sean Busbey
> > +1 Fokko Driesprong
> >
> > Non-binding:
> > +1 Lunjie Jin
> > +1 Michael A. Smith
> > +1 Ismaël Mejía
> > +1 Brian Lachniet
> >
> > I would like to thank the community from participating in the release.
> Both
> > in providing patches for Apache Avro and giving the RC's a try.
> >
> > Yesterday I did some additional testing against Apache Spark (similar to
> > Nandor :-) and Apache Flink and did not find any blocking issues. There
> are
> > some incompatibilities, but most of them are; revoking Jackson from the
> > public API and deprecating Joda.
> >
> > I'll publish the artifacts to the different repositories soon. On this, I
> > consider myself a polyglot programmer, but my knowledge is limited on
> .Net,
> > Perl, and Javascript, so I might ask some help from the community here as
> > well.
> >
> > Thanks all, and keep up the good work.
> >
> > Cheers, Fokko Driesprong
> >
> > Op di 14 mei 2019 om 14:48 schreef Nandor Kollar
> > :
> >
> > > +1 (non-binding)
> > >
> > > * verified signatures and checksums
> > > * ran unit tests, passed
> > > * did a quick sanity test against Spark master: upgraded Spark Avro
> > > dependency to this release candidate, ran spark-avro tests, all passed
> > (for
> > > the record, two minor Maven changes are required due to removal of
> > Hadoop 1
> > > support and making org.tukaani.xz provided dependency - looks like
> Spark
> > > tests against each codec). spark-hive tests failed, because deprecated
> > > methods were removed in Avro (related to Jackson removal from public
> API,
> > > and Hive uses them), but I think it is expected, upgrading to a major
> > > release could have breaking changes.
> > >
> > > Nandor
> > >
> > > On Sun, May 12, 2019 at 7:11 AM Sean Busbey  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > -- Good:
> > > > * signatures
> > > > * checksums
> > > > * tag/ref lines up with src artifact (except for one file mentioned
> > > below)
> > > > * LICENSE/NOTICE spot check
> > > > * apache rat says the license files for all the artifacts are fine
> > > (except
> > > > it couldn't understand the gem; using the gem cli confirms it's also
> > > > correct)
> > > >
> > > > -- We Can Be Better Later:
> > > > * filed AVRO-2395 because the "java" convenience binary part of the
> > dist
> > > > section is redundant
> > > > * in the future please post the specific staged maven repository
> > instead
> > > > of somewhere within the staged repository group. that'll help us
> avoid
> > > > possible conflicts should someone forget to cancel a staged RC or
> > > > accidentally stage an additional repository after the vote is called.
> > > FWIW
> > > > as far as I can tell from the Nexus UI, this is the staged repo for
> > this
> > > > VOTE (and it's what I verified):
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapacheavro-1020/
> > > >
> > > > * When unpacking the source tarball I got this warning on OSX.
> > > >
> > > > > Busbey-MBA:1.9.0-RC4 busbey$ tar -C src_untar -xzf
> > > >
> > dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/avro-src-1.9.0.tar.gz
> > > > > tar: copyfile unpack
> > > >
> > >
> >
> (avro-src-1.9.0/lang/java/mapred/

Re: [VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Thank you all,

I'm happy to announce, the vote for Avro 1.9.0 has passed.

Binding:
+1 Daniel Kulp
+1 Sean Busbey
+1 Fokko Driesprong

Non-binding:
+1 Lunjie Jin
+1 Michael A. Smith
+1 Ismaël Mejía
+1 Brian Lachniet

I would like to thank the community from participating in the release. Both
in providing patches for Apache Avro and giving the RC's a try.

Yesterday I did some additional testing against Apache Spark (similar to
Nandor :-) and Apache Flink and did not find any blocking issues. There are
some incompatibilities, but most of them are; revoking Jackson from the
public API and deprecating Joda.

I'll publish the artifacts to the different repositories soon. On this, I
consider myself a polyglot programmer, but my knowledge is limited on .Net,
Perl, and Javascript, so I might ask some help from the community here as
well.

Thanks all, and keep up the good work.

Cheers, Fokko Driesprong

Op di 14 mei 2019 om 14:48 schreef Nandor Kollar
:

> +1 (non-binding)
>
> * verified signatures and checksums
> * ran unit tests, passed
> * did a quick sanity test against Spark master: upgraded Spark Avro
> dependency to this release candidate, ran spark-avro tests, all passed (for
> the record, two minor Maven changes are required due to removal of Hadoop 1
> support and making org.tukaani.xz provided dependency - looks like Spark
> tests against each codec). spark-hive tests failed, because deprecated
> methods were removed in Avro (related to Jackson removal from public API,
> and Hive uses them), but I think it is expected, upgrading to a major
> release could have breaking changes.
>
> Nandor
>
> On Sun, May 12, 2019 at 7:11 AM Sean Busbey  wrote:
>
> > +1 (binding)
> >
> > -- Good:
> > * signatures
> > * checksums
> > * tag/ref lines up with src artifact (except for one file mentioned
> below)
> > * LICENSE/NOTICE spot check
> > * apache rat says the license files for all the artifacts are fine
> (except
> > it couldn't understand the gem; using the gem cli confirms it's also
> > correct)
> >
> > -- We Can Be Better Later:
> > * filed AVRO-2395 because the "java" convenience binary part of the dist
> > section is redundant
> > * in the future please post the specific staged maven repository instead
> > of somewhere within the staged repository group. that'll help us avoid
> > possible conflicts should someone forget to cancel a staged RC or
> > accidentally stage an additional repository after the vote is called.
> FWIW
> > as far as I can tell from the Nexus UI, this is the staged repo for this
> > VOTE (and it's what I verified):
> >
> > https://repository.apache.org/content/repositories/orgapacheavro-1020/
> >
> > * When unpacking the source tarball I got this warning on OSX.
> >
> > > Busbey-MBA:1.9.0-RC4 busbey$ tar -C src_untar -xzf
> > dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/avro-src-1.9.0.tar.gz
> > > tar: copyfile unpack
> >
> (avro-src-1.9.0/lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc)
> > failed: No such file or directory
> >
> > I don't think it should be a blocker because it's shown up in prior RCs
> > since 2012 and AFAICT things are fine despite it, save a unit test.
> >
> > * An upgrade guide will help a bunch of folks given the time since last
> > release and this being a major version.
> >
> > For example, someone asked about how incompatible things are. So I ran
> the
> > Java API Compliance Checker on the java libraries, since it's
> (relatively)
> > easy. After filtering out the "avro.shaded" package and excluding the
> > hadoop1 specific jars from 1.8.2:
> >
> > > Busbey-MBA:1.9.0-RC4 busbey$ japi-compliance-checker -l "apache avro"
> > -d1 avro-1.8.2-jacc.xml -d2 avro-1.9.0-jacc.xml -skip-packages
> > skip-packages.txt
> > > Preparing, please wait ...
> > > Using Java 1.8.0_161
> > > Reading classes 1.8.2 ...
> > > WARNING: skipping "internal" packages
> > > NOTE: use --keep-internal option to check them
> > > Reading classes 1.9.0-rc4 ...
> > > WARNING: skipping "internal" packages
> > > Comparing classes ...
> > > Creating compatibility report ...
> > > Binary compatibility: 94.6%
> > > Source compatibility: 93.6%
> > > Total binary compatibility problems: 155, warnings: 26
> > > Total source compatibility problems: 160, warnings: 4
> > > Report: compat_reports/apache
> avro/1.8.2_to_1.9.0-rc4/compat_report.html
> >
> > The report is here:
> >
> >
> >
&g

Re: Python 3 support

[Driesprong, Fokko]

Michael, after some careful thoughts I think it is too late to remove 3.4
support in Avro 1.9 since we're passed RC3. I've updated the tickets to
target this for 1.10.

Cheers, Fokko

Op ma 6 mei 2019 om 10:32 schreef Driesprong, Fokko :

> +1 from my side
>
> Thanks Michael for reviving this thread again. Since 3.4 isn't officially
> supported anymore, I think it is save to drop support for Avro 1.9 as well
> to speed up the deprecation process. In practice it will still work against
> 3.4 until we start using breaking features such as type annotation,
> f-strings etc.
>
> Cheers, Fokko
>
> Op za 4 mei 2019 om 20:47 schreef Rajkumar Natarajan  >
>
>> +1
>>
>> On Sat, May 4, 2019, 2:37 PM sajuukk  wrote:
>>
>> > +1
>> >
>> > W dniu 04.05.2019 o 16:10, Michael A. Smith pisze:
>> > > Hi, Avro devs, it's time to wake this thread up for a checkpoint.
>> > >
>> > > I glanced at pypi stats today and it looks like 3.4 downloads are
>> usually
>> > > below 1000. More importantly, Python has already EOL'ed python 3.4.
>> Do we
>> > > want to consider python 3.4 support officially dropped as of the 1.9
>> > > release?
>> > >
>> > > That is, even if python 3.4 happens to work in 1.9 when it is
>> released,
>> > > only versions 3.5 and up will be officially supported?
>> > >
>> > > I don't believe this is terribly controversial, so I'll ask for a vote
>> > now.
>> > >
>> > > Please reply +1 if you believe that avro-python3 as of 1.9 considers
>> 3.4
>> > > deprecated.
>> > > Please reply -1 if you believe that in avro-python3 1.9 we should
>> > continue
>> > > to make efforts to work in python 3.4.
>> > > Please reply 0 if you want to make comments, but not vote.
>> > >
>> > >
>> https://www.apache.org/foundation/voting.html#votes-on-code-modification
>> > >
>> > >
>> > > On Sat, Nov 10, 2018 at 7:42 PM Piotr Gołąb 
>> wrote:
>> > >
>> > >> Hi Michael,
>> > >>
>> > >> Ok, I think it's good idea to drop the Python 3.4 support after the
>> last
>> > >> release of 3.4. We can show deprecation warnings before. By the way,
>> > this
>> > >> will also be a good opportunity for some refactoring. As for the
>> Avro on
>> > >> Python 2.7, my approach would be just leave it as it is now and don't
>> > >> introduce new features.
>> > >>
>> > >> Best,
>> > >> Piotr
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Sent from:
>> > >> http://apache-avro.679487.n3.nabble.com/Avro-Developers-f679485.html
>> > >>
>> >
>>
>

[VOTE] Release Apache Avro 1.9.0 RC4

[Driesprong, Fokko]

Hi everyone,

Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.

The commit id is 3c76495e9524ef322726d03d7ee406be89e8fde0
* This corresponds to the tag: release-1.9.0-rc4
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc4

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/

This release includes 272 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/1294
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* Remove Jackson classes from public API
* Apache Avro is built by default with Java 8
* Apache Avro is compiled and tested with Java 11 to guarantee compatibility
* Apache Avro MapReduce is compiled and tested with Hadoop 3
* Apache Avro is now leaner, multiple dependencies were removed: guava,
paranamer, commons-codec, and commons-logging
* Introduce JMH Performance Testing Framework
* Add Snappy support for C++ DataFile
* and many, many more!

Since RC1, two commits have been added:
* https://jira.apache.org/jira/browse/AVRO-2381
* https://jira.apache.org/jira/browse/AVRO-2383

Since RC2: The SHA1/MD5 checksums have been replaced with SHA512

Since RC3:
* Regression failure, the customEncode methods are public again.
* The release tarball does not contain snapshot anymore

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 11th of May 2019.

[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet

Cheers, Fokko Driesprong

Re: Python 3 support

[Driesprong, Fokko]

+1 from my side

Thanks Michael for reviving this thread again. Since 3.4 isn't officially
supported anymore, I think it is save to drop support for Avro 1.9 as well
to speed up the deprecation process. In practice it will still work against
3.4 until we start using breaking features such as type annotation,
f-strings etc.

Cheers, Fokko

Op za 4 mei 2019 om 20:47 schreef Rajkumar Natarajan 

> +1
>
> On Sat, May 4, 2019, 2:37 PM sajuukk  wrote:
>
> > +1
> >
> > W dniu 04.05.2019 o 16:10, Michael A. Smith pisze:
> > > Hi, Avro devs, it's time to wake this thread up for a checkpoint.
> > >
> > > I glanced at pypi stats today and it looks like 3.4 downloads are
> usually
> > > below 1000. More importantly, Python has already EOL'ed python 3.4. Do
> we
> > > want to consider python 3.4 support officially dropped as of the 1.9
> > > release?
> > >
> > > That is, even if python 3.4 happens to work in 1.9 when it is released,
> > > only versions 3.5 and up will be officially supported?
> > >
> > > I don't believe this is terribly controversial, so I'll ask for a vote
> > now.
> > >
> > > Please reply +1 if you believe that avro-python3 as of 1.9 considers
> 3.4
> > > deprecated.
> > > Please reply -1 if you believe that in avro-python3 1.9 we should
> > continue
> > > to make efforts to work in python 3.4.
> > > Please reply 0 if you want to make comments, but not vote.
> > >
> > >
> https://www.apache.org/foundation/voting.html#votes-on-code-modification
> > >
> > >
> > > On Sat, Nov 10, 2018 at 7:42 PM Piotr Gołąb  wrote:
> > >
> > >> Hi Michael,
> > >>
> > >> Ok, I think it's good idea to drop the Python 3.4 support after the
> last
> > >> release of 3.4. We can show deprecation warnings before. By the way,
> > this
> > >> will also be a good opportunity for some refactoring. As for the Avro
> on
> > >> Python 2.7, my approach would be just leave it as it is now and don't
> > >> introduce new features.
> > >>
> > >> Best,
> > >> Piotr
> > >>
> > >>
> > >>
> > >> --
> > >> Sent from:
> > >> http://apache-avro.679487.n3.nabble.com/Avro-Developers-f679485.html
> > >>
> >
>

Re: [VOTE] Release Apache Avro 1.9.0 RC3

[Driesprong, Fokko]

 <
> > > katrin.skogl...@avanza.se>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I just managed to test the new RC, specifically the Java code
> > > generation,
> > > > with one of the libraries we use. I then noticed that their
> > > generated java
> > > > code no longer compiles because of a change in access level
> of
> > two
> > > methods.
> > > >
> > > > The generated methods customEncode and customDecode are
> > protected in
> > > this
> > > > version of Avro. They used to be public. This means that the
> > > generated java
> > > > code for schemas using nested records with different
> > namespaces will
> > > no
> > > > longer compile.
> > > >
> > > > I think it would be good to fix this before releasing since
> it
> > is a
> > > real
> > > > easy fix, unless there is a good reason why the access level
> > was
> > > changed to
> > > > protected?
> > > >
> > > > I'll start a PR for this anyway, please let me know if you
> > want the
> > > fix in
> > > > some other way or if I should create a Jira first (new to the
> > process
> > > > here).
> > > >
> > > > //Katrin
> > > >
> > > >
> > > > On 2019-04-30, 12:55, "Driesprong, Fokko"
>  > >
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Since the last release of Apache Avro 1.8.2 on May 31,
> > 2017. Two
> > > years
> > > > later,
> > > > I'm thrilled to propose the following RC to be released
> as
> > > official
> > > > Apache
> > > > Avro 1.9.0 release.
> > > >
> > > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > > * This corresponds to the tag: release-1.9.0-rc3
> > > > *
> > https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > > >
> > > > This release includes 272 Jira issues:
> > > >
> > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > > * Deprecate Joda-Time in favor of Java8 JSR310 and
> setting
> > it as
> > > > default
> > > > * Remove support for Hadoop 1.x
> > > > * Move from Jackson 1.x to 2.9
> > > > * Add ZStandard Codec
> > > > * Lots of updates on the dependencies to fix CVE's
> > > > * and many, many more!
> > > >
> > > > Since RC1, two commits have been added:
> > > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > > * https://jira.apache.org/jira/browse/AVRO-2383
> > > >
> > > > Since RC2 the SHA1/MD5 checksums have been replaced with
> > SHA512
> > > >
> > > > Please download, verify, and test. This vote will remain
> > open
> > > for at
> > > > least
> > > > 72 hours. Given sufficient votes, I would like to close
> it
> > on or
> > > about
> > > > midnight
> > > > on Saturday, 4th of May 2019.
> > > >
> > > > [ ] +1 Release this as Apache Avro 1.9.0
> > > > [ ] +0
> > > > [ ] -1 Do not release this because...
> > > >
> > > > Consider this a +1 (non-binding) from my side:
> > > > * Compiled the new version of Parquet against the Divolte
> > > collector and
> > > > Apache Parquet
> > > >
> > > > Cheers, Fokko Driesprong
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>

Re: [VOTE] Release Apache Avro 1.9.0 RC3

[Driesprong, Fokko]

Thank you Ismaël for looking into it and for the additional interesting
info.

Are you referring to the SNAPSHOT in the tests? I can remove these:
root@83c927afb89b:/avro# grep -R "SNAPSHOT" .
./lang/js/node_modules/uglify-js/tools/domprops.json:
"ORDERED_NODE_SNAPSHOT_TYPE",
./lang/js/node_modules/uglify-js/tools/domprops.json:
"UNORDERED_NODE_SNAPSHOT_TYPE",
./lang/java/archetypes/avro-service-archetype/src/test/integration/projects/basic/archetype.properties:version=0.1-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/idl/pom-jsr310.xml:
1.9.0-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml:
1.9.0-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-jsr310.xml:
  1.9.0-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-joda.xml:
1.9.0-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/schema/pom-jsr310.xml:
1.9.0-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/schema/pom-joda.xml:
1.9.0-SNAPSHOT
./lang/ruby/Rakefile:VERSION =
File.open('../../share/VERSION.txt').read.sub('-SNAPSHOT', '.pre1').chomp
./lang/ruby/.gem/gems/json_pure-2.1.0/data/prototype.js:  null,
XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
./doc/examples/java-example/pom.xml:  1.0-SNAPSHOT

These have slipped the `mvn versions:set`.

Cheers, Fokko

Op di 30 apr. 2019 om 17:24 schreef Ismaël Mejía :

> I did a quick review on the JIRA issues included and extracted some
> extra interesting info worth of addint to the release notes. Up to you
> to choose which matter or not to be added.
>
> * Remove Jackson classes from public API
> * Avro is built by default with Java 8
> * Avro is compiled and tested too with Java 11 to guarantee compatibility
> * Avro MapReduce is also compiled and tested with Hadoop 3
> * Avro is now leaner, multiple dependencies were removed: guava,
> paranamer, commons-codec and commons-logging
> * Introduce JMH Performance Testing Framework
> * Add Snappy support for C++ DataFile
>
> On Tue, Apr 30, 2019 at 4:50 PM Ismaël Mejía  wrote:
> >
> > Sorry, playing the killjoy again.
> >
> > It seems the files (and more critical the poms) still have the -SNAPSHOT
> suffix.
> > Also the comment of Dan Kulp about the extra directory structure in
> > the main file  build/avro-1.9.0 directory would be a nice extra thing
> > to fix.
> >
> > On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko 
> wrote:
> > >
> > > Hi everyone,
> > >
> > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > > later,
> > > I'm thrilled to propose the following RC to be released as official
> Apache
> > > Avro 1.9.0 release.
> > >
> > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > * This corresponds to the tag: release-1.9.0-rc3
> > > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > >
> > > The release tarball, signature, and checksums are here:
> > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > >
> > > Binary artifacts for Java are staged in Nexus here:
> > > *
> > >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > >
> > > This release includes 272 Jira issues:
> > > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> default
> > > * Remove support for Hadoop 1.x
> > > * Move from Jackson 1.x to 2.9
> > > * Add ZStandard Codec
> > > * Lots of updates on the dependencies to fix CVE's
> > > * and many, many more!
> > >
> > > Since RC1, two commits have been added:
> > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > * https://jira.apache.org/jira/browse/AVRO-2383
> > >
> > > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> > >
> > > Please download, verify, and test. This vote will remain open for at
> least
> > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > midnight
> > > on Saturday, 4th of May 2019.
> > >
> > > [ ] +1 Release this as Apache Avro 1.9.0
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Consider this a +1 (non-binding) from my side:
> > > * Compiled the new version of Parquet against the Divolte collector and
> > > Apache Parquet
> > >
> > > Cheers, Fokko Driesprong
>

[VOTE] Release Apache Avro 1.9.0 RC3

[Driesprong, Fokko]

Hi everyone,

Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.

The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
* This corresponds to the tag: release-1.9.0-rc3
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/

This release includes 272 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/1294
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* and many, many more!

Since RC1, two commits have been added:
* https://jira.apache.org/jira/browse/AVRO-2381
* https://jira.apache.org/jira/browse/AVRO-2383

Since RC2 the SHA1/MD5 checksums have been replaced with SHA512

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 4th of May 2019.

[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet

Cheers, Fokko Driesprong

Re: [VOTE] Release Apache Avro 1.9.0 RC2

[Driesprong, Fokko]

Thanks Daniel for updating the build scripts. Canceling the vote due to
SHA1/MD5 checksums.

Cheers, Fokko

Op ma 29 apr. 2019 om 20:14 schreef Driesprong, Fokko :

> Thanks for bringing this up Ismaël. Let me update the checksums.
>
> Cheers, Fokko
>
> Op ma 29 apr. 2019 om 19:49 schreef Daniel Kulp :
>
>> This should be easily fixed by just adding the relevant sha512, no need
>> to completely rebuild unless something else pops up.
>>
>> I’ve updated the Avro build.sh to generate the sha512 instead of the
>> md5/sha1files so if we do need to rebuild, that should be take care of
>> now.
>>
>> Dan
>>
>>
>> > On Apr 29, 2019, at 11:01 AM, Ismaël Mejía  wrote:
>> >
>> > Thanks a lot Fokko for putting this out, I don't know if this is a
>> reason to -1
>> > the vote but we should not supply MD5 or SHA1 checksums, and we should
>> add
>> > SHA-256 and/or SHA-512 following the Apache distribution policy.
>> >
>> > https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>> >
>> > This is a change that happened probably like 1y ago but since it has
>> been long
>> > since Avro's latest release we need to upgrade the scripts to produce
>> the
>> > suggested checksums.
>> >
>> > On Mon, Apr 29, 2019 at 12:57 PM Driesprong, Fokko 
>> wrote:
>> >>
>> >> Hi everyone,
>> >>
>> >> Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
>> >> later,
>> >> I'm thrilled to propose the following RC to be released as official
>> Apache
>> >> Avro 1.9.0 release.
>> >>
>> >> The commit id is 8dbe05a17363a1281482e8611cfead4c04645f47
>> >> * This corresponds to the tag: release-1.9.0-rc2
>> >> * https://github.com/apache/avro/releases/tag/release-1.9.0-rc2/
>> >>
>> >> The release tarball, signature, and checksums are here:
>> >> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc2/
>> >>
>> >> You can find the KEYS file here:
>> >> * https://dist.apache.org/repos/dist/dev/avro/KEYS
>> >>
>> >> Binary artifacts for Java are staged in Nexus here:
>> >> *
>> >>
>> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
>> >>
>> >> This release includes 272 Jira issues:
>> >> https://issues.apache.org/jira/projects/AVRO/versions/1294
>> >> * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
>> default
>> >> * Remove support for Hadoop 1.x
>> >> * Move from Jackson 1.x to 2.9
>> >> * Add ZStandard Codec
>> >> * Lots of updates on the dependencies to fix CVE's
>> >> * and many, many more!
>> >>
>> >> Since RC1, two commits have been added:
>> >> * https://jira.apache.org/jira/browse/AVRO-2381
>> >> * https://jira.apache.org/jira/browse/AVRO-2383
>> >>
>> >> Please download, verify, and test. This vote will remain open for at
>> least
>> >> 72 hours. Given sufficient votes, I would like to close it on or about
>> >> midnight
>> >> on Thursday, 2nd of May 2019.
>> >>
>> >> [ ] +1 Release this as Apache Avro 1.9.0
>> >> [ ] +0
>> >> [ ] -1 Do not release this because...
>> >>
>> >> Consider this a +1 (non-binding) from my side:
>> >> * Compiled the new version of Parquet against the Divolte collector and
>> >> Apache Parquet
>> >>
>> >> Cheers, Fokko Driesprong
>>
>> --
>> Daniel Kulp
>> dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
>> http://dankulp.com/blog>
>> Talend Community Coder - http://talend.com <http://coders.talend.com/>
>>
>

Re: [VOTE] Release Apache Avro 1.9.0 RC2

[Driesprong, Fokko]

Thanks for bringing this up Ismaël. Let me update the checksums.

Cheers, Fokko

Op ma 29 apr. 2019 om 19:49 schreef Daniel Kulp :

> This should be easily fixed by just adding the relevant sha512, no need to
> completely rebuild unless something else pops up.
>
> I’ve updated the Avro build.sh to generate the sha512 instead of the
> md5/sha1files so if we do need to rebuild, that should be take care of
> now.
>
> Dan
>
>
> > On Apr 29, 2019, at 11:01 AM, Ismaël Mejía  wrote:
> >
> > Thanks a lot Fokko for putting this out, I don't know if this is a
> reason to -1
> > the vote but we should not supply MD5 or SHA1 checksums, and we should
> add
> > SHA-256 and/or SHA-512 following the Apache distribution policy.
> >
> > https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> >
> > This is a change that happened probably like 1y ago but since it has
> been long
> > since Avro's latest release we need to upgrade the scripts to produce the
> > suggested checksums.
> >
> > On Mon, Apr 29, 2019 at 12:57 PM Driesprong, Fokko 
> wrote:
> >>
> >> Hi everyone,
> >>
> >> Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> >> later,
> >> I'm thrilled to propose the following RC to be released as official
> Apache
> >> Avro 1.9.0 release.
> >>
> >> The commit id is 8dbe05a17363a1281482e8611cfead4c04645f47
> >> * This corresponds to the tag: release-1.9.0-rc2
> >> * https://github.com/apache/avro/releases/tag/release-1.9.0-rc2/
> >>
> >> The release tarball, signature, and checksums are here:
> >> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc2/
> >>
> >> You can find the KEYS file here:
> >> * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >>
> >> Binary artifacts for Java are staged in Nexus here:
> >> *
> >>
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> >>
> >> This release includes 272 Jira issues:
> >> https://issues.apache.org/jira/projects/AVRO/versions/1294
> >> * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
> >> * Remove support for Hadoop 1.x
> >> * Move from Jackson 1.x to 2.9
> >> * Add ZStandard Codec
> >> * Lots of updates on the dependencies to fix CVE's
> >> * and many, many more!
> >>
> >> Since RC1, two commits have been added:
> >> * https://jira.apache.org/jira/browse/AVRO-2381
> >> * https://jira.apache.org/jira/browse/AVRO-2383
> >>
> >> Please download, verify, and test. This vote will remain open for at
> least
> >> 72 hours. Given sufficient votes, I would like to close it on or about
> >> midnight
> >> on Thursday, 2nd of May 2019.
> >>
> >> [ ] +1 Release this as Apache Avro 1.9.0
> >> [ ] +0
> >> [ ] -1 Do not release this because...
> >>
> >> Consider this a +1 (non-binding) from my side:
> >> * Compiled the new version of Parquet against the Divolte collector and
> >> Apache Parquet
> >>
> >> Cheers, Fokko Driesprong
>
> --
> Daniel Kulp
> dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com <http://coders.talend.com/>
>

[VOTE] Release Apache Avro 1.9.0 RC2

[Driesprong, Fokko]

Hi everyone,

Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.

The commit id is 8dbe05a17363a1281482e8611cfead4c04645f47
* This corresponds to the tag: release-1.9.0-rc2
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc2/

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc2/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/

This release includes 272 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/1294
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* and many, many more!

Since RC1, two commits have been added:
* https://jira.apache.org/jira/browse/AVRO-2381
* https://jira.apache.org/jira/browse/AVRO-2383

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Thursday, 2nd of May 2019.

[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet

Cheers, Fokko Driesprong

Re: [VOTE] Release Apache Avro 1.9.0 RC1

[Driesprong, Fokko]

Ok, I'm canceling the vote for now. Thank you all for testing, and Brian
for uploading it to NuGet.

I'll incorporate the patches from Tim Perkins and Raman Gupta. I'll send
out RC2 soon.

Cheers, Fokko Driesprong

Op za 27 apr. 2019 om 14:08 schreef Brian Lachniet :

> +1
>
> I pushed the v1.9.0-rc1 package
> <https://www.nuget.org/packages/Apache.Avro/1.9.0-rc1> to NuGet for the C#
> side of the house.
>
> On Fri, Apr 26, 2019 at 3:06 PM Daniel Kulp  wrote:
>
> >
> > Two minor issues:
> >
> > 1) It’s strange to extract the src tarball and have it be in a
> > “build/avro-src-1.9.0” directory.  I’d like to see the “build” level
> > removed.   Not a big deal, something we can fix in the future.
> >
> > 2) I’m getting some random test failures in java/mapred when run via
> > “build.sh docker-test", and not sure why.   If I just run maven outside
> of
> > docker it runs fine.   Also not something to block a release, but a bit
> > strange.
> >
> > Anyway, +1 as I don’t see either of them as a blocker.
> >
> > Dan
> >
> >
> >
> >
> > > On Apr 26, 2019, at 5:53 AM, Driesprong, Fokko 
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > > later,
> > > I'm thrilled to propose the following RC to be released as official
> > Apache
> > > Avro 1.9.0 release.
> > >
> > > The commit id is 4607730012293fe1e58957760e8f7b5474abd408
> > > * This corresponds to the tag: release-1.9.0-rc1
> > > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc1
> > >
> > > The release tarball, signature, and checksums are here:
> > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc1/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > >
> > > Binary artifacts for Java are staged in Nexus here:
> > > *
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > >
> > > This release includes 270 Jira issues:
> > > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> default
> > > * Remove support for Hadoop 1.x
> > > * Move from Jackson 1.x to 2.9
> > > * Add ZStandard Codec
> > > * Lots of updates on the dependencies to fix CVE's
> > > * and many, many more!
> > >
> > > Please download, verify, and test. This vote will remain open for at
> > least
> > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > midnight
> > > on Monday, 29 April 2019.
> > >
> > > [ ] +1 Release this as Apache Avro 1.9.0
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Consider this a +1 (non-binding) from my side:
> > > * Compiled the new version of Parquet against the Divolte collector and
> > > Apache Parquet
> > >
> > > Cheers, Fokko Driesprong
> >
> > --
> > Daniel Kulp
> > dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com <http://coders.talend.com/>
> >
>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>

[VOTE] Release Apache Avro 1.9.0 RC1

[Driesprong, Fokko]

Hi everyone,

Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.

The commit id is 4607730012293fe1e58957760e8f7b5474abd408
* This corresponds to the tag: release-1.9.0-rc1
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc1

The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc1/

You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS

Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/

This release includes 270 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/1294
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* and many, many more!

Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Monday, 29 April 2019.

[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...

Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet

Cheers, Fokko Driesprong

Re: Feature freeze Apache Avro 1.9

[Driesprong, Fokko]

Hi Ivan,

Thanks for the input. Personally, I'm not using the UUID at all. Any
comments on it Ryan? Since it isn't introduced in any release, we could
remove it if there is a consensus if it doesn't add any value to the Avro
format.

Apart from that, I did some testing last week on some internal projects,
and it went surprisingly well. I would like to proceed into cutting a first
RC.

Cheers, Fokko

Op di 2 apr. 2019 om 03:49 schreef Ivan Greene :

> Somehow only now after I sent this last message did I realize the
> conversion can already be controlled by adding the LogicalType to
> ReflectData as in TestReflectLogicalTypes.java :) So never mind on that
> second part.
>
> However it's worth making a consideration about whether we are interested
> in revising the UUID logicalType to be 16-byte fixed rather than a 36-byte
> string before it is documented in this release.
>
> -- Ivan
>
> On Mon, Apr 1, 2019 at 8:30 PM Ivan Greene 
> wrote:
>
> > Another outstanding bit for the release in my mind is the state of the
> > UUID logicalType. As Ryan commented here:
> >
> https://issues.apache.org/jira/browse/AVRO-2021?focusedCommentId=16242524=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16242524
> > there is currently an implemented logical type of 'uuid', however it uses
> > the naive toString/fromString rather than a more compact 16-byte fixed. I
> > think we should think about whether we really need this logical type as
> it
> > doesn't provide much benefit as it is. What do you think?
> >
> > Related is UUID support for ReflectData, which is currently nil (we
> cannot
> > even annotate @Stringable; UUID has no String constructor). Basically the
> > only option for using UUIDs in ReflectData is the user implementing their
> > own CustomEncoding. A long unanswered query on another Jira is here:
> >
> https://issues.apache.org/jira/browse/AVRO-1497?focusedCommentId=16283656=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16283656
> > We could either:
> > - Provide CustomEncoding implementation(s), such as UUIDAsStringEncoding
> > and/or UUIDAsFixedEncoding
> > - Add support for LogicalTypes in ReflectData definitions, through a new
> > annotation a la @AvroLogicalType("timestamp-millis"), etc. where we can
> use
> > the uuid logicalType if we decide to keep that in
> >
> > Taking a look at the history it seems as though LogicalTypes and
> > CustomEncodings share some of the same intentions. It's not clear what
> the
> > better option is in this case. CustomEncoding predates LogicalType by a
> few
> > years, but CustomEncoding is more Java-centric.
> >
> > Please let me know what the thoughts are. I should have some time to work
> > on what we decide on this week. Would like to get these items in for 1.9.
> >
> > -- Ivan
> >
>

Re: Feature freeze Apache Avro 1.9

[Driesprong, Fokko]

Thanks all for the feedback.

Regarding the discussion Joda time, I've made a PR to completely remove the
Joda dependency: https://github.com/apache/avro/pull/495

What do you guys think, could we push this for 1.9? This would obviously
break backward compatibility, but then again, people should not use Joda
time anymore.

Cheers, Fokko

Op vr 29 mrt. 2019 om 10:37 schreef Nandor Kollar
:

> I totally support making JSR310 as the default instead of Joda
> time. Should we consider removing the JSR310 from the class names, for
> example rename Jsr310TimeConversions to TimeConversions and the existing
> TimeConversions to JodaTimeConversions, and deprecate the latter?
>
> On Thu, Mar 28, 2019 at 6:53 PM Daniel Kulp  wrote:
>
> > I made few commits today that change the generated code from the Schema
> > compiler a bit….  The changes make Avro 1.9 a bit more incompatible with
> > 1.8, but since they are going to have to migrate anyway, I thought it
> would
> > be important to make the changes now rather then forcing everyone to do
> so
> > in 1.10.   The changes:
> >
> > 1) Default for “time” classes change from Joda to JSR310.  There is a
> flag
> > to specify joda if they need/want it, but I think the “default” should be
> > what we plan on supporting going forward, and I don’t think joda should
> be
> > it. At this point, joda should go away.  :)
> >
> > 2) Private fields  - we were, by default, generating “@Deprecated public”
> > fields.   The idea of generating deprecated code by default annoys me.
> >  Thus, I changed the default to “private”.  Again, there is a setting to
> > make them public (or deprecated/public) if they want it, but for the
> > default, I believe private is what we want.
> >
> > While those will increase the effort to migrate from 1.8 to 1.9, I think
> > it’s better to do that now than waiting any longer.
> >
> > Thoughts?
> > Dan
> >
> >
> > > On Mar 26, 2019, at 9:31 AM, Driesprong, Fokko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to cut the branch for Apache Avro 1.9 release this Friday, and
> > > start moving to a release candidate so we can test. If there are any
> > > features that you would like to get in, please let me know.
> > >
> > > Cheers, Fokko
> >
> > --
> > Daniel Kulp
> > dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com <http://coders.talend.com/>
> >
>

Re: Automatic formatting of Java code

[Driesprong, Fokko]

Thanks Ismaël for taking the effort to get this in. This will add some
additional work now, but it will make it easier in the long run because we
will have less merge conflicts.

Op di 26 mrt. 2019 om 15:57 schreef Ismaël Mejía :

> Hello,
>
> As part of the changes for the upcoming 1.9.0 version. We introduced
> automatic code formatting for the Java codebase by relying on a maven
> plugin called Spotless that we can use to enforce and auto format new
> contributions by relying in an Eclipse Java code formatter [2].
>
> Contributors can configure the formatter in their favorite IDE, but
> the easiest way is to rely in the maven plugin from the command line
> tool. You can find more details in how to use it to prepare your
> PRs/patches in the "How to contribution" webpage [3].
>
> As a consequence of this change being merged today various PRs need to
> be updated / rebased. So if you have something pending or if you are
> working on a new contribution please pay attention to this.
>
> Sorry for the inconvenience and hope you find this new approch to code
> formatting useful.
>
> [1] https://issues.apache.org/jira/browse/AVRO-2353
> [2]
> https://github.com/apache/avro/blob/master/lang/java/eclipse-java-formatter.xml
> [3] https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
>

Feature freeze Apache Avro 1.9

[Driesprong, Fokko]

Hi all,

I'd like to cut the branch for Apache Avro 1.9 release this Friday, and
start moving to a release candidate so we can test. If there are any
features that you would like to get in, please let me know.

Cheers, Fokko

Re: Status of 1.9....

[Driesprong, Fokko]

After fixing AVRO-1810 there is one blocker left:
https://issues.apache.org/jira/browse/AVRO-1979

Thiru, can you give a status update on this one?

I'd like to cut the branch for 1.9 next week if there are no further
objections.

Cheers, Fokko

Op ma 4 feb. 2019 om 17:06 schreef Thiruvalluvan MG
:

>  Hi Fokko,
> Thank you for your effort. I can work with you to the release done.
> Let's plan the tasks and execute them between us.
> Thank you,
> Thiru
> On Monday, 4 February, 2019, 2:20:38 PM IST, Driesprong, Fokko
>  wrote:
>
>  I'll go through the pending tickets today. Is there a PMC which is willing
> to shepherd the release?
>
> I'm happy to help with getting everything ready, and with a
> colllegeau, I'll refactor our Divolte event collector
> <https://divolte.io/> with
> the new version of Avro to do some proper testing and verification of the
> 1.9.
>
> Cheers, Fokko
>
> Op di 22 jan. 2019 om 00:53 schreef Brian Lachniet :
>
> > That resolved my issue. I'm now able to assign issues to myself. Thank
> you!
> >
> > On Mon, Jan 21, 2019 at 3:15 PM Driesprong, Fokko 
> > wrote:
> >
> > > Brian, I recently added you as a contributor to the Jira. Could you try
> > > again to assign the ticket?
> > >
> > > I'm happy to help with the 1.9 release.
> > >
> > > Cheers, Fokko
> > >
> > > Op vr 18 jan. 2019 om 00:10 schreef Brian Lachniet <
> blachn...@gmail.com
> > >:
> > >
> > > > It appears that I don't have permissions to edit or assign issues to
> > > > myself. Is that something I should have permissions for, or should I
> > ask
> > > > someone to assign specific issues to me?
> > > >
> > > > On Thu, Jan 17, 2019, 8:19 AM Daniel Kulp  > > >
> > > > >
> > > > >
> > > > > > On Jan 16, 2019, at 9:29 PM, Thiruvalluvan MG
> > > > 
> > > > > wrote:
> > > > > >
> > > > > > To get a sense of what needs to be done, can you please review
> the
> > > > > pending issues for 1.9.0? According to JIRA, there are 43
> unresolved
> > > > issues
> > > > > with Fix Version 1.9.0. Issue Navigator - ASF JIRA
> > > > > >
> > > > >
> > > > > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > > >
> > > > >
> > > > >
> > > > > > Please add the fix version to the issues outside the list that
> > should
> > > > be
> > > > > resolved for the release and remove the fix version if some of the
> > > issues
> > > > > in the list need not be or cannot be resolved for 1.9.0. Also if
> you
> > > > intend
> > > > > to resolve it in the near future, please assign the issues to
> > yourself.
> > > > > Thank you.
> > > > > > I've done the exercise for C++. There are three C++ issues
> > unresolved
> > > > > and I'll address them before the weekend.
> > > > > > Thank you,
> > > > >
> > > > > Well, the main point is that not all of those 43 issues are going
> to
> > > get
> > > > > done for 1.9.  If someone is not actively working on it to be done
> in
> > > the
> > > > > next week or so, I’m going to remove them from 1.9.  I don’t see
> the
> > > > point
> > > > > in holding up a release that fixes 180 issues waiting for stuff
> that
> > > may
> > > > or
> > > > > may not ever get complete.  Thus, everyone, please take a look at
> > the
> > > > list
> > > > > and if you aren’t working on something that is assigned to you,
> > remove
> > > it
> > > > > from 1.9.
> > > > >
> > > > > That said, I’ll try and take a look at the 17 that have “Patch
> > > > > Available”.  If it’s just a matter of reviewing and applying a
> > patch,
> > > > that
> > > > > shouldn’t be too hard.
> > > > >
> > > > > Dan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > ThiruOn Thursday, 17 January, 2019, 6:51:39 AM IST, Brian
> > > Lachniet
> > > > <
> > > > > blachn...@gmail.com> wrote:
> > > > > >
> > > > > > I'd really like to get the changes to deploy C# NuGet packages
&

Re: Tagging PRs in github

[Driesprong, Fokko]

Nice one Ismaël, looking good!

Op vr 8 feb. 2019 om 10:25 schreef Ismaël Mejía :

> For info, INFRA installed the `probot-autolabeler`github app and I did
> a PR to add the automatica labeling patterns, so now we have automatic
> labeling working for the PRs.
>
> This is the first auto-labeled PR, Yay !
> https://github.com/apache/avro/pull/450
>
> Regards,
> Ismaël
>
> On Wed, Dec 5, 2018 at 6:14 PM Ismaël Mejía  wrote:
> >
> > Good idea Michael, it seems we can automate this via
> > https://github.com/probot/autolabeler
> > I created a ticket in INFRA, let's hope it can be easily setup.
> > https://issues.apache.org/jira/browse/INFRA-17367
> > I will keep you updated when ready.
> >
> > On Tue, Dec 4, 2018 at 10:12 PM Driesprong, Fokko 
> wrote:
> > >
> > > Definitely, thanks a lot Ismaël!
> > >
> > > Op za 1 dec. 2018 om 00:21 schreef Michael A. Smith <
> mich...@smith-li.com>:
> > >
> > > > Thanks for doing that. It was very needed.
> > > >
> > > > Can we automate doing this somehow.
> > > >
> > > > On Thu, Nov 29, 2018 at 17:52 Ismaël Mejía 
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I tagged all open PRs in github using the Label feature and the
> > > > > respective languages as part of the effort to get more reviews
> done.
> > > > > Hopefully people and committers can filter quickly their favorite
> > > > > language and this encourages them to help review.
> > > > >
> > > > > Sorry for taking this initiative without contacting dev@ first
> and for
> > > > > the additional email spam that some of you guys involved in the
> > > > > reviews probably received.
> > > > >
> > > > > I hope other committers agree with this idea so we can keep this
> > > > > tagging in the future (of course as well as the equivalent
> > > > > "Components"  tag in Jira too.
> > > > >
> > > > > Regards,
> > > > > Ismaël
> > > > >
> > > >
>

Re: Status of 1.9....

[Driesprong, Fokko]

I'll go through the pending tickets today. Is there a PMC which is willing
to shepherd the release?

I'm happy to help with getting everything ready, and with a
colllegeau, I'll refactor our Divolte event collector
<https://divolte.io/> with
the new version of Avro to do some proper testing and verification of the
1.9.

Cheers, Fokko

Op di 22 jan. 2019 om 00:53 schreef Brian Lachniet :

> That resolved my issue. I'm now able to assign issues to myself. Thank you!
>
> On Mon, Jan 21, 2019 at 3:15 PM Driesprong, Fokko 
> wrote:
>
> > Brian, I recently added you as a contributor to the Jira. Could you try
> > again to assign the ticket?
> >
> > I'm happy to help with the 1.9 release.
> >
> > Cheers, Fokko
> >
> > Op vr 18 jan. 2019 om 00:10 schreef Brian Lachniet  >:
> >
> > > It appears that I don't have permissions to edit or assign issues to
> > > myself. Is that something I should have permissions for, or should I
> ask
> > > someone to assign specific issues to me?
> > >
> > > On Thu, Jan 17, 2019, 8:19 AM Daniel Kulp  > >
> > > >
> > > >
> > > > > On Jan 16, 2019, at 9:29 PM, Thiruvalluvan MG
> > > 
> > > > wrote:
> > > > >
> > > > > To get a sense of what needs to be done, can you please review the
> > > > pending issues for 1.9.0? According to JIRA, there are 43 unresolved
> > > issues
> > > > with Fix Version 1.9.0. Issue Navigator - ASF JIRA
> > > > >
> > > >
> > > > https://issues.apache.org/jira/projects/AVRO/versions/1294
> > > >
> > > >
> > > >
> > > > > Please add the fix version to the issues outside the list that
> should
> > > be
> > > > resolved for the release and remove the fix version if some of the
> > issues
> > > > in the list need not be or cannot be resolved for 1.9.0. Also if you
> > > intend
> > > > to resolve it in the near future, please assign the issues to
> yourself.
> > > > Thank you.
> > > > > I've done the exercise for C++. There are three C++ issues
> unresolved
> > > > and I'll address them before the weekend.
> > > > > Thank you,
> > > >
> > > > Well, the main point is that not all of those 43 issues are going to
> > get
> > > > done for 1.9.  If someone is not actively working on it to be done in
> > the
> > > > next week or so, I’m going to remove them from 1.9.  I don’t see the
> > > point
> > > > in holding up a release that fixes 180 issues waiting for stuff that
> > may
> > > or
> > > > may not ever get complete.   Thus, everyone, please take a look at
> the
> > > list
> > > > and if you aren’t working on something that is assigned to you,
> remove
> > it
> > > > from 1.9.
> > > >
> > > > That said, I’ll try and take a look at the 17 that have “Patch
> > > > Available”.   If it’s just a matter of reviewing and applying a
> patch,
> > > that
> > > > shouldn’t be too hard.
> > > >
> > > > Dan
> > > >
> > > >
> > > >
> > > >
> > > > > ThiruOn Thursday, 17 January, 2019, 6:51:39 AM IST, Brian
> > Lachniet
> > > <
> > > > blachn...@gmail.com> wrote:
> > > > >
> > > > > I'd really like to get the changes to deploy C# NuGet packages in (
> > > > > https://github.com/apache/avro/pull/428). Other than that, I think
> > > > anything
> > > > > else in the C# realm can wait for later releases.
> > > > >
> > > > > On Wed, Jan 16, 2019, 1:12 PM Daniel Kulp  > > > >
> > > > >> Just a quick question:
> > > > >>
> > > > >> What’s the status of the things people are working for 1.9?I
> > think
> > > > >> we’re getting really close to having something that would be a
> good
> > > 1.9
> > > > >> release.  We’ve accomplished a LOT with well over 100 PR’s merged
> > and
> > > > tons
> > > > >> of JIRA’s closed and such and I’d like to get a release out.  Can
> > > > anything
> > > > >> left be postponed to 1.10 or 1.9.1?
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> --
> > > > >> Daniel Kulp
> > > > >> dk...@apache.org <mailto:dk...@apache.org> -
> > http://dankulp.com/blog
> > > <
> > > > >> http://dankulp.com/blog>
> > > > >> Talend Community Coder - http://talend.com <
> > http://coders.talend.com/
> > > >
> > > >
> > > > --
> > > > Daniel Kulp
> > > > dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog
> <
> > > > http://dankulp.com/blog>
> > > > Talend Community Coder - http://talend.com <
> http://coders.talend.com/>
> > > >
> > >
> >
>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachn...@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>

Re: Status of 1.9....

[Driesprong, Fokko]

Brian, I recently added you as a contributor to the Jira. Could you try
again to assign the ticket?

I'm happy to help with the 1.9 release.

Cheers, Fokko

Op vr 18 jan. 2019 om 00:10 schreef Brian Lachniet :

> It appears that I don't have permissions to edit or assign issues to
> myself. Is that something I should have permissions for, or should I ask
> someone to assign specific issues to me?
>
> On Thu, Jan 17, 2019, 8:19 AM Daniel Kulp 
> >
> >
> > > On Jan 16, 2019, at 9:29 PM, Thiruvalluvan MG
> 
> > wrote:
> > >
> > > To get a sense of what needs to be done, can you please review the
> > pending issues for 1.9.0? According to JIRA, there are 43 unresolved
> issues
> > with Fix Version 1.9.0. Issue Navigator - ASF JIRA
> > >
> >
> > https://issues.apache.org/jira/projects/AVRO/versions/1294
> >
> >
> >
> > > Please add the fix version to the issues outside the list that should
> be
> > resolved for the release and remove the fix version if some of the issues
> > in the list need not be or cannot be resolved for 1.9.0. Also if you
> intend
> > to resolve it in the near future, please assign the issues to yourself.
> > Thank you.
> > > I've done the exercise for C++. There are three C++ issues unresolved
> > and I'll address them before the weekend.
> > > Thank you,
> >
> > Well, the main point is that not all of those 43 issues are going to get
> > done for 1.9.  If someone is not actively working on it to be done in the
> > next week or so, I’m going to remove them from 1.9.  I don’t see the
> point
> > in holding up a release that fixes 180 issues waiting for stuff that may
> or
> > may not ever get complete.   Thus, everyone, please take a look at the
> list
> > and if you aren’t working on something that is assigned to you, remove it
> > from 1.9.
> >
> > That said, I’ll try and take a look at the 17 that have “Patch
> > Available”.   If it’s just a matter of reviewing and applying a patch,
> that
> > shouldn’t be too hard.
> >
> > Dan
> >
> >
> >
> >
> > > ThiruOn Thursday, 17 January, 2019, 6:51:39 AM IST, Brian Lachniet
> <
> > blachn...@gmail.com> wrote:
> > >
> > > I'd really like to get the changes to deploy C# NuGet packages in (
> > > https://github.com/apache/avro/pull/428). Other than that, I think
> > anything
> > > else in the C# realm can wait for later releases.
> > >
> > > On Wed, Jan 16, 2019, 1:12 PM Daniel Kulp  > >
> > >> Just a quick question:
> > >>
> > >> What’s the status of the things people are working for 1.9?I think
> > >> we’re getting really close to having something that would be a good
> 1.9
> > >> release.  We’ve accomplished a LOT with well over 100 PR’s merged and
> > tons
> > >> of JIRA’s closed and such and I’d like to get a release out.  Can
> > anything
> > >> left be postponed to 1.10 or 1.9.1?
> > >>
> > >> Thanks!
> > >>
> > >> --
> > >> Daniel Kulp
> > >> dk...@apache.org  - http://dankulp.com/blog
> <
> > >> http://dankulp.com/blog>
> > >> Talend Community Coder - http://talend.com  >
> >
> > --
> > Daniel Kulp
> > dk...@apache.org  - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com 
> >
>

Re: Modern PHP support

[Driesprong, Fokko]

Hi Gildas,

This would be very welcome. Currently, I'm working on getting the CI to
work with a newer base image . But
the tests only pass against a specific version of PHP 5.6. Let me know if
there is anything that I can do.

You could use git blame
 to
check who worked on the PHP stuff, but most of it has been migrated from
SVN so a lot of history is lost. If you feel confident in taking ownership
of the PHP, please do! :-)

Cheers, Fokko

Op di 18 dec. 2018 om 17:26 schreef Gildas Quéméner <
gildas.queme...@gmail.com>:

> Hello Daniel,
>
> Thanks for your answer.
>
> I will prepare a pull request in the next days (basically more or less with
> the content of https://gitlab.com/Jaumo/avro-php/).
>
> The most important things to do in order to make this package usable by any
> php projects out there is:
>
> 1) To synchronize a separate read-only repository with the content of
> https://github.com/apache/avro/tree/master/lang/php/lib (I'm looking at
> various solution to do so) => required by 3).
> 2) To setup a serious CI/CD pipeline which jobs will be to run the tests
> (against several php environments), check the code consistency and
> synchronize the read only repository.
> 3) To register the php package on packagist.org
>
> I would gladly take responsibility for these actions as I have experience
> in this area and will directly benefit from them.
>
> Problem is that this would require some administrative permission on the
> org/repository and I would probably need a referent at the ASF to know how
> things are done in your organization.
> Is there already someone in charge of the PHP implementation I could
> contact directly?
> If you have some apply process to become an official maintainer as well,
> please let me know.
>
> Cheers.
>
> *Gildas Quéméner*
>
>
> On Tue, Dec 18, 2018 at 8:11 PM Daniel Kulp  wrote:
>
> > We would definitely welcome any php contributions.   If you have any
> > specific fixes or updates or anything, PR’s would be more than welcome.
> >  It would be great if someone with some PHP expertise could take some
> > ownership in collecting various fixes/enhancements from various locations
> > and getting pull requests submitted for review.   That would be awesome.
> >
> > At this point, we won’t be splitting out the various languages to
> separate
> > repositories.   We have discussed it in the past and we MAY do it in the
> > future, but right now we’re trying to concentrate on getting 1.9 out (and
> > then likely a patch or two follow up) and splitting the repo right now
> > would be too disruptive to that process.
> >
> >
> > Dan
> >
> >
> > > On Dec 18, 2018, at 1:26 AM, Gildas Quéméner <
> gildas.queme...@gmail.com>
> > wrote:
> > >
> > > Hello,
> > >
> > > We've been working for a few months on using Avro encoded messages
> > between
> > > our services in my company.
> > >
> > > Our backend heavily relies on PHP, whereas some orbiting services are
> > > written in Java/Kotlin/Python.
> > > At first glance, the lake of official support for PHP looked like a big
> > > no-go for us.
> > >
> > > I know that there is an official library, embedded in the avro library
> at
> > > https://github.com/apache/avro/tree/master/lang/php/lib.
> > >
> > > However this library suffers from blocking issues:
> > > - It is not available through the de facto standard Composer dependency
> > > manager
> > > - It is not tested
> > > - It does not rely on autoloading PSR
> > > - It uses global namespace
> > > - It uses unknown php functions
> > > - It lacks some spec features (support for logicalType attribute for
> > > instance)
> > > - It is buggy
> > >
> > > Despite these issues, latest commit is 1 year old (about a typo), and
> the
> > > one before that is almost 3 years old. Thus, I think it is safe to say
> > that
> > > this library is not maintained anymore.
> > >
> > > Many organizations have realized such issue and have created their own
> > fork
> > > of the library, none of them being thoroughly maintained and having the
> > > same level of bug fixing.
> > >
> > > Here are a few examples:
> > > - https://github.com/wikimedia/avro-php
> > > - https://github.com/flix-tech/avro-php
> > > - https://github.com/researchgate/avro-php
> > > - https://gitlab.com/Jaumo/avro-php (I am the maintainer of this one)
> > >
> > > I have tried to contribute to Wikimedia's fork, but they also seem to
> > have
> > > dropped support (see activity in
> > > https://gerrit.wikimedia.org/r/q/status:open+project:avro-php). The
> > awesome
> > > thing is that they have brought a test suite (that we are reusing in
> our
> > > fork)!
> > >
> > > Flix-tech is doing an awesome job at providing a Schema Registry API
> > client
> > > as well as an Avro Serializer/Deserializer using their Schema Registry
> > > client.
> > >
> > > There are great initiatives out there to modernize this library, but
> none
> > > of 

Re: Overtesting Java Avro?

[Driesprong, Fokko]

Good point Ismaël,

I also noticed it. I don't know why we run the tests twice. The main
question is if the custom_ encoder hits other branches of the code. Maybe
check this using the code coverage? I'm all in for making the tests simpler.

Cheers, Fokko

Op wo 12 dec. 2018 om 11:30 schreef Ismaël Mejía 

> Hello,
>
> While working in some build improvements for Avro I noticed that ALL
> the modules of Avro execute the tests twice, once normally and once
> with the system property
> “org.apache.avro.specific.use_custom_coders=true”. Is this really
> necessary in all modules or if we just do this for avro-core (aka
> lang/java/avro) will we be covered?
>
> I can provide a quick fix PR to do this if the latter makes sense.
>
> Regards,
> Ismaël
>

Re: Tagging PRs in github

[Driesprong, Fokko]

Definitely, thanks a lot Ismaël!

Op za 1 dec. 2018 om 00:21 schreef Michael A. Smith :

> Thanks for doing that. It was very needed.
>
> Can we automate doing this somehow.
>
> On Thu, Nov 29, 2018 at 17:52 Ismaël Mejía  wrote:
>
> > Hello,
> >
> > I tagged all open PRs in github using the Label feature and the
> > respective languages as part of the effort to get more reviews done.
> > Hopefully people and committers can filter quickly their favorite
> > language and this encourages them to help review.
> >
> > Sorry for taking this initiative without contacting dev@ first and for
> > the additional email spam that some of you guys involved in the
> > reviews probably received.
> >
> > I hope other committers agree with this idea so we can keep this
> > tagging in the future (of course as well as the equivalent
> > "Components"  tag in Jira too.
> >
> > Regards,
> > Ismaël
> >
>