Re: Docker build image is currently broken

[Ismaël Mejía]

I don't think we should stop the release because of that since all the
artifacts
are available and still can be tested without docker.

We should definitely for the future try to pin the OS docker image so this
don't
happen again to a stable version and maybe get the exact Java related JDK
and
tools by other means. Agree that we have to work more on reproducibility of
builds.

Re: [VOTE] Release Apache Avro 1.9.2 RC2

[Ismaël Mejía]

+1 (binding)

- Validated signatures for src/doc
- Build Java version for source, all tests passing
- Compared backwards compatibility with 1.9.1
https://gateway.pinata.cloud/ipfs/QmbQqR63ZEKWTvAG6jBQMr1AUBkMdu67LdbUWHMSDZ2GMy
- Validated installed artifacts to upgrade parquet-mr, everything smooth

Great job Ryan!



On Mon, Feb 10, 2020 at 10:23 PM Daniel Kulp  wrote:

> +1
>
> Looks pretty good to me.   Thanks for tackling this!
>
> Dan
>
>
>
> > On Feb 7, 2020, at 5:47 AM, Ryan Skraba  wrote:
> >
> > Hi everyone,
> >
> > I'd like to propose the following RC2 to be released as the official
> Apache
> > Avro 1.9.2 release.
> >
> > The commit id is bf20128ca6138a830b2ea13e0490f3df6b035639
> > * This corresponds to the tag: release-1.9.2-rc2
> > * https://github.com/apache/avro/releases/tag/release-1.9.2-rc2
> >
> > The release tarball, signature, and checksums are here (revision 37932):
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.2-rc2/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.2/
> >
> > This release includes 71 Jira issues (Thanks Ismael for the Fix
> > Version clean-up):
> >
> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.9.2
> >
> > * Some bug fixes:
> >  - C#: AVRO-2606 handle multidimensional arrays of custom types
> >  - Java: AVRO-2592 Avro decimal fails on some conditions
> >  - Java: AVRO-2641 Generated code results in java.lang.ClassCastException
> >  - Java: AVRO-2663 Projection on nested records does not work
> >  - Python: AVRO-2429 unknown logical types should fall back
> > * Improvements:
> >  - Java: AVRO-2247 Improve Java reading performance with a new reader
> >  - Python: AVRO-2104 Schema normalisation and fingerprint support for
> Python 3
> > * Work to unify Python2 and Python3 APIs in preparation for sunset.
> > * Improved tests
> > * Improved, more reliable builds.
> > * Improved readability
> > * Upgrade dependencies to latest versions, including CVE fixes.
> > * And more...
> >
> > Since RC1, five commits have been added, fixing the following JIRA:
> > * https://jira.apache.org/jira/browse/AVRO-2663 (to complete the fix in
> RC1)
> > * https://jira.apache.org/jira/browse/AVRO-2729 Upgrade Velocity to
> version 2.2
> > * https://jira.apache.org/jira/browse/AVRO-2727 Upgrade hadoop3 to
> version 3.2.1
> > * https://jira.apache.org/jira/browse/AVRO-2726 Upgrade Jackson to
> > version 2.10.2
> >
> > Please download, verify, and test. This vote will remain open for at
> least
> > 72 hours. Given sufficient votes, I would like to close after noon UTC
> > Wednesday, February
> > 12th
> >
> > [ ] +1 Release this as Apache Avro 1.9.2
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Best regards, Ryan Skraba
>
> --
> Daniel Kulp
> dk...@apache.org  - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com 
>

[jira] [Commented] (AVRO-2716) Unused local variable

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034520#comment-17034520
 ] 

Hudson commented on AVRO-2716:
--

SUCCESS: Integrated in Jenkins build AvroJava #825 (See 
[https://builds.apache.org/job/AvroJava/825/])
AVRO-2716 Removed unused variable in Resolver (#808) (github: 
[https://github.com/apache/avro/commit/c50a87e1d4a92232b5ec13997549822b51a3c02e])
* (edit) lang/java/avro/src/main/java/org/apache/avro/Resolver.java


> Unused local variable
> -
>
> Key: AVRO-2716
> URL: https://issues.apache.org/jira/browse/AVRO-2716
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.1
>Reporter: Zoltan Farkas
>Priority: Minor
> Fix For: 1.10.0
>
>
> there is a unused local variable at:
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L233



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2716) Unused local variable

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034497#comment-17034497
 ] 

ASF subversion and git services commented on AVRO-2716:
---

Commit c50a87e1d4a92232b5ec13997549822b51a3c02e in avro's branch 
refs/heads/master from Jan Hentschel
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=c50a87e ]

AVRO-2716 Removed unused variable in Resolver (#808)



> Unused local variable
> -
>
> Key: AVRO-2716
> URL: https://issues.apache.org/jira/browse/AVRO-2716
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.1
>Reporter: Zoltan Farkas
>Priority: Minor
> Fix For: 1.10.0
>
>
> there is a unused local variable at:
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L233



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AVRO-2716) Unused local variable

[Fokko Driesprong (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2716.

Fix Version/s: 1.10.0
   Resolution: Fixed

> Unused local variable
> -
>
> Key: AVRO-2716
> URL: https://issues.apache.org/jira/browse/AVRO-2716
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.1
>Reporter: Zoltan Farkas
>Priority: Minor
> Fix For: 1.10.0
>
>
> there is a unused local variable at:
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L233



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2732) Docker image build is broken on master

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034415#comment-17034415
 ] 

Hudson commented on AVRO-2732:
--

SUCCESS: Integrated in Jenkins build AvroJava #824 (See 
[https://builds.apache.org/job/AvroJava/824/])
AVRO-2732: Fix openjdk changes by using stretch base image (#809) (github: 
[https://github.com/apache/avro/commit/ad2a8933d45488e4b0b306d14e1be50a87c33cf3])
* (edit) share/docker/Dockerfile


> Docker image build is broken on master
> --
>
> Key: AVRO-2732
> URL: https://issues.apache.org/jira/browse/AVRO-2732
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: docker
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 1.10.0
>
>
> It seems we had the bad luck that our Dockerfile building/validation image got
> broken because of a change in its upstream parent image openjdk:11 that was 
> upgraded
> to the latest version of its parent version (debian:stable). Debian stable 
> does
> not include Java 8 anymore, so when you build the image (as I did to validate
> the release) it is broken.
> We should fix this by installing Java 'manually' in the docker image. To avoid
> this kind of breakages in the future we should only rely on a pinned OS image
> and install all dev dependencies from there probably install Java.
> There are also some additional warnings that it would be good to fix, e.g. 
> apt-utils and wheel in pip upgrade.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AVRO-2732) Docker image build is broken on master

[Fokko Driesprong (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2732.

Fix Version/s: 1.10.0
   Resolution: Fixed

> Docker image build is broken on master
> --
>
> Key: AVRO-2732
> URL: https://issues.apache.org/jira/browse/AVRO-2732
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: docker
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 1.10.0
>
>
> It seems we had the bad luck that our Dockerfile building/validation image got
> broken because of a change in its upstream parent image openjdk:11 that was 
> upgraded
> to the latest version of its parent version (debian:stable). Debian stable 
> does
> not include Java 8 anymore, so when you build the image (as I did to validate
> the release) it is broken.
> We should fix this by installing Java 'manually' in the docker image. To avoid
> this kind of breakages in the future we should only rely on a pinned OS image
> and install all dev dependencies from there probably install Java.
> There are also some additional warnings that it would be good to fix, e.g. 
> apt-utils and wheel in pip upgrade.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2732) Docker image build is broken on master

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034381#comment-17034381
 ] 

ASF subversion and git services commented on AVRO-2732:
---

Commit ad2a8933d45488e4b0b306d14e1be50a87c33cf3 in avro's branch 
refs/heads/master from RyanSkraba
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=ad2a893 ]

AVRO-2732: Fix openjdk changes by using stretch base image (#809)



> Docker image build is broken on master
> --
>
> Key: AVRO-2732
> URL: https://issues.apache.org/jira/browse/AVRO-2732
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: docker
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> It seems we had the bad luck that our Dockerfile building/validation image got
> broken because of a change in its upstream parent image openjdk:11 that was 
> upgraded
> to the latest version of its parent version (debian:stable). Debian stable 
> does
> not include Java 8 anymore, so when you build the image (as I did to validate
> the release) it is broken.
> We should fix this by installing Java 'manually' in the docker image. To avoid
> this kind of breakages in the future we should only rely on a pinned OS image
> and install all dev dependencies from there probably install Java.
> There are also some additional warnings that it would be good to fix, e.g. 
> apt-utils and wheel in pip upgrade.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Docker build image is currently broken

[Ryan Skraba]

This weekend, openjdk bumped their debian base images from stretch to
buster, bringing in a few package and language tool changes.

Currently all PRs are red on travis.  For local dev using the docker
tools, it will probably break if you pull the new new openjdk:11 image
(or openjdk:8 for branch-1.9)

Ismael raised https://issues.apache.org/jira/browse/AVRO-2732 for Java
8 (thanks!), but it looks like there are also issues with Ruby and
python.

I'd like to see a quick fix to unblock this, and I'm looking into two
strategies: (1) moving our tools/build forward for the newer, more
stabler, more secure tools in the new openjdk image and (2) unbumping
the base image back to more familiar debian-stretch.

Some other points:
* For Java artifacts built on Apache jenkins, the `mvn install` on
https://builds.apache.org/blue/organizations/jenkins/AvroJava/ should
still be OK of course.
* If you build locally and you don't pull the new openjdk base images,
you should be OK.
* If you build locally and don't use the docker tools, that should be fine too.

For the ongoing 1.9.2 RC2, there shouldn't be any changes to the
released artifacts and code, but the docker image on the
release-1.9.2-rc2 tag no longer works (note that this is also true for
the docker image on release-1.9.1 tag).

We can probably have a longer conversation about The Right Thing To Do
for the future with respect to reproducibility, travis, yetus, docker,
tools, etc!

All my best, Ryan