Re: Apache Hadoop 3.5.0 release candidates

2026-02-02 Thread Steve Loughran 
I've updated the bin commands in apache/hadoop-release-support.git to probe
for hadoop azure class and the "hadoop s3guard command"; you could add
another target to check for google gcs class (keeping it out that existing
target so I can do the 3.4 checks on it).

That repo is commit-then-review, so add what you want straight away

On Mon, 2 Feb 2026 at 18:52, Chris Nauroth  wrote:

> Thanks everyone! At this point, I'm able to create successful 3.5.0 RCs for
> both x86 and ARM. We have just a few open blockers remaining: HADOOP-19785,
> HADOOP-19790 and YARN-11918. All of these have patches close to completion.
> I'm thinking the cloud storage jar problem that -1'd the 3.4.3 RC will also
> block 3.5.0.
>
> If you think anything else should be a blocker, please let me know and
> raise it to blocker status in JIRA. If we get all of the blockers closed,
> then I'll start release preparations later this week.
>
> Chris Nauroth
>
>
> On Tue, Jan 27, 2026 at 12:33 PM Steve Loughran 
> wrote:
>
> > +1 for the later build; 20.4 has to recompile native boost and protoc
> > before it can begin, which is a pain too.
> >
> > The openssl issue is interesting.
> >
> > internally we now build a custom wildfly.jar which includes the arm
> native
> > libraries, but build it on RHEL8, not RHEL9, so it still links on older
> > versions. I think the current wildfly jar needs the later rhel/debian
> > releases, and is the only binding to openssl which works there. PITA.
> >
> > ASF releases have to stick with the official wildfly releases.
> >
> >
> > On Tue, 27 Jan 2026 at 19:54, Chris Nauroth  wrote:
> >
> > > Thank you, Cheng. That's a good idea. I marked it as a blocker.
> > >
> > > Chris Nauroth
> > >
> > >
> > > On Tue, Jan 27, 2026 at 1:27 AM Cheng Pan  wrote:
> > >
> > > > Please also consider moving to Ubuntu 24.04 for releasing[1],
> > > > the native libs compiled on Ubuntu 20.04 is not fully compatible
> > > > with new OS, e.g. OpenSSL, this will stop a lot of users trying
> > > > Hadoop by just downloading official pre-built binary tarball instead
> > > > of building by themselves.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HADOOP-19774
> > > >
> > > > Thanks,
> > > > Cheng Pan
> > > >
> > > > On 2026/01/26 23:23:05 Chris Nauroth wrote:
> > > > > Hello everyone,
> > > > >
> > > > > I'm happy to report that I've done a successful dry run of
> creating a
> > > > 3.5.0
> > > > > release candidate from the trunk branch, with a few additional
> > patches
> > > > > applied locally. I plan to proceed with the full release process as
> > > soon
> > > > as
> > > > > we resolve the following blockers:
> > > > >
> > > > > HADOOP-19785 :
> > mvn
> > > > site
> > > > > fails in JDK17
> > > > > HADOOP-19790 :
> > > Build
> > > > > Failing on Trunk
> > > > > HADOOP-19794 :
> > > > > create-release
> > > > > version parsing incompatible with Java 17
> > > > >
> > > > > The first of these is the most challenging IMO. There are competing
> > > > > proposals to either 1) switch to some selective
> maven-javadoc-plugin
> > > > > filtering, or 2) explore additional manipulations of internal JDK
> > APIs
> > > to
> > > > > keep our annotation-based visibility mechanism working. I'd
> > appreciate
> > > > any
> > > > > additional contributor help here to unblock the release.
> > > > >
> > > > > The second looks like we need a fairly trivial update to
> > documentation
> > > > and
> > > > > pom.xml files to require a compatible Maven version. I can look
> into
> > > > > helping with this.
> > > > >
> > > > > The third just needs a one-line patch in the release script, for
> > which
> > > > I've
> > > > > already sent a pull request.
> > > > >
> > > > > Additionally, I have never done the ARM release, so I could use a
> > > little
> > > > > help understanding that. From what I can tell, it's a matter of
> > > following
> > > > > the main release wiki...
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
> > > > >
> > > > > ...and then following instructions from the hadoop-release-support
> > repo
> > > > to
> > > > > add ARM binaries:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hadoop-release-support?tab=readme-ov-file#building-arm64-binaries
> > > > >
> > > > > Does that sound right at a high level? Let me know if I'm missing
> > > > something.
> > > > >
> > > > > We're getting close!
> > > > >
> > > > > Chris Nauroth
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > > >
> > >
> >
>

Re: Apache Hadoop 3.5.0 release candidates

2026-02-02 Thread Chris Nauroth 
Thanks everyone! At this point, I'm able to create successful 3.5.0 RCs for
both x86 and ARM. We have just a few open blockers remaining: HADOOP-19785,
HADOOP-19790 and YARN-11918. All of these have patches close to completion.
I'm thinking the cloud storage jar problem that -1'd the 3.4.3 RC will also
block 3.5.0.

If you think anything else should be a blocker, please let me know and
raise it to blocker status in JIRA. If we get all of the blockers closed,
then I'll start release preparations later this week.

Chris Nauroth


On Tue, Jan 27, 2026 at 12:33 PM Steve Loughran  wrote:

> +1 for the later build; 20.4 has to recompile native boost and protoc
> before it can begin, which is a pain too.
>
> The openssl issue is interesting.
>
> internally we now build a custom wildfly.jar which includes the arm native
> libraries, but build it on RHEL8, not RHEL9, so it still links on older
> versions. I think the current wildfly jar needs the later rhel/debian
> releases, and is the only binding to openssl which works there. PITA.
>
> ASF releases have to stick with the official wildfly releases.
>
>
> On Tue, 27 Jan 2026 at 19:54, Chris Nauroth  wrote:
>
> > Thank you, Cheng. That's a good idea. I marked it as a blocker.
> >
> > Chris Nauroth
> >
> >
> > On Tue, Jan 27, 2026 at 1:27 AM Cheng Pan  wrote:
> >
> > > Please also consider moving to Ubuntu 24.04 for releasing[1],
> > > the native libs compiled on Ubuntu 20.04 is not fully compatible
> > > with new OS, e.g. OpenSSL, this will stop a lot of users trying
> > > Hadoop by just downloading official pre-built binary tarball instead
> > > of building by themselves.
> > >
> > > [1] https://issues.apache.org/jira/browse/HADOOP-19774
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > > On 2026/01/26 23:23:05 Chris Nauroth wrote:
> > > > Hello everyone,
> > > >
> > > > I'm happy to report that I've done a successful dry run of creating a
> > > 3.5.0
> > > > release candidate from the trunk branch, with a few additional
> patches
> > > > applied locally. I plan to proceed with the full release process as
> > soon
> > > as
> > > > we resolve the following blockers:
> > > >
> > > > HADOOP-19785 :
> mvn
> > > site
> > > > fails in JDK17
> > > > HADOOP-19790 :
> > Build
> > > > Failing on Trunk
> > > > HADOOP-19794 :
> > > > create-release
> > > > version parsing incompatible with Java 17
> > > >
> > > > The first of these is the most challenging IMO. There are competing
> > > > proposals to either 1) switch to some selective maven-javadoc-plugin
> > > > filtering, or 2) explore additional manipulations of internal JDK
> APIs
> > to
> > > > keep our annotation-based visibility mechanism working. I'd
> appreciate
> > > any
> > > > additional contributor help here to unblock the release.
> > > >
> > > > The second looks like we need a fairly trivial update to
> documentation
> > > and
> > > > pom.xml files to require a compatible Maven version. I can look into
> > > > helping with this.
> > > >
> > > > The third just needs a one-line patch in the release script, for
> which
> > > I've
> > > > already sent a pull request.
> > > >
> > > > Additionally, I have never done the ARM release, so I could use a
> > little
> > > > help understanding that. From what I can tell, it's a matter of
> > following
> > > > the main release wiki...
> > > >
> > > > https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
> > > >
> > > > ...and then following instructions from the hadoop-release-support
> repo
> > > to
> > > > add ARM binaries:
> > > >
> > > >
> > >
> >
> https://github.com/apache/hadoop-release-support?tab=readme-ov-file#building-arm64-binaries
> > > >
> > > > Does that sound right at a high level? Let me know if I'm missing
> > > something.
> > > >
> > > > We're getting close!
> > > >
> > > > Chris Nauroth
> > > >
> > >
> > >
> > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> > >
> >
>

Re: Apache Hadoop 3.5.0 release candidates

2026-01-27 Thread Steve Loughran 
On Mon, 26 Jan 2026 at 23:23, Chris Nauroth  wrote:

> Hello everyone,
>
> I'm happy to report that I've done a successful dry run of creating a 3.5.0
> release candidate from the trunk branch, with a few additional patches
> applied locally. I plan to proceed with the full release process as soon as
> we resolve the following blockers:
>

Awesome!


>
> HADOOP-19785 : mvn
> site
> fails in JDK17
> HADOOP-19790 : Build
> Failing on Trunk
> HADOOP-19794 :
> create-release
> version parsing incompatible with Java 17
>
> The first of these is the most challenging IMO. There are competing
> proposals to either 1) switch to some selective maven-javadoc-plugin
> filtering, or 2) explore additional manipulations of internal JDK APIs to
> keep our annotation-based visibility mechanism working. I'd appreciate any
> additional contributor help here to unblock the release.
>

no idea there I"m afraid.

>
> The second looks like we need a fairly trivial update to documentation and
> pom.xml files to require a compatible Maven version. I can look into
> helping with this.
>
> The third just needs a one-line patch in the release script, for which I've
> already sent a pull request.
>
> Additionally, I have never done the ARM release, so I could use a little
> help understanding that. From what I can tell, it's a matter of following
> the main release wiki...
>
> https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
>
> ...and then following instructions from the hadoop-release-support repo to
> add ARM binaries:
>
>
> https://github.com/apache/hadoop-release-support?tab=readme-ov-file#building-arm64-binaries
>
> Does that sound right at a high level? Let me know if I'm missing
> something.
>

I've just pushed out an update to that repo, the ant targets to scp down
the binary tar from an arm system, after which
the copy with rename and sign is similar to  a local build.

Only do a -asfrelease or -deploy with one of the builds; because of all the
network pain I did it with the arm one this time. Which means that the jars
in the mvn repo will match that release, not the x86. Welcome to the future.

Email me directly if you want some more assistance there.

Apache Hadoop 3.5.0 release candidates

2026-01-26 Thread Chris Nauroth 
Hello everyone,

I'm happy to report that I've done a successful dry run of creating a 3.5.0
release candidate from the trunk branch, with a few additional patches
applied locally. I plan to proceed with the full release process as soon as
we resolve the following blockers:

HADOOP-19785 : mvn site
fails in JDK17
HADOOP-19790 : Build
Failing on Trunk
HADOOP-19794 :
create-release
version parsing incompatible with Java 17

The first of these is the most challenging IMO. There are competing
proposals to either 1) switch to some selective maven-javadoc-plugin
filtering, or 2) explore additional manipulations of internal JDK APIs to
keep our annotation-based visibility mechanism working. I'd appreciate any
additional contributor help here to unblock the release.

The second looks like we need a fairly trivial update to documentation and
pom.xml files to require a compatible Maven version. I can look into
helping with this.

The third just needs a one-line patch in the release script, for which I've
already sent a pull request.

Additionally, I have never done the ARM release, so I could use a little
help understanding that. From what I can tell, it's a matter of following
the main release wiki...

https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease

...and then following instructions from the hadoop-release-support repo to
add ARM binaries:

https://github.com/apache/hadoop-release-support?tab=readme-ov-file#building-arm64-binaries

Does that sound right at a high level? Let me know if I'm missing something.

We're getting close!

Chris Nauroth