Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

go ahead and cut it...I'd forgotten about it. thanks, steve On Fri, 31 Mar 2023 at 06:44, Ayush Saxena wrote: > We have a daily build running for 3.3.5: > https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/ > > We have already released it, so I feel we can disable it. Will

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

We have a daily build running for 3.3.5: https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/ We have already released it, so I feel we can disable it. Will do it tomorrow, if nobody objects. In case the one who configured it wants to do it early, feel free to do so. We already

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

ok, here's my summary, even though most of the binding voters forgot to declare they were on the PMC. +1 binding Steve Loughran Chris Nauroth Masatake Iwasaki Ayush Saxena Xiaoqiao He +1 non-binding Viraj Jasani 0 or -1 votes: none. Accordingly: the release is good! I will send the formal

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

+1 Thank you for the release candidate, Steve! * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests passed. * mvn --fail-never

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

+1(Binding) * Built from source (x86 & ARM) * Successful Native Build (x86 & ARM) * Verified Checksums (x86 & ARM) * Verified Signature (x86 & ARM) * Checked the output of hadoop version (x86 & ARM) * Verified the output of hadoop checknative (x86 & ARM) * Ran some basic HDFS shell commands. *

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

+1 * Verified signature and checksum of the source tarball. * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package -DskipTests -Pnative -Pdist -Dtar`. * Setup pseudo cluster with HDFS and YARN. * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the result. * Run

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

+1 + verified the signature and checksum of the source tarball. + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 with native profile enabled. + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. + created encryption zone, put

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

Count me in as well. I am almost done. So, you have 3 potential votes, can be happy now :) Thanx Steve for the efforts!!! -Ayush > On 19-Mar-2023, at 2:46 AM, Chris Nauroth wrote: > > Yes, I'm in progress on verification, so you can expect to get a vote from > me. Thank you, Steve! > >

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

Yes, I'm in progress on verification, so you can expect to get a vote from me. Thank you, Steve! Chris Nauroth On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta wrote: > Hi Steve > > I will also do it by today/tomorrow. > > Thanks, > Ashutosh > > On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, >

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

Hi Steve I will also do it by today/tomorrow. Thanks, Ashutosh On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, wrote: > Thank you for this! > > Can anyone else with time do a review too? i really want to get this one > done, now the HDFS issues are all resolved. > > I do not want this release

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

Thank you for this! Can anyone else with time do a review too? i really want to get this one done, now the HDFS issues are all resolved. I do not want this release to fall by the wayside through lack of votes alone. In fact, I would be very unhappy On Sat, 18 Mar 2023 at 06:47, Viraj Jasani

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

+1 (non-binding) * Signature/Checksum: ok * Rat check (1.8.0_341): ok - mvn clean apache-rat:check * Built from source (1.8.0_341): ok - mvn clean install -DskipTests * Built tar from source (1.8.0_341): ok - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true Containerized

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

and my vote My vote +1 binding I've been using the RCs for a while as my CLI entry point, and testing it through other builds for this RC * Local builds of cloudstore * fs-api-shim * spark * built and ran my cloud integration tests, which now include large CVS file jobs which should show the

[VOTE] Release Apache Hadoop 3.3.5 (RC3)

Apache Hadoop 3.3.5 Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5. What we would like is for anyone who can to verify the tarballs, especially anyone who can try the arm64 binaries as we want to include them too. The RC is available at:

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

thanks. now looking at a critical kerby CVE ( https://github.com/apache/hadoop/pull/5458) and revisited one for netty from last week i am never a fan of last-minute jar updates, but if we don't ship with them we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18 months On

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

> OK. Could you have a go with a (locally built) patch release Just validated the same on the latest HEAD of branch-3.3.5, which includes the two HDFS Jiras I mentioned plus one additional one: * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

i looked at that test and wondered if it it was just being brittle to time. I'm not a fan of those -there's one in abfs which is particularly bad for me- maybe we could see if the test can be cut as it is quite a slow one On Sat, 4 Mar 2023 at 18:28, Viraj Jasani wrote: > A minor update on

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

A minor update on ITestS3AConcurrentOps#testParallelRename I was previously connected to a vpn due to which bandwidth was getting throttled earlier. Ran the test again today without vpn and had no issues (earlier only 40% of the overall putObject were able to get completed within timeout). On

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

On Sat, 4 Mar 2023 at 01:47, Erik Krogen wrote: > Thanks Steve. I see now that the branch cut was way back in October so I > definitely understand your frustration here! > > This made me realize that HDFS-16832 > , which resolves a very > similar

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

Thanks Steve. I see now that the branch cut was way back in October so I definitely understand your frustration here! This made me realize that HDFS-16832 , which resolves a very similar issue as the aforementioned HDFS-16923, is also missing from

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

shipping broken hdfs isn't something we'd want to do, but if we can be confident that all other issues can be addressed in RC3 then I'd be happy. On Fri, 3 Mar 2023 at 05:09, Ayush Saxena wrote: > I will highlight that I am completely fed up with doing this release and >> really want to get it

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

> > I will highlight that I am completely fed up with doing this release and > really want to get it out the way -for which I depend on support from as > many other developers as possible. hmm, I can feel the pain. I tried to find if there is any config or any workaround which can dodge this

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

While this RC is not going to be final, I just wanted to share the results of the testing I have done so far with RC1 and RC2. * Signature: ok * Checksum : ok * Rat check (1.8.0_341): ok - mvn clean apache-rat:check * Built from source (1.8.0_341): ok - mvn clean install -DskipTests * Built

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

well, lets see what others say. we don't want to ship stuff with serious regression to hdfs. I will highlight that I am completely fed up with doing this release and really want to get it out the way -for which I depend on support from as many other developers as possible. Erik, right now what

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

Hi folks, apologies for being late to the release conversation, but I think we need to get HDFS-16923 into 3.3.5. HDFS-16732 , which also went into 3.3.5, introduced an issue whereby Observer

[VOTE] Release Apache Hadoop 3.3.5 (RC2)

Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5. We need anyone who can to verify the source and binary artifacts, including those JARs staged on maven, the site documentation and the arm64 tar file. The RC is available at:

Re: [VOTE] Release Apache Hadoop 3.3.5

> > And i > think we need to change the PR template to mention transitive updates in > the license bit too Not sure if that is gonna help, People might ignore that or check that in overconfidence. No harm though.. BTW Ozone has some cool stuff to handle this, it was added here:

Re: [VOTE] Release Apache Hadoop 3.3.5

need this pr in too, https://github.com/apache/hadoop/pull/5429 1. cuts back on some transitive dependencies from hadoop-aliyun 2. fixes LICENSE-bin to be correct #2 is the blocker...and it looks like 3.2.x will also need fixup as well as the later ones -hadoop binaries have shipped

Re: [VOTE] Release Apache Hadoop 3.3.5

And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent maven releases (3.9.0) on a new local build with maven updated on homebrew (which i needed for spark). so a code change too. That issue doesn't surface on our release dockers, but will hit other people. especially over time.

Re: [VOTE] Release Apache Hadoop 3.3.5

ok, let me cancel, update those jiras and kick off again. that will save anyone else having to do their homework On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma wrote: > I'm now -1 as I found the wrong information on the top page (index.md). > > > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow

Re: [VOTE] Release Apache Hadoop 3.3.5

I'm now -1 as I found the wrong information on the top page (index.md). > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS Router Based Federation. The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before, though it is actually not in branch-3.3. I corrected

Re: [VOTE] Release Apache Hadoop 3.3.5

Thanks for driving the release, Steve and Mukund. I found that there were some jiras with wrong fix versions. The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1: - HDFS-16845 - HADOOP-18345 The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1 (and it is not

[VOTE] Release Apache Hadoop 3.3.5

Apache Hadoop 3.3.5 Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5. What we would like is for anyone who can to verify the tarballs, especially anyone who can try the arm64 binaries as we want to include them too. The RC is available at:

Re: [VOTE] Release Apache Hadoop 3.3.5

thanks pulling in a few of the recent changes which seem needed/important, now wondering about the javadocs. i will add a new probe for this in our automated release ant bulld so we can't cut a release without that https://github.com/steveloughran/validate-hadoop-client-artifacts On Mon, 2 Jan

Re: [VOTE] Release Apache Hadoop 3.3.5

> > > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it > mentioned anywhere in change log or release notes. Are they actually > up-to-date? > > I think it is because of what Ayush said. They are marked as done but not fixed. Also, some were marked with wrong fix versions,

Re: [VOTE] Release Apache Hadoop 3.3.5

I haven't got a chance to deep dive into HADOOP-18324 which is claimed to be the reason for these failures. Most probably will try to check next week if it is still there. >From the PR uploaded on HDFS-16853

Re: [VOTE] Release Apache Hadoop 3.3.5

Is it a problem limited to MiniDFSCluster, or is it a broader problem of RPC client resource cleanup? The patch is changing connection close cleanup, so I assumed the latter. If so, then it could potentially impact applications integrating with the RPC clients. If the problem is limited to

Re: [VOTE] Release Apache Hadoop 3.3.5

> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it > mentioned anywhere in change log or release notes. Are they actually > up-to-date? I don't think there is any issue with the ReleaseNotes generation as such but with the Resolution type of this ticket, It ain't marked as

Re: [VOTE] Release Apache Hadoop 3.3.5

-1, because if I'm understanding the potential impact of HDFS-16853 correctly, then it's serious enough to fix before a release. (I could change my vote if someone wants to make a case that it's not that serious.) Otherwise, this RC was looking good: * Verified all checksums. * Verified all

Re: [VOTE] Release Apache Hadoop 3.3.5

Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it mentioned anywhere in change log or release notes. Are they actually up-to-date? On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki wrote: > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to > > dependency

Re: [VOTE] Release Apache Hadoop 3.3.5

- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency change. For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not expected in hbase-shaded-with-hadoop-check-invariants. Updating hbase-shaded/pom.xml is expected to be the fix as done in

Re: [VOTE] Release Apache Hadoop 3.3.5

Thanks for your great effort for the new release, Steve and Mukund. +1 while it would be nice if we can address missed Javadocs. + verified the signature and checksum. + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile enabled. + launched pseudo distributed

Re: [VOTE] Release Apache Hadoop 3.3.5

Mostly or may be all of those failures are due to HADOOP-18324 , there is a Jira tracking issues with TestLeaseRecovery2 linked to that as well HDFS-16853 -Ayush On Wed, 28 Dec 2022 at 09:13,

Re: [VOTE] Release Apache Hadoop 3.3.5

-0 (non-binding) Output of hadoop-vote.sh: * Signature: ok * Checksum : ok * Rat check (1.8.0_341): ok - mvn clean apache-rat:check * Built from source (1.8.0_341): ok - mvn clean install -DskipTests * Built tar from source (1.8.0_341): ok - mvn clean package -Pdist -DskipTests -Dtar

Re: [VOTE] Release Apache Hadoop 3.3.5

I'm not quite ready to vote yet, pending some additional testing. However, I wanted to give a quick update that ARM support is looking good from my perspective. I focused on verifying the native bits that would need to be different for ARM vs. x64. Here is what I did: * Ran "file " on all native

[VOTE] Release Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5. Given the time of year it's a bit unrealistic to run a 5 day vote and expect people to be able to test it thoroughly enough to make this the one we can ship. What we would like is for anyone who can to verify the