Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

Hi folks,

The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.

In the order of the voting timeline:

binding (+6)

Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan

non-binding(+2)

Suhail, Ahmar
Dongjoon Hyun

Thank you very much for helping with the review and validation!

We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.

Best Regards,
Shilun Fan.

On Fri, Mar 15, 2024 at 8:32 AM slfan1989  wrote:

> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma 
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He :
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>>>
>>> > >  Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> 
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop:  true
>>> >>
>>> >>
>>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>>> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1
>>> >> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
>>> >> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
>>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>>> shared
>>> >> object file: No such file or directory)!
>>> >> ISA-L:   false libhadoop was built without ISA-L support
>>> >> PMDK:false The native code was built without PMDK support.
>>> >>
>>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>>> >> /usr/lib/aarch64-linux-gnu/l
>>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>> >>
>>> >> Anyone got any insights on how I should set up this (debian-based) OS
>>> >> here?
>>> >> I know it's only a small box but with arm64 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

+1 (binding).

* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.

Best Regards,
Shilun Fan.

On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma 
wrote:

> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He :
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>>
>> > >  Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> 
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop:  true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L:   false libhadoop was built without ISA-L support
>> >> PMDK:false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>> >>
>> >> Anyone got any insights on how I should set up this (debian-based) OS
>> >> here?
>> >> I know it's only a small box but with arm64 VMs becoming available in
>> >> cloud
>> >> infras, it'd be good to know if they are similar.
>> >>
>> >> Note: checknative itself is happy; but checknative -a will fail
>> because of
>> >> this -though it's an OS setup issue, nothing related to the hadoop
>> >> binaries.
>> >>
>> >> steve
>> >>
>> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He 
>> wrote:
>> >>
>> >> > Hi Shilun, Counter should be with yourself vote, where the current
>> >> summary
>> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >> > Thanks again.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 
>> wrote:
>> >> >
>> >> > > As of now, we have collected 5 affirmative 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Takanobu Asanuma]

+1 (binding).

Thanks for the great work, Shilun Fan.

* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations

Regards,
- Takanobu Asanuma

2024年3月14日(木) 15:19 Xiaoqiao He :

> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>
> > >  Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran  >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop:  true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L:   false libhadoop was built without ISA-L support
> >> PMDK:false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He 
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >> summary
> >> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >> > Thanks again.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 
> wrote:
> >> >
> >> > > As of now, we have collected 5 affirmative votes, with 4 votes
> binding
> >> > and
> >> > > 1 vote non-binding.
> >> > >
> >> > > Thank you very much for voting and verifying!
> >> > >
> >> > > This voting will continue until March 15th, this Friday.
> >> > >
> >> > > Best Regards,
> >> > > Shilun Fan.
> >> > >
> >> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >> >  >> > > >
> >> > > wrote:
> >> > >
> >> > > > +1 binding
> >> > > >
> >> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> >> seen
> >> > it
> >> > > > earlier. been 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.

Best Regards,
- He Xiaoqiao

On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:

> >  Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran 
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop:  true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib:true /lib/aarch64-linux-gnu/libz.so.1
>> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L:   false libhadoop was built without ISA-L support
>> PMDK:false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > > > > >
>> > > wrote:
>> > >
>> > > > +1 binding
>> > > >
>> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
>> seen
>> > it
>> > > > earlier. been testing it this afternoon:
>> > > >
>> > > > pulled the latest version of
>> > > > https://github.com/apache/hadoop-release-support
>> > > > (note, this module is commit-then-review; whoever is working
>> > > on/validating
>> > > > a release can commit as they go along. This is not production
>> code...)
>> > > >
>> > > > * went through the "validating a release" step, validating maven
>> > > artifacts
>> > > > * building the same downstream modules which built for me last time
>> > (avro
>> > > > too complex; hboss not aws v2 in apache yet)
>> > > >
>> > > > spark build is still ongoing, but I'm not going to wait. It 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

>  Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.

Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:

"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."

So, do put an explicit +1, before you count yourself. Good Luck!!!

-Ayush

[1] https://www.apache.org/foundation/voting.html#ReleaseVotes

On Tue, 12 Mar 2024 at 17:27, Steve Loughran 
wrote:

> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop:  true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib:true /lib/aarch64-linux-gnu/libz.so.1
> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L:   false libhadoop was built without ISA-L support
> PMDK:false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

followup: overnight work happy too.

one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing

> bin/hadoop checknative

2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop:  true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3

Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.

Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.

steve

On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:

> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>  > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.

Best Regards,
- He Xiaoqiao

On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:

> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran  >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.

Thank you very much for voting and verifying!

This voting will continue until March 15th, this Friday.

Best Regards,
Shilun Fan.

On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran 
wrote:

> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

+1 binding

(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:

pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)

* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)

spark build is still ongoing, but I'm not going to wait. It is building,
which is key.

The core changes I needed in are at the dependency level and I've
verified they are good.

Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.

So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2





On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Mukund Madhav Thakur]

+1 (binding)
Thanks for driving the release, Shilun Fan.


   - Verified checksums and signatures.
   - Build the source on macOS and Java 8.
   - Ran AWS and Azure tests ( some known failures, created Jira's
   HADOOP-19106 it is a test side issue only mostly related to test configs,
   also as Ahmar pointed out)
   - Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
   - Compiled gcs-connector and ran tests using the 3.4.0 version.


On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He  wrote:

> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar  wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > .
> >
> > --
> > *From:* slfan1989 
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > priv...@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

+1 (binding)

[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.

Best Regards,
- He Xiaoqiao


On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar  wrote:

> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> .
>
> --
> *From:* slfan1989 
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> priv...@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Suhail, Ahmar]

+1 (non-binding)


Thanks for driving the release, Shilun.


* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains 
Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 
express) via hadoop fs commands.

We have some failures in S3A tests, but these are expected due to one of the 
public landsat buckets we relied on no longer being available, these tests were 
fixed in HADOOP-19057.



From: slfan1989 
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; 

Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi folks,

Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/

The git tag is release-3.4.0-RC3, commit bd8b77f398f

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md

This is off branch-3.4.0 and is the first big release since 3.3.6.

Key changes include

* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes

Differences from Hadoop-3.4.0-RC2

* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.

Best Regards,
Shilun Fan.

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Masatake Iwasaki]

+1 (binding)

Thanks for the great work, Shilun Fan.

+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
  and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
  + ran some example jobs on encryption zone.
  + ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.

- downstream like HBase can not be built against 3.4.0 without addressing
  changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
  SM4 support (e.g. Red Hat) while it falls back to Java implementation.
  I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering 
this)::

$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL 
Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is 
your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)

Masatake Iwasaki

On 2024/03/05 7:07, slfan1989 wrote:

Hi folks,

Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/

The git tag is release-3.4.0-RC3, commit bd8b77f398f

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md

This is off branch-3.4.0 and is the first big release since 3.3.6.

Key changes include

* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes

Differences from Hadoop-3.4.0-RC2

* From branch-3.4 to branch-3.4.0 backport 2 Prs
 * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
 * HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.

Best Regards,
Shilun Fan.



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist

I think it due to HADOOP-17524
, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon

I will abstain

-Ayush

On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He  wrote:

> cc @Chao Sun  @zhang...@apache.org
>  @Ayush Saxena 
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He  wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:
>>
>>> @Xiaoqiao He  @Ayush Saxena 
>>> @Steve
>>> Loughran  @Mukund Madhav Thakur
>>>  @Takanobu
>>> Asanuma  @Masatake Iwasaki 
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

cc @Chao Sun  @zhang...@apache.org 
 @Ayush Saxena 
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.

Best Regards,
- He Xiaoqiao


On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He  wrote:

> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:
>
>> @Xiaoqiao He  @Ayush Saxena 
>> @Steve
>> Loughran  @Mukund Madhav Thakur
>>  @Takanobu
>> Asanuma  @Masatake Iwasaki 
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.

Best Regards,
- He Xiaoqiao

On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:

> @Xiaoqiao He  @Ayush Saxena 
> @Steve
> Loughran  @Mukund Madhav Thakur
>  @Takanobu
> Asanuma  @Masatake Iwasaki 
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

@Xiaoqiao He  @Ayush Saxena  @Steve
Loughran  @Mukund Madhav Thakur
 @Takanobu
Asanuma  @Masatake Iwasaki 

Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!

Best Regards,
Shilun Fan.

On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>