followup: overnight work happy too.
one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing
> bin/hadoop checknative
2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop: true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib/aarch64-linux-gnu/libz.so.1
zstd : true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3
Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.
Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.
steve
On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He <hexiaoq...@apache.org> wrote:
> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989 <slfan1...@apache.org> wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> <ste...@cloudera.com.invalid
> > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989 <slfan1...@apache.org> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > > >
> > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > > >
> > > > Key changes include
> > > >
> > > > * S3A: Upgrade AWS SDK to V2
> > > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > > * YARN Federation improvements
> > > > * YARN Capacity Scheduler improvements
> > > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > > * HDFS EC: Code Enhancements and Bug Fixes
> > > > * Transitive CVE fixes
> > > >
> > > > Differences from Hadoop-3.4.0-RC2
> > > >
> > > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > > (80b4bb68159c)
> > > > * Use hadoop-release-support[1] for packaging and verification.
> > > > * Add protobuf compatibility issue description
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > [1] hadoop-release-support:
> > > > https://github.com/apache/hadoop-release-support
> > > > Thanks to steve for providing hadoop-release-support.
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > > >
> > >
> >
>
