Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1526/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-common-project/hadoop-common Possible null pointer dereference in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:[line 332] spotbugs : module:hadoop-common-project Possible null pointer dereference in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:[line 332] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-client Redundant nullcheck of sockStreamList, which is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:[line 158] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-httpfs Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] spotbugs : module:hadoop-yarn-project/hadoop-yarn org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 555] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-rbf Redundant nullcheck of dns, which is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:[line 1093] spotbugs : module:hadoop-hdfs-project Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] Redundant nullcheck of sockStreamList, which is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:[line 158] Redundant nullcheck of dns, which is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:[line
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
followup: overnight work happy too. one interesting pain point is that on a raspberry pi 64 os checknative complains that libcrypto is missing > bin/hadoop checknative 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built without PMDK support. 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher. java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)! at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method) at org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90) at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111) Native library checking: hadoop: true /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0 zlib:true /lib/aarch64-linux-gnu/libz.so.1 zstd : true /lib/aarch64-linux-gnu/libzstd.so.1 bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1 openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)! ISA-L: false libhadoop was built without ISA-L support PMDK:false The native code was built without PMDK support. which happens because its not in /lib/aarch64-linux-gnu but instead in /usr/lib/aarch64-linux-gnu/l ls -l /usr/lib/aarch64-linux-gnu/libcrypto* -rw-r--r-- 1 root root 2739952 Sep 19 13:09 /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 -rw-r--r-- 1 root root 4466856 Oct 27 13:40 /usr/lib/aarch64-linux-gnu/libcrypto.so.3 Anyone got any insights on how I should set up this (debian-based) OS here? I know it's only a small box but with arm64 VMs becoming available in cloud infras, it'd be good to know if they are similar. Note: checknative itself is happy; but checknative -a will fail because of this -though it's an OS setup issue, nothing related to the hadoop binaries. steve On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He wrote: > Hi Shilun, Counter should be with yourself vote, where the current summary > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline. > Thanks again. > > Best Regards, > - He Xiaoqiao > > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 wrote: > > > As of now, we have collected 5 affirmative votes, with 4 votes binding > and > > 1 vote non-binding. > > > > Thank you very much for voting and verifying! > > > > This voting will continue until March 15th, this Friday. > > > > Best Regards, > > Shilun Fan. > > > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran > > > > > wrote: > > > > > +1 binding > > > > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen > it > > > earlier. been testing it this afternoon: > > > > > > pulled the latest version of > > > https://github.com/apache/hadoop-release-support > > > (note, this module is commit-then-review; whoever is working > > on/validating > > > a release can commit as they go along. This is not production code...) > > > > > > * went through the "validating a release" step, validating maven > > artifacts > > > * building the same downstream modules which built for me last time > (avro > > > too complex; hboss not aws v2 in apache yet) > > > > > > spark build is still ongoing, but I'm not going to wait. It is > building, > > > which is key. > > > > > > The core changes I needed in are at the dependency level and I've > > > verified they are good. > > > > > > Oh, and I've also got my raspberry p5 doing the download of the arm > > > stuff for its checknative; not expecting problems. > > > > > > So: i've got some stuff still ongoing, but the core changes to > packaging > > > are in and the rest I'm not worried about -they shouldn't block the > > release > > > as I already validated them on RC2 > > > > > > > > > > > > > > > > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 wrote: > > > > > > > Hi folks, > > > > > > > > Xiaoqiao He and I have put together a release candidate (RC3) for > > Hadoop > > > > 3.4.0. > > > > > > > > What we would like is for anyone who can to verify the tarballs, > > > especially > > > > anyone who can try the arm64 binaries as we want to include them too. > > > > > > > > The RC is available at: > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/ > > > > > > > > The git tag is release-3.4.0-RC3, commit bd8b77f398f > > > > > > > > The maven artifacts are staged at > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1408 > > > > > > > > You can find my public key at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Change log > > >
Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop
Thanx Everyone. I think we have an agreement around dropping the Hbase v1 support. I have created a ticket: https://issues.apache.org/jira/browse/HADOOP-19107 FYI. The HBase v2 build is broken now. I tried " mvn clean install -DskipTests -Dhbase.profile=2.0" & it didn't work (It worked couple of months back IIRC), Some sl4j exclusion needs to be added I believe Maybe an upgrade of the HBase v2 version might solve it, if not we need to handle it in our code. We can chase the upgrade of v2 version as Bryan mentioned either in a different ticket parallely or together with this as well :-) In case anyone has objections/suggestions do shout out here or on the ticket or both -Ayush On Mon, 11 Mar 2024 at 19:22, Steve Loughran wrote: > +1 for cutting hbase 1; it only reduces dependency pain (no more protobuf > 2.5!) > > Created the JIRA on that a few days back > https://issues.apache.org/jira/browse/YARN-11658 > > On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault > wrote: > > > Hbase v1 is EOL for a while now, so option 2 probably makes sense. While > > you are at it you should probably update the hbase2 version, because > 2.2.x > > is also very old and EOL. 2.5.x is the currently maintained release for > > hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as > > well. > > > > On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena wrote: > > > > > Hi Folks, > > > As of now we have two profiles for HBase: one for HBase v1(1.7.1) & > other > > > for v2(2.2.4). The versions are specified over here: [1], how to build > is > > > mentioned over here: [2] > > > > > > As of now we by default run our Jenkins "only" for HBase v1, so we have > > > seen HBase v2 profile silently breaking a couple of times. > > > > > > Considering there are stable versions for HBase v2 as per [3] & HBase > v2 > > > seems not too new, I have some suggestions, we can consider: > > > > > > * Make HBase v2 profile as the default profile & let HBase v1 profile > > stay > > > in our code. > > > * Ditch HBase v1 profile & just lets support HBase v2 profile. > > > * Let everything stay as is, just add a Jenkins job/ Github action > which > > > compiles HBase v2 as well, so we make sure no change breaks it. > > > > > > Personally I would go with the second option, the last HBase v1 release > > > seems to be 2 years back, it might be pulling in some > > > problematic transitive dependencies & it will open scope for us to > > support > > > HBase 3.x when they have a stable release in future. > > > > > > > > > Let me know your thoughts!!! > > > > > > -Ayush > > > > > > > > > [1] > > > > > > > > > https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207 > > > > > > [2] > > > > > > > > > https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172 > > > > > > [3] https://hbase.apache.org/downloads.html > > > > > >
