Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1530/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-common-project/hadoop-common Possible null pointer dereference in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:[line 332] spotbugs : module:hadoop-common-project Possible null pointer dereference in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value of called method Dereferenced at ValueQueue.java:[line 332] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-client Redundant nullcheck of sockStreamList, which is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:[line 158] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-httpfs Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] spotbugs : module:hadoop-yarn-project/hadoop-yarn org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 555] spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-rbf Redundant nullcheck of dns, which is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:[line 1093] spotbugs : module:hadoop-hdfs-project Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] Redundant nullcheck of sockStreamList, which is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:is known to be non-null in org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant null check at PeerCache.java:[line 158] Redundant nullcheck of dns, which is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:is known to be non-null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType) Redundant null check at RouterRpcServer.java:[line 1093]
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
Hi folks, The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of 8 affirmative votes, with 6 binding and 2 non-binding. In the order of the voting timeline: binding (+6) Masatake Iwasaki Xiaoqiao He Mukund Madhav Thakur Steve Loughran Takanobu Asanuma Shilun Fan non-binding(+2) Suhail, Ahmar Dongjoon Hyun Thank you very much for helping with the review and validation! We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be officially released in the next 1-2 days. Best Regards, Shilun Fan. On Fri, Mar 15, 2024 at 8:32 AM slfan1989 wrote: > +1 (binding). > > * Verified the signature and checksum of all tarballs. > * Built from the source tarball on macOS 10.14 and OpenJDK 8. > * Execute basic HDFS/YARN commands and run the WordCount MapReduce job. > * Check the Yarn federation page and basic functionalities. > * Read the basic help documentation. > > Best Regards, > Shilun Fan. > > On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma > wrote: > >> +1 (binding). >> >> Thanks for the great work, Shilun Fan. >> >> * Verified signatures and checksums >> * Successfully built from source with native code >> * Deployed a distributed cluster (on K8s) >> * Successfully ran some Erasure Coding operations with ISA-L codec >> * Successfully ran some HDFS RBF operations >> >> Regards, >> - Takanobu Asanuma >> >> 2024年3月14日(木) 15:19 Xiaoqiao He : >> >>> Thanks Ayush for highlighting this information. Absolutely true, we >>> should >>> count RM's vote when explicit +1 here. >>> >>> Best Regards, >>> - He Xiaoqiao >>> >>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena wrote: >>> >>> > > Counter should be with yourself vote, where the current summary >>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline. >>> > >>> > Just on the process: The release manager needs to "explicitly" vote >>> like >>> > any other before counting their own vote, there has been a lot of >>> > discussions around that at multiple places & the official apache doc >>> has >>> > been updated as well [1], the last paragraph reads: >>> > >>> > "Note that there is no implicit +1 from the release manager, or from >>> > anyone in any ASF vote. Only explicit votes are valid. The release >>> manager >>> > is encouraged to vote on releases, like any reviewer would do." >>> > >>> > So, do put an explicit +1, before you count yourself. Good Luck!!! >>> > >>> > -Ayush >>> > >>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes >>> > >>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran >>> >>> > wrote: >>> > >>> >> followup: overnight work happy too. >>> >> >>> >> one interesting pain point is that on a raspberry pi 64 os checknative >>> >> complains that libcrypto is missing >>> >> >>> >> > bin/hadoop checknative >>> >> >>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded & >>> >> initialized native-bzip2 library system-native >>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded & >>> >> initialized native-zlib library >>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L >>> support >>> >> is not available in your platform... using builtin-java codec where >>> >> applicable >>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was >>> built >>> >> without PMDK support. >>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load >>> OpenSSL >>> >> Cipher. >>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so >>> (libcrypto.so: >>> >> cannot open shared object file: No such file or directory)! >>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native >>> Method) >>> >> at >>> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90) >>> >> at >>> >> >>> >> >>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111) >>> >> Native library checking: >>> >> hadoop: true >>> >> >>> >> >>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0 >>> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1 >>> >> zstd : true /lib/aarch64-linux-gnu/libzstd.so.1 >>> >> bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1 >>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open >>> shared >>> >> object file: No such file or directory)! >>> >> ISA-L: false libhadoop was built without ISA-L support >>> >> PMDK:false The native code was built without PMDK support. >>> >> >>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in >>> >> /usr/lib/aarch64-linux-gnu/l >>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto* >>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09 >>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 >>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40 >>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3 >>> >> >>> >> Anyone got any insights on how I should set up this (debian-based) OS >>> >> here? >>> >> I know it's only a small box but with arm64
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.server.namenode.ha.TestPipelinesFailover hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.8M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1333/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K]
