Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1523/

[Mar 8, 2024, 12:48:38 PM] (github) HADOOP-19043. S3A: Regression: 
ITestS3AOpenCost fails on prefetch test runs (#6465)




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen spotbugs xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-common-project/hadoop-common 
   Possible null pointer dereference in 
org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value 
of called method Dereferenced at 
ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due 
to return value of called method Dereferenced at ValueQueue.java:[line 332] 

spotbugs :

   module:hadoop-common-project 
   Possible null pointer dereference in 
org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value 
of called method Dereferenced at 
ValueQueue.java:org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due 
to return value of called method Dereferenced at ValueQueue.java:[line 332] 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs-client 
   Redundant nullcheck of sockStreamList, which is known to be non-null in 
org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant 
null check at PeerCache.java:is known to be non-null in 
org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant 
null check at PeerCache.java:[line 158] 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs-httpfs 
   Redundant nullcheck of xAttrs, which is known to be non-null in 
org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) 
Redundant null check at HttpFSFileSystem.java:is known to be non-null in 
org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) 
Redundant null check at HttpFSFileSystem.java:[line 1373] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may 
return null, but is declared @Nonnull At ServiceScheduler.java:is declared 
@Nonnull At ServiceScheduler.java:[line 555] 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs-rbf 
   Redundant nullcheck of dns, which is known to be non-null in 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)
 Redundant null check at RouterRpcServer.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)
 Redundant null check at RouterRpcServer.java:[line 1093] 

spotbugs :

   module:hadoop-hdfs-project 
   Redundant nullcheck of xAttrs, which is known to be non-null in 
org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) 
Redundant null check at HttpFSFileSystem.java:is known to be non-null in 
org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) 
Redundant null check at HttpFSFileSystem.java:[line 1373] 
   Redundant nullcheck of sockStreamList, which is known to be non-null in 
org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant 
null check at PeerCache.java:is known to be non-null in 
org.apache.hadoop.hdfs.PeerCache.getInternal(DatanodeID, boolean) Redundant 
null check at PeerCache.java:[line 158] 
   Redundant nullcheck of dns, which is known to be non-null in 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getCachedDatanodeReport(HdfsConstants$DatanodeReportType)
 Redundant null check at RouterRpcServer.java:is known to be non-null in 

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/642/

[Mar 7, 2024, 9:39:01 AM] (github) HADOOP-19090. Use protobuf-java 3.23.4. 
(#6593). Contributed by PJ Fanning.
[Mar 8, 2024, 12:48:38 PM] (github) HADOOP-19043. S3A: Regression: 
ITestS3AOpenCost fails on prefetch test runs (#6465)


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Suhail, Ahmar]

+1 (non-binding)


Thanks for driving the release, Shilun.


* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains 
Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 
express) via hadoop fs commands.

We have some failures in S3A tests, but these are expected due to one of the 
public landsat buckets we relied on no longer being available, these tests were 
fixed in HADOOP-19057.



From: slfan1989 
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; 

Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi folks,

Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/

The git tag is release-3.4.0-RC3, commit bd8b77f398f

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md

This is off branch-3.4.0 and is the first big release since 3.3.6.

Key changes include

* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes

Differences from Hadoop-3.4.0-RC2

* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.

Best Regards,
Shilun Fan.

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.TestLeaseRecovery2 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover 
   hadoop.hdfs.TestDFSInotifyEventInputStream 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.TestSLSRunner 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-compile-javac-root.txt
  [488K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-mvnsite-root.txt
  [572K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [220K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.8M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1326/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt
  [16K]