Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
We have a daily build running for 3.3.5: https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/ We have already released it, so I feel we can disable it. Will do it tomorrow, if nobody objects. In case the one who configured it wants to do it early, feel free to do so. We already have one for branch-3.3 which runs weekly which most probably most of us don't follow :) -Ayush On Wed, 22 Mar 2023 at 00:20, Steve Loughran wrote: > > ok, here's my summary, even though most of the binding voters forgot to > declare they were on the PMC. > > +1 binding > > Steve Loughran > Chris Nauroth > Masatake Iwasaki > Ayush Saxena > Xiaoqiao He > > +1 non-binding > > Viraj Jasani > > > 0 or -1 votes: none. > > > Accordingly: the release is good! > > I will send the formal announcement out tomorrow > > A big thank you to everyone who qualified the RC, I know its a lot of work. > We can now get this out and *someone else* can plan the followup. > > > steve > > On Mon, 20 Mar 2023 at 16:01, Chris Nauroth wrote: > > > +1 > > > > Thank you for the release candidate, Steve! > > > > * Verified all checksums. > > * Verified all signatures. > > * Built from source, including native code on Linux. > > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > > -Drequire.zstd -DskipTests > > * Tests passed. > > * mvn --fail-never clean test -Pnative -Dparallel-tests > > -Drequire.snappy -Drequire.zstd -Drequire.openssl > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > * Checked dependency tree to make sure we have all of the expected library > > updates that are mentioned in the release notes. > > * mvn -o dependency:tree > > * Confirmed that hadoop-openstack is now just a stub placeholder artifact > > with no code. > > * For ARM verification: > > * Ran "file " on all native binaries in the ARM tarball to confirm > > they actually came out with ARM as the architecture. > > * Output of hadoop checknative -a on ARM looks good. > > * Ran a MapReduce job with the native bzip2 codec for compression, and > > it worked fine. > > * Ran a MapReduce job with YARN configured to use > > LinuxContainerExecutor and verified launching the containers through > > container-executor worked. > > > > Chris Nauroth > > > > > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena wrote: > > > > > +1(Binding) > > > > > > * Built from source (x86 & ARM) > > > * Successful Native Build (x86 & ARM) > > > * Verified Checksums (x86 & ARM) > > > * Verified Signature (x86 & ARM) > > > * Checked the output of hadoop version (x86 & ARM) > > > * Verified the output of hadoop checknative (x86 & ARM) > > > * Ran some basic HDFS shell commands. > > > * Ran some basic Yarn shell commands. > > > * Played a bit with HDFS Erasure Coding. > > > * Ran TeraGen & TeraSort > > > * Browed through NN, DN, RM & NM UI > > > * Skimmed over the contents of website. > > > * Skimmed over the contents of maven repo. > > > * Selectively ran some HDFS & CloudStore tests > > > > > > Thanx Steve for driving the release. Good Luck!!! > > > > > > -Ayush > > > > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He > > wrote: > > > > > > > > +1 > > > > > > > > * Verified signature and checksum of the source tarball. > > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > > > -DskipTests -Pnative -Pdist -Dtar`. > > > > * Setup pseudo cluster with HDFS and YARN. > > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > > > > result. > > > > * Run example mr applications and check the result - Pi & wordcount. > > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > > etc. > > > > > > > > Thanks Steve for your work. > > > > > > > > Best Regards, > > > > - He Xiaoqiao > > > > > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > > > iwasak...@oss.nttdata.com> > > > >> wrote: > > > >> > > > >> +1 > > > >> > > > >> + verified the signature and checksum of the source tarball. > > > >> > > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK > > 8 > > > >> with native profile enabled. > > > >> + launched pseudo distributed cluster including kms and httpfs with > > > >> Kerberos and SSL enabled. > > > >> + created encryption zone, put and read files via httpfs. > > > >> + ran example MR wordcount over encryption zone. > > > >> + checked the binary of container-executor. > > > >> > > > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky > > > Linux > > > >> 8 (aarch64). > > > >> + ran smoke-tests of hdfs, yarn and mapreduce. > > > >> + built site documentation and skimmed the contents. > > > >> + Javadocs are contained. > > > >> > > > >> Thanks, > > > >> Masatake Iwasaki > > > >> > > > >>> On 2023/03/16 4:47, Steve Loughran wrote: > > > >>> Apache Hadoop 3.3.5 > > > >>> > > > >>> Mukund and I have put together a release candidate (RC3) for Hadoop > > > >> 3.3.5. > > > >>> > > > >>> What we would like is for
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/ [Mar 28, 2023, 11:00:57 AM] (github) HADOOP-18606. ABFS: Add reason in x-ms-client-request-id on a retried API call. (#5299) [Mar 28, 2023, 3:32:01 PM] (github) HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 [Mar 29, 2023, 11:13:10 PM] (github) HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) (#5374) -1 overall The following subsystems voted -1: blanks pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes hadoop.hdfs.server.namenode.ha.TestEditLogTailer hadoop.yarn.client.api.impl.TestAMRMClient hadoop.yarn.sls.TestSLSRunner cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-compile-javac-root.txt [380K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/xml.txt [28K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/results-javadoc-javadoc-root.txt [972K] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2.txt [12K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [544K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [52K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/102/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2.txt [16K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/982/ No changes ERROR: File 'out/email-report.txt' does not exist - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/ [Mar 29, 2023, 5:43:13 PM] (github) HDFS-16959. RBF: State store cache loading metrics (#5497) [Mar 29, 2023, 11:12:02 PM] (github) HADOOP-18548. Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems (#5355) -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:root Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] Failed junit tests : hadoop.mapreduce.v2.TestUberAM hadoop.mapreduce.v2.TestMRJobsWithProfiler hadoop.mapreduce.v2.TestMRJobs cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/artifact/out/results-compile-javac-root.txt [528K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1181/artifact/out/results-checkstyle-root.txt [13M] hadolint:
[jira] [Created] (HADOOP-18685) Amazon S3 changing ACL defaults
Daniel Carl Jones created HADOOP-18685: -- Summary: Amazon S3 changing ACL defaults Key: HADOOP-18685 URL: https://issues.apache.org/jira/browse/HADOOP-18685 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Daniel Carl Jones In April 2023, Amazon S3 will be disabling ACLs by default on all new buckets. [https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/] In S3A, we have tests that rely on ACLs being enabled. We should either update test documentation to tell developers to enable ACLs on buckets, or we should make the ACL tests opt-in if they are used infrequently by S3A users. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-compile-javac-root.txt [508K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-mvnsite-root.txt [588K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [452K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/981/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K]
