Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/ [Dec 7, 2023, 2:16:23 AM] (github) HDFS-17262. Fixed the verbose log.warn in DFSUtil.addTransferRateMetric(). (#6290). Contributed by Xing Lin. [Dec 7, 2023, 1:08:47 PM] (github) HADOOP-18997. S3A: Add option fs.s3a.s3express.create.session to disable S3Express CreateSession (#6316) [Dec 7, 2023, 1:11:11 PM] (github) YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. -1 overall The following subsystems voted -1: blanks hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-hadolint.txt [24K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/results-javadoc-javadoc-root.txt [244K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1434/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [36K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19008) S3A: Upgrade AWS SDK to 2.21.41
Steve Loughran created HADOOP-19008: --- Summary: S3A: Upgrade AWS SDK to 2.21.41 Key: HADOOP-19008 URL: https://issues.apache.org/jira/browse/HADOOP-19008 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0, 3.3.7-aws Reporter: Steve Loughran Assignee: Steve Loughran sdk 2.21.41 is out and logging now picks up the log4j.properties options. move to this ASAP -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18999) S3A: debug logging for http traffic to S3 stores
[
https://issues.apache.org/jira/browse/HADOOP-18999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18999.
-
Resolution: Not A Problem
no longer needed as 2.21.41 logs properly!
> S3A: debug logging for http traffic to S3 stores
>
>
> Key: HADOOP-18999
> URL: https://issues.apache.org/jira/browse/HADOOP-18999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> AWS SDK bundle.jar logging doesn't set up right.
> {code}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> {code}
> Cloudstore commands have a -debug option to force set this through log4j
> APIs; this does work.
> Proposed:
> * add reflection-based ability to set/query log4j log levels (+tests,
> obviously)
> * add a new log `org.apache.hadoop.fs.s3a.logging.sdk`
> * if set to DEBUG, DefaultS3ClientFactory will enable logging on the aws
> internal/shaded classes
> this allows log4j.properties to turn on logging; reflection ensures all is
> well on other log back-ends and when unshaded aws sdk jars are in use
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19007) S3A: transfer manager not wired up to s3a executor pool
Steve Loughran created HADOOP-19007: --- Summary: S3A: transfer manager not wired up to s3a executor pool Key: HADOOP-19007 URL: https://issues.apache.org/jira/browse/HADOOP-19007 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran S3ClientFactory.createS3TransferManager() doesn't use the executor declared in S3ClientCreationParameters.transferManagerExecutor * method needs to take S3ClientCreationParameters * and set the transfer manager executor -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19006) no valid user IDs when importing KEYS in gpg
Nikita Eshkeev created HADOOP-19006:
---
Summary: no valid user IDs when importing KEYS in gpg
Key: HADOOP-19006
URL: https://issues.apache.org/jira/browse/HADOOP-19006
Project: Hadoop Common
Issue Type: Bug
Reporter: Nikita Eshkeev
h1. What I did
# Download the KEYS file: https://dlcdn.apache.org/hadoop/common/KEYS
# Import the KEYS file:
{{gpg --import KEYS}}
h1. What happened
There is the error while importing the KEYS file:
{{gpg: key B56E5591: no valid user IDs}}
h1. What is expected
No errors when importing the KEYS file
h1. Observation
The import definitely worked on October 18 2023, but now it doesn't
h1. Environment
h2. Operating System
{{$ cat /etc/os-release}}
{{NAME="Amazon Linux"}}
{{VERSION="2"}}
{{ID="amzn"}}
{{ID_LIKE="centos rhel fedora"}}
{{VERSION_ID="2"}}
{{PRETTY_NAME="Amazon Linux 2"}}
{{ANSI_COLOR="0;33"}}
{{CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"}}
h2. GPG Version
{{$ gpg --version}}
{{gpg (GnuPG) 2.0.22}}
{{libgcrypt 1.5.3}}
{{Copyright (C) 2013 Free Software Foundation, Inc.}}
{{License GPLv3+: GNU GPL version 3 or later
}}
{{This is free software: you are free to change and redistribute it.}}
{{There is NO WARRANTY, to the extent permitted by law.}}
{{Home: ~/.gnupg}}
{{Supported algorithms:}}
{{Pubkey: RSA, ?, ?, ELG, DSA}}
{{Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,}}
{{CAMELLIA128, CAMELLIA192, CAMELLIA256}}
{{Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224}}
{{Compression: Uncompressed, ZIP, ZLIB, BZIP2}}
h2. SHA256 checksum of the KEYS file
{{$ sha256sum KEYS}}
{{807ad6cc5e1fdedf17884a8da9389549e72b9ad5b8e6fc5050382619c7a19ec6 KEYS}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18989) Use thread pool to improve the speed of creating control files in TestDFSIO
[ https://issues.apache.org/jira/browse/HADOOP-18989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuyan Zhang resolved HADOOP-18989. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use thread pool to improve the speed of creating control files in TestDFSIO > --- > > Key: HADOOP-18989 > URL: https://issues.apache.org/jira/browse/HADOOP-18989 > Project: Hadoop Common > Issue Type: Improvement > Components: benchmarks, common >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When we use TestDFSIO tool to test the throughouts of HDFS clusters, we found > it is so slow in the creating controll files stage. > After refering to the source code, we found that method createControlFile try > to create control files serially. It can be improved by using thread pool. > After optimizing, the TestDFSIO tool runs quicker than before. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/ [Dec 7, 2023, 2:09:41 AM] (Masatake Iwasaki) Bump bower from 1.7.7 to 1.8.8 (#1683) [Dec 7, 2023, 2:09:43 AM] (Masatake Iwasaki) YARN-10020. Fix build instruction of hadoop-yarn-ui. (#1750) [Dec 7, 2023, 2:37:13 AM] (Masatake Iwasaki) YARN-10037. Upgrade build tools for YARN Web UI v2. [Dec 7, 2023, 5:12:08 AM] (Masatake Iwasaki) YARN-10331. Upgrade node.js to 10.21.0. (#2106) -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.tools.util.TestProducerConsumer hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [456K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1234/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K]
