[jira] [Created] (HADOOP-18115) EvaluatingStatisticsMap::entrySet may not need parallelstream
Rajesh Balamohan created HADOOP-18115: - Summary: EvaluatingStatisticsMap::entrySet may not need parallelstream Key: HADOOP-18115 URL: https://issues.apache.org/jira/browse/HADOOP-18115 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Rajesh Balamohan Attachments: Screenshot 2022-02-04 at 11.10.39 AM.png When large number of S3AInputStreams are opened, it ends up showing up in profile, as internally parallelstream makes use of fork and join. If it is not mandatory, it can be refactored to get rid of parallelstream. Here is the relevant profile output for ref. !Screenshot 2022-02-04 at 11.10.39 AM.png|width=632,height=429! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18114) Documentation Syntax Error Fix > AWS Assumed Roles
Joey Krabacher created HADOOP-18114: --- Summary: Documentation Syntax Error Fix > AWS Assumed Roles Key: HADOOP-18114 URL: https://issues.apache.org/jira/browse/HADOOP-18114 Project: Hadoop Common Issue Type: Task Components: documentation, fs/s3 Reporter: Joey Krabacher Configuration property "fs.s3a.assumed.role.credentials.provider" was mis-documented without the "f" in front as "s.s3a.assumed.role.credentials.provider" PR already exists and will update the title to reflect this ticket. https://github.com/apache/hadoop/pull/3949 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/ [Jan 28, 2022 6:52:33 AM] (Akira Ajisaka) YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) [Jan 28, 2022 8:05:32 AM] (Akira Ajisaka) HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) [Jan 31, 2022 4:18:36 AM] (noreply) HDFS-16303. Improve handling of datanode lost while decommissioning (#3921) [Jan 31, 2022 4:19:40 AM] (Akira Ajisaka) HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) -1 overall The following subsystems voted -1: asflicense blanks compile golang mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) shadedclient Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.yarn.sls.appmaster.TestAMSimulator compile: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-compile-root.txt [1.4M] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-compile-root.txt [1.4M] golang: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-compile-root.txt [1.4M] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-compile-root.txt [1.4M] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/results-checkstyle-root.txt [14M] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-mvnsite-root.txt [68K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/patch-javadoc-root.txt [0] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-build-tools.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-project.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-annotations.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-project-dist.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-assemblies.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-maven-plugins.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-minikdc.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/41/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-auth.txt [0]
[jira] [Resolved] (HADOOP-18101) Bump aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1
[
https://issues.apache.org/jira/browse/HADOOP-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng resolved HADOOP-18101.
-
Fix Version/s: 3.4.0
Resolution: Fixed
> Bump aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1
> --
>
> Key: HADOOP-18101
> URL: https://issues.apache.org/jira/browse/HADOOP-18101
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Aswin Shakil Balasubramanian
>Assignee: Aswin Shakil Balasubramanian
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The current aliyun-sdk-oss 3.13.0 is affected by
> [CVE-2021-33813|https://github.com/advisories/GHSA-2363-cqg2-863c] due to
> jdom 2.0.6. maven-shade-plugin is also affected by the CVE.
> Bumping aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1 will resolve this issue
> {code:java}
> [INFO] +- org.apache.maven.plugins:maven-shade-plugin:jar:3.2.1:provided
> [INFO] | +-
> org.apache.maven.shared:maven-artifact-transfer:jar:0.10.0:provided
> [INFO] | +- org.jdom:jdom2:jar:2.0.6:provided
> ..
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.1:compile
> [INFO] | +- org.jdom:jdom2:jar:2.0.6:compile
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/770/ [Feb 2, 2022 5:42:17 AM] (noreply) HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/253/ [Feb 1, 2022 8:41:25 AM] (Ayush Saxena) Revert "HADOOP-18024. SocketChannel is not closed when IOException happens in Server$Listener.doAccept (#3719)" [Feb 1, 2022 5:35:59 PM] (noreply) YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) [Feb 1, 2022 5:37:40 PM] (noreply) YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) [Feb 2, 2022 5:42:17 AM] (noreply) HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945) [Feb 3, 2022 2:51:06 AM] (noreply) HDFS-16435. Remove no need TODO comment for ObserverReadProxyProvider (#3912). Contributed by tomscut. - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18113) remove filtering of directory markers in s3a RenameOperation
Steve Loughran created HADOOP-18113: --- Summary: remove filtering of directory markers in s3a RenameOperation Key: HADOOP-18113 URL: https://issues.apache.org/jira/browse/HADOOP-18113 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran S3A RenameOperation tries to filter out renaming intermediate directory markers so as to avoid retaining them when copying from an authoritiative dir to a non-auth one. we did this to support installations where dir marker retention was only enabled for auth dirs * nobody, AFAIK, does this..it is all or nothing * all recent hadoop releases, including earlier branches, is now marker aware * the extras complexity makes debugging rename problems harder. let's just cut it out. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperJournalManager hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperJournalManager hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-mvnsite-root.txt [556K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/562/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K]
[jira] [Created] (HADOOP-18112) Rename operation fails during multi object delete of size more than 1000.
Mukund Thakur created HADOOP-18112: -- Summary: Rename operation fails during multi object delete of size more than 1000. Key: HADOOP-18112 URL: https://issues.apache.org/jira/browse/HADOOP-18112 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.1 Reporter: Mukund Thakur Assignee: Mukund Thakur -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
