Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1505/ [Feb 18, 2024, 1:15:40 AM] (github) HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang. [Feb 19, 2024, 12:34:59 AM] (github) HDFS-17361. DiskBalancer: Query command support with multiple nodes (#6508) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19082) Update AWS SDK V2 to 2.24.1
Harshit Gupta created HADOOP-19082: -- Summary: Update AWS SDK V2 to 2.24.1 Key: HADOOP-19082 URL: https://issues.apache.org/jira/browse/HADOOP-19082 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.4.0 Reporter: Harshit Gupta Assignee: Harshit Gupta Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-6901) Parsing large compressed files with HADOOP-1722 spawns multiple mappers per file
[ https://issues.apache.org/jira/browse/HADOOP-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rick Weber resolved HADOOP-6901. Resolution: Abandoned Marking as abandoned. Issues is 14 years old and Dumbo usage is no longer and issue/problem. > Parsing large compressed files with HADOOP-1722 spawns multiple mappers per > file > > > Key: HADOOP-6901 > URL: https://issues.apache.org/jira/browse/HADOOP-6901 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.21.0 > Environment: Hadoop v0.20.2 + HADOOP-1722 >Reporter: Rick Weber >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > This was originally discovered while using Dumbo to parse a very large (2G) > compressed file. By default, Dumbo will attempt to use the AutoInputFormat > as the input format. > Here is my use case: > I have a large (2Gb) compressed file. It's compressed using the default > method, which is Gzip based and is unsplittable. Due to the size, the > default implementation of AutoInputFormat says that this file is splittable. > As a result, this file is split into about 35 parts, and each one is assigned > to a Map job. > However, since the file itself is unsplittable, each Map job winds up parsing > the file again from the beginning. This basically means my job has 35x the > data, and takes 35x long to run. > If I set "-inputformat text", this problem does not appear in dumbo. If I > manually call the streaming jar and use AutoInputFormat, this > problem appears. > Looking at the code in AutoInputFormat, it appears that it uses the default > isSplittable() method from InputFormat, which indicates everything is > splittable. I think that this class should define it's own isSplittable > method similar to what is defined in the TextInputFormat class, which > basically says it's splittable if it's not compressed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.hdfs.TestFileCreation hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-compile-javac-root.txt [508K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-mvnsite-root.txt [592K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.8M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K]
