Re: [VOTE] Release Apache Hadoop 3.2.2 - RC1
Thanks Akira and Sunil for your checks. Will prepare RC2 shortly. Considering that HDFS-15643 has resolved and committed to 3.2.3, I think we could backport to 3.2.2. FYI. Thanks again. Regards, He Xiaoqiao On Fri, Nov 6, 2020 at 10:35 AM Sunil Govindan wrote: > Thanks Akira. I just saw that the revert of HADOOP-17306 is now pushed to > the 3.2.2 branch as well. > > Let's do one more RC. Thanks @Xiaoqiao He for > helping. > > Thanks > Sunil > > On Fri, Nov 6, 2020 at 6:58 AM Akira Ajisaka wrote: > >> -1 >> >> - YARN resource localization is broken by HADOOP-17306 and it has been >> reverted. It should be reverted from 3.2.2 as well. (Thank you Jim >> Brennan for the report!) >> - Would you include HDFS-15643 in RC2? This fixes checksum error in EC >> with ISA-L. >> >> Thank you He Xiaoqiao for preparing the release candidates. >> >> -Akira >> >> On Tue, Nov 3, 2020 at 7:48 PM Xiaoqiao He wrote: >> > >> > Hi folks, >> > >> > I have put together a release candidate (RC1) for Hadoop-3.2.2. >> > >> > It contains *473[1]* fixed jira issues since 3.2.1 release which also >> > include many features and improvements(Please reference the full set of >> > release notes). >> > >> > The RC is available at: >> > http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC1 >> > The RC tag in github is here: >> > https://github.com/apache/hadoop/tree/release-3.2.2-RC1 >> > The maven artifacts are staged at: >> > https://repository.apache.org/content/repositories/orgapachehadoop-1286 >> > >> > You can find my public key at: >> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS or >> > https://people.apache.org/keys/committer/hexiaoqiao.asc directly. >> > >> > Please try the release and vote. The vote will run for 5 days until >> > 2020/11/09 at 00:00 CST. >> > >> > I have done a simple testing with my pseudo cluster. >> > * Verify gpg sign and md5sum. >> > * Setup pseudo cluster with HDFS and YARN. >> > * Run simple Shell/Job. >> > * Check the web UI of >> > NameNode/DFSRouter/DataNode/Resourcemanager/NodeManager. >> > My +1 to start. >> > >> > NOTE: >> > A. Explanation about RC1 rather than RC0. After code freeze for RC0 >> there >> > are several new critical issues merged so create RC1. >> > B. Thanks Sunil/Wei-Chiu/Sammi Chen/Eric Badger for your very helpful >> guide. >> > >> > Thanks >> > He Xiaoqiao >> > >> > [1] >> > >> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948 >> >> - >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> >>
Re: [VOTE] Release Apache Hadoop 3.2.2 - RC1
Thanks Akira. I just saw that the revert of HADOOP-17306 is now pushed to the 3.2.2 branch as well. Let's do one more RC. Thanks @Xiaoqiao He for helping. Thanks Sunil On Fri, Nov 6, 2020 at 6:58 AM Akira Ajisaka wrote: > -1 > > - YARN resource localization is broken by HADOOP-17306 and it has been > reverted. It should be reverted from 3.2.2 as well. (Thank you Jim > Brennan for the report!) > - Would you include HDFS-15643 in RC2? This fixes checksum error in EC > with ISA-L. > > Thank you He Xiaoqiao for preparing the release candidates. > > -Akira > > On Tue, Nov 3, 2020 at 7:48 PM Xiaoqiao He wrote: > > > > Hi folks, > > > > I have put together a release candidate (RC1) for Hadoop-3.2.2. > > > > It contains *473[1]* fixed jira issues since 3.2.1 release which also > > include many features and improvements(Please reference the full set of > > release notes). > > > > The RC is available at: > > http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC1 > > The RC tag in github is here: > > https://github.com/apache/hadoop/tree/release-3.2.2-RC1 > > The maven artifacts are staged at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1286 > > > > You can find my public key at: > > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS or > > https://people.apache.org/keys/committer/hexiaoqiao.asc directly. > > > > Please try the release and vote. The vote will run for 5 days until > > 2020/11/09 at 00:00 CST. > > > > I have done a simple testing with my pseudo cluster. > > * Verify gpg sign and md5sum. > > * Setup pseudo cluster with HDFS and YARN. > > * Run simple Shell/Job. > > * Check the web UI of > > NameNode/DFSRouter/DataNode/Resourcemanager/NodeManager. > > My +1 to start. > > > > NOTE: > > A. Explanation about RC1 rather than RC0. After code freeze for RC0 there > > are several new critical issues merged so create RC1. > > B. Thanks Sunil/Wei-Chiu/Sammi Chen/Eric Badger for your very helpful > guide. > > > > Thanks > > He Xiaoqiao > > > > [1] > > > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948 > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
[jira] [Created] (HADOOP-17360) Log the remote address for authentication success
Ahmed Hussein created HADOOP-17360: -- Summary: Log the remote address for authentication success Key: HADOOP-17360 URL: https://issues.apache.org/jira/browse/HADOOP-17360 Project: Hadoop Common Issue Type: Bug Components: ipc Reporter: Ahmed Hussein Assignee: Ahmed Hussein IPC server logs "Authentication successful for USER". Unlike the hdfs audit log that includes the address with every request, the kms audit log is 10s aggregate per-user op counts. Including the remote address would make debugging much easier. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/49/ [Nov 4, 2020 5:53:46 AM] (Xiaoqiao He) HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. [Nov 4, 2020 5:12:31 PM] (noreply) HADOOP-17341. Upgrade commons-codec to 1.15 (#2428) [Nov 5, 2020 12:18:03 AM] (Wei-Chiu Chuang) Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" -1 overall The following subsystems voted -1: blanks findbugs mvnsite pathlen shadedclient unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml findbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:[line 694] findbugs : module:hadoop-hdfs-project Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory) Redundant null check at DataStorage.java:[line 694] findbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:[line 343] Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:[line 356] Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 333] findbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState) Redundant null check at ResourceLocalizationService.java:is known to be non-null in
[jira] [Reopened] (HADOOP-17306) RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
[
https://issues.apache.org/jira/browse/HADOOP-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan reopened HADOOP-17306:
--
I have reverted this commit from trunk, branch-3.3, branch-3.2, and
branch-3.2.2.
[~vinayakumarb] please address all of the unit test failures when you resubmit.
I also think we need to review references to modified times in the source base
to be sure we are not breaking things with this change. Yarn Resource
Localization is one area, but there may be others. Timestamps are sometimes
stored in state-stores, so there may be compatibility issues with this change
as well.
> RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
>
>
> Key: HADOOP-17306
> URL: https://issues.apache.org/jira/browse/HADOOP-17306
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.
> This api looses milliseconds due to JDK bug.
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]
> This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still
> being used in many productions.
> Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns
> correct time.
> Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as
> workaround.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/ [Nov 4, 2020 3:58:44 AM] (noreply) HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. [Nov 4, 2020 5:53:46 AM] (Xiaoqiao He) HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. [Nov 4, 2020 5:12:31 PM] (noreply) HADOOP-17341. Upgrade commons-codec to 1.15 (#2428) [Nov 5, 2020 12:18:03 AM] (Wei-Chiu Chuang) Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" -1 overall The following subsystems voted -1: pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.security.TestLdapGroupsMapping hadoop.hdfs.TestMultipleNNPortQOP hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.yarn.server.nodemanager.TestNodeManagerResync hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.nodemanager.TestNodeManagerReboot hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch hadoop.yarn.server.nodemanager.TestDeletionService hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked hadoop.fs.azure.TestNativeAzureFileSystemMocked hadoop.fs.azure.TestBlobMetadata hadoop.fs.azure.TestNativeAzureFileSystemConcurrency hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck hadoop.fs.azure.TestNativeAzureFileSystemContractMocked hadoop.fs.azure.TestWasbFsck hadoop.fs.azure.TestOutOfBandAzureBlobOperations cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-compile-javac-root.txt [568K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-patch-pylint.txt [60K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-patch-shelldocs.txt [44K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/whitespace-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/whitespace-tabs.txt [2.0M] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/diff-javadoc-javadoc-root.txt [2.0M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [228K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/316/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [344K]
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/ No changes -1 overall The following subsystems voted -1: asflicense hadolint jshint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-tools/hadoop-azure/src/config/checkstyle.xml hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml Failed junit tests : hadoop.util.TestReadWriteDiskValidator hadoop.crypto.key.TestKeyProviderFactory hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.namenode.TestEditLogRace hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.tools.TestDistCpSystem hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver jshint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-patch-jshint.txt [208K] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-compile-javac-root.txt [456K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-patch-pylint.txt [60K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/whitespace-tabs.txt [1.3M] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/xml.txt [4.0K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/diff-javadoc-javadoc-root.txt [20K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [216K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [284K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/107/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [116K]
[jira] [Created] (HADOOP-17359) [Hadoop-Tools]S3A MultiObjectDeleteException after uploading a file
Xun REN created HADOOP-17359:
Summary: [Hadoop-Tools]S3A MultiObjectDeleteException after
uploading a file
Key: HADOOP-17359
URL: https://issues.apache.org/jira/browse/HADOOP-17359
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.10.0
Reporter: Xun REN
Hello,
I am using org.apache.hadoop.fs.s3a.S3AFileSystem as implementation for S3
related operation.
When I upload a file onto a path, it returns an error:
{code:java}
20/11/05 11:49:13 ERROR s3a.S3AFileSystem: Partial failure of delete, 1
errors20/11/05 11:49:13 ERROR s3a.S3AFileSystem: Partial failure of delete, 1
errorscom.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
objects could not be deleted (Service: null; Status Code: 200; Error Code:
null; Request ID: 767BEC034D0B5B8A; S3 Extended Request ID:
JImfJY9hCl/QvninqT9aO+jrkmyRpRcceAg7t1lO936RfOg7izIom76RtpH+5rLqvmBFRx/++g8=;
Proxy: null), S3 Extended Request ID:
JImfJY9hCl/QvninqT9aO+jrkmyRpRcceAg7t1lO936RfOg7izIom76RtpH+5rLqvmBFRx/++g8= at
com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2287)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1137)
at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1389)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2304)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2270)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$WriteOperationHelper.writeSuccessful(S3AFileSystem.java:2768)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:371)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:74)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:108)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:69) at
org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:128) at
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:488)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:410)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:327) at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:299) at
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:281) at
org.apache.hadoop.fs.shell.Command.processArguments(Command.java:265) at
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at
org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:285)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119) at
org.apache.hadoop.fs.shell.Command.run(Command.java:175) at
org.apache.hadoop.fs.FsShell.run(FsShell.java:317) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at
org.apache.hadoop.fs.FsShell.main(FsShell.java:380)20/11/05 11:49:13 ERROR
s3a.S3AFileSystem: bv/: "AccessDenied" - Access Denied
{code}
The problem is that Hadoop tries to create fake directories to map with S3
prefix and it cleans them after the operation. The cleaning is done from the
parent folder until the root folder.
If we don't give the corresponding permission for some path, it will encounter
this problem:
[https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2296-L2301]
During uploading, I don't see any "fake" directories are created. Why should we
clean them if it is not really created ?
It is the same for the other operations like rename or mkdir where the
"deleteUnnecessaryFakeDirectories" method is called.
Maybe the solution is to check the deleting permission before it calls the
deleteObjects method.
To reproduce the problem:
# With a bucket named my_bucket, we have the path s3://my_bucket/a/b/c inside
# The corresponding user has only permission on the path b and sub-path inside.
# We do the command "hdfs dfs -mkdir s3://my_bucket/a/b/c/d"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
