[jira] [Resolved] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-16430. Resolution: Fixed > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16554) mvn javadoc:javadoc fails in hadoop-aws
Akira Ajisaka created HADOOP-16554:
--
Summary: mvn javadoc:javadoc fails in hadoop-aws
Key: HADOOP-16554
URL: https://issues.apache.org/jira/browse/HADOOP-16554
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Akira Ajisaka
mvn javadoc:javadoc fails in hadoop-aws module.
{noformat}
[ERROR]
/(snip)/hadoop-mirror/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java:619:
error: reference not found
[ERROR] * See {@link CommitOperations#revertCommit(SinglePendingCommit)}.
[ERROR] ^
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Moving Submarine to a separate Apache project proposal
Thanks everyone for voting, support, and suggestions! Best, Wangda On Sun, Sep 8, 2019 at 10:57 PM Sree Vaddi wrote: > Congratulations, Wangda. Thank you. > > Let's work towards it's better adoption. > > > Thank you. > /Sree > > > > > On Sunday, September 8, 2019, 10:30:49 PM PDT, Wangda Tan < > wheele...@gmail.com> wrote: > > > Thanks everybody for voting! Let me conclude the voting thread: > > We got 16 binding +1s from PMCs. And 29 +1s from Committers/Contributors. > With no veto. > > The vote now passed, we will work with the Apache community to get new > project created and continue the spin-off process. > > Best, > Wangda Tan > > > On Sun, Sep 8, 2019 at 11:24 AM Chen Liang wrote: > > > Late +1. I am interested in the project too. Please include me. > > > > Regards, > > Chen > > > > Varun Saxena 于2019年9月7日周六 上午10:35写道： > > > >> Belated +1. > >> I too am interested in the project. Please include me. > >> > >> Regards, > >> Varun Saxena > >> > >> On Sat, Sep 7, 2019 at 9:35 PM Sunil Govindan > wrote: > >> > >> > +1 to the proposal. > >> > Thanks to the community for the great response. > >> > > >> > - Sunil > >> > > >> > On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan > wrote: > >> > > >> > > Hi all, > >> > > > >> > > As we discussed in the previous thread [1], > >> > > > >> > > I just moved the spin-off proposal to CWIKI and completed all TODO > >> parts. > >> > > > >> > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal > >> > > > >> > > If you have interests to learn more about this. Please review the > >> > proposal > >> > > let me know if you have any questions/suggestions for the proposal. > >> This > >> > > will be sent to board post voting passed. (And please note that the > >> > > previous voting thread [2] to move Submarine to a separate Github > repo > >> > is a > >> > > necessary effort to move Submarine to a separate Apache project but > >> not > >> > > sufficient so I sent two separate voting thread.) > >> > > > >> > > Please let me know if I missed anyone in the proposal, and reply if > >> you'd > >> > > like to be included in the project. > >> > > > >> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM > >> PDT. > >> > > > >> > > Thanks, > >> > > Wangda Tan > >> > > > >> > > [1] > >> > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E > >> > > [2] > >> > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E > >> > > > >> > > >> > > >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] Failed CTEST tests : test_test_libhdfs_ops_hdfs_static test_test_libhdfs_threaded_hdfs_static test_test_libhdfs_zerocopy_hdfs_static test_test_native_mini_dfs test_libhdfs_threaded_hdfspp_test_shim_static test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static libhdfs_mini_stress_valgrind_hdfspp_test_static memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static test_libhdfs_mini_stress_hdfspp_test_shim_static test_hdfs_ext_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.TestMultipleNNPortQOP hadoop.hdfs.tools.TestDFSZKFailoverController hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-compile-javac-root.txt [332K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-patch-pylint.txt [220K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/diff-patch-shelldocs.txt [44K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/whitespace-eol.txt [9.6M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/xml.txt [16K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-mawo_hadoop-yarn-applications-mawo-core-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/branch-findbugs-hadoop-hdds_client.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1254/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [16K]
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.namenode.TestNameNodeMXBean hadoop.hdfs.TestDFSRemove hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.TestTrashWithSecureEncryptionZones hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.mapreduce.lib.input.TestMRKeyValueTextInputFormat hadoop.mapred.TestReduceFetch cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/whitespace-tabs.txt [1.3M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [340K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/439/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt [12K]
