This is an automated email from the ASF dual-hosted git repository.
mthakur pushed a change to branch feature-vectored-io
in repository https://gitbox.apache.org/repos/asf/hadoop.git
discard dac8c48e914 HADOOP-18106: Handle memory fragmentation in S3A Vectored
IO. (#4445)
omit 7668f8d6e69 HADOOP-18105 Implement buffer pooling with weak references
(#4263)
omit 9843cc041ab HADOOP-18107 Adding scale test for vectored reads for
large file (#4273)
omit eccfb1b3758 HADOOP-18104: S3A: Add configs to configure
minSeekForVectorReads and maxReadSizeForVectorReads (#3964)
omit 44fa8a44a2f HADOOP-11867. Add a high-performance vectored read API.
(#3904)
add 6cbeae2e52c HDFS-16581.Print node status when executing printTopology.
(#4321)
add 9e3fc40ecb5 HDFS-16613. EC: Improve performance of decommissioning dn
with many ec blocks (#4398)
add 7bfff63774d HADOOP-18289. Remove WhiteBox in hadoop-kms module. (#4433)
add 4893f003958 HDFS-16600. Fix deadlock of fine-grain lock for
FsDatastImpl of DataNode. (#4367). Contributed by ZanderXu.
add 020201cb65e Queue filter in CS UI v1 does not work as expected.
Contributed by Chengbing Liu.
add e199da3fae1 HADOOP-17833. Improve Magic Committer performance (#3289)
add 80446dcd086 YARN-11172. Fix TestClientRMTokens#testDelegationToken
introduced by HDFS-16563. (#4408)
add 62e44761020 YARN-10122. Support signalToContainer API for Federation.
(#4421)
add e38e13be03d HADOOP-18288. Total requests and total requests per sec
served by RPC servers (#4431)
add cb0421095b8 HDFS-16634. Dynamically adjust slow peer report size on
JMX metrics (#4448)
add cfceaebde60 HDFS-16064. Determine when to invalidate corrupt replicas
based on number of usable replicas (#4410)
add 4f425b641cb YARN-9827.Fix Http Response code in
GenericExceptionHandler (#4393)
add a77d52284f9 HADOOP-18255. Fix fsdatainputstreambuilder.md reference to
hadoop branch-3.3 (#4378)
add efc2761d32a HDFS-16635.Fixed javadoc error in Java 11 (#4451)
add 477b67a3358 HADOOP-18266. Using HashSet/ TreeSet Constructor for
hadoop-common (#4365)
add 10fc865d3ce MAPREDUCE-7387. Fix TestJHSSecurity#testDelegationToken
AssertionError due to HDFS-16563 (#4428). Contributed by fanshilun.
add 36c4be819ff MAPREDUCE-7369. Fixed MapReduce tasks timing out when
spends more time on MultipleOutputs#close (#4247)
add 5d08ffa769d YARN-11182. Refactor TestAggregatedLogDeletionService: 2nd
phase. Contributed by Szilard Nemeth.
add 3a66348fda2 YARN-11185. Pending app metrics are increased doubly when
a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori
new f3f71434b58 HADOOP-11867. Add a high-performance vectored read API.
(#3904)
new 307bd14e1e2 HADOOP-18104: S3A: Add configs to configure
minSeekForVectorReads and maxReadSizeForVectorReads (#3964)
new 06407903ceb HADOOP-18107 Adding scale test for vectored reads for
large file (#4273)
new b5520ac778f HADOOP-18105 Implement buffer pooling with weak references
(#4263)
new 645bcd64802 HADOOP-18106: Handle memory fragmentation in S3A Vectored
IO. (#4445)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (dac8c48e914)
\
N -- N -- N refs/heads/feature-vectored-io (645bcd64802)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../security/authentication/util/KerberosUtil.java | 2 +-
.../security/authentication/KerberosTestUtils.java | 4 +-
.../java/org/apache/hadoop/conf/Configuration.java | 2 +-
.../hadoop/fs/CommonConfigurationKeysPublic.java | 8 +
.../java/org/apache/hadoop/fs/FileContext.java | 3 +-
.../org/apache/hadoop/fs/audit/AuditConstants.java | 5 +
.../apache/hadoop/fs/audit/CommonAuditContext.java | 8 +-
.../apache/hadoop/fs/sftp/SFTPConnectionPool.java | 4 +-
.../org/apache/hadoop/fs/shell/CommandFormat.java | 2 +-
.../java/org/apache/hadoop/fs/shell/find/Find.java | 2 +-
.../hadoop/fs/statistics/StoreStatisticNames.java | 3 +
.../apache/hadoop/fs/viewfs/ViewFileSystem.java | 2 +-
.../org/apache/hadoop/io/compress/CodecPool.java | 2 +-
.../serializer/avro/AvroReflectSerialization.java | 2 +-
.../java/org/apache/hadoop/ipc/ProtocolProxy.java | 2 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 70 +++
.../org/apache/hadoop/ipc/metrics/RpcMetrics.java | 10 +
.../apache/hadoop/metrics2/lib/MutableRates.java | 5 +-
.../metrics2/lib/MutableRatesWithAggregation.java | 4 +-
.../hadoop/net/AbstractDNSToSwitchMapping.java | 2 +-
.../org/apache/hadoop/net/NetworkTopology.java | 2 +-
.../hadoop/security/CompositeGroupsMapping.java | 2 +-
.../org/apache/hadoop/security/NetgroupCache.java | 2 +-
.../security/authorize/AccessControlList.java | 4 +-
.../hadoop/security/authorize/ProxyServers.java | 2 +-
.../security/http/RestCsrfPreventionFilter.java | 4 +-
.../AbstractDelegationTokenSecretManager.java | 8 +-
.../web/DelegationTokenAuthenticationHandler.java | 2 +-
.../org/apache/hadoop/util/FileBasedIPList.java | 2 +-
.../org/apache/hadoop/util/HostsFileReader.java | 2 +-
.../org/apache/hadoop/util/JsonSerialization.java | 5 +-
.../apache/hadoop/util/ShutdownHookManager.java | 2 +-
.../functional/CloseableTaskPoolSubmitter.java | 2 +-
.../hadoop-common/src/site/markdown/Metrics.md | 2 +
.../filesystem/fsdatainputstreambuilder.md | 2 +-
.../filesystem/fsdataoutputstreambuilder.md | 59 +-
.../hadoop/conf/TestCommonConfigurationFields.java | 6 +-
.../apache/hadoop/crypto/key/TestValueQueue.java | 9 +-
.../kms/TestLoadBalancingKMSClientProvider.java | 23 +-
.../org/apache/hadoop/fs/TestCommandFormat.java | 8 +-
.../apache/hadoop/fs/TestHarFileSystemBasics.java | 2 +-
.../java/org/apache/hadoop/fs/TestListFiles.java | 2 +-
.../test/java/org/apache/hadoop/fs/TestTrash.java | 2 +-
.../hadoop/fs/contract/ContractTestUtils.java | 13 +-
.../java/org/apache/hadoop/io/TestIOUtils.java | 2 +-
.../apache/hadoop/io/compress/TestCodecPool.java | 4 +-
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 58 ++
.../org/apache/hadoop/test/GenericTestUtils.java | 6 +-
.../apache/hadoop/test/MultithreadedTestUtil.java | 4 +-
.../hadoop/crypto/key/kms/server/KMSAudit.java | 6 +-
.../hadoop/crypto/key/kms/server/TestKMS.java | 12 +-
.../hadoop/crypto/key/kms/server/TestKMSAudit.java | 14 +-
.../hdfs/server/blockmanagement/BlockManager.java | 18 +-
.../server/blockmanagement/DatanodeManager.java | 18 +-
.../server/blockmanagement/SlowPeerTracker.java | 11 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 34 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 4 +-
.../hadoop/hdfs/server/namenode/NameNode.java | 15 +-
.../namenode/startupprogress/package-info.java | 2 +-
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 60 +-
.../org/apache/hadoop/hdfs/TestDecommission.java | 282 ++++++++++
.../hdfs/qjournal/server/TestJournalNodeSync.java | 4 +
.../blockmanagement/TestDatanodeManager.java | 25 +-
.../server/namenode/TestNameNodeReconfigure.java | 54 ++
.../namenode/ha/TestConsistentReadsObserver.java | 3 +
.../org/apache/hadoop/hdfs/tools/TestDFSAdmin.java | 66 ++-
.../hadoop/mapred/TaskAttemptListenerImpl.java | 15 +-
.../hadoop/mapred/TestTaskAttemptListenerImpl.java | 57 +-
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 7 +
.../manifest/files/AbstractManifestData.java | 1 -
.../src/main/resources/mapred-default.xml | 7 +
.../hadoop/mapreduce/security/TestJHSSecurity.java | 20 +-
hadoop-tools/hadoop-aws/pom.xml | 33 ++
.../java/org/apache/hadoop/fs/s3a/Constants.java | 17 +
.../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 59 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 378 ++++++++-----
.../apache/hadoop/fs/s3a/S3AInstrumentation.java | 18 +-
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 14 +-
.../java/org/apache/hadoop/fs/s3a/Statistic.java | 13 +
.../apache/hadoop/fs/s3a/WriteOperationHelper.java | 88 +--
.../org/apache/hadoop/fs/s3a/WriteOperations.java | 35 +-
.../apache/hadoop/fs/s3a/api/RequestFactory.java | 10 +-
.../hadoop/fs/s3a/commit/AbstractS3ACommitter.java | 604 ++++++++++++---------
.../hadoop/fs/s3a/commit/CommitConstants.java | 34 +-
.../fs/s3a/commit/CommitterStatisticNames.java | 211 +++++++
.../fs/s3a/commit/InternalCommitterConstants.java | 6 +
.../fs/s3a/commit/MagicCommitIntegration.java | 22 +-
.../org/apache/hadoop/fs/s3a/commit/Tasks.java | 423 ---------------
.../hadoop/fs/s3a/commit/files/PendingSet.java | 44 +-
.../fs/s3a/commit/files/PersistentCommitData.java | 115 +++-
.../fs/s3a/commit/files/SinglePendingCommit.java | 36 +-
.../hadoop/fs/s3a/commit/files/SuccessData.java | 119 +++-
.../hadoop/fs/s3a/commit/files/package-info.java | 7 +-
.../fs/s3a/commit/impl/AuditContextUpdater.java | 90 +++
.../hadoop/fs/s3a/commit/impl/CommitContext.java | 401 ++++++++++++++
.../fs/s3a/commit/{ => impl}/CommitOperations.java | 247 ++++-----
.../s3a/commit/{ => impl}/CommitUtilsWithMR.java | 68 ++-
.../s3a/commit/{files => impl}/package-info.java | 28 +-
.../fs/s3a/commit/magic/MagicCommitTracker.java | 55 +-
.../fs/s3a/commit/magic/MagicS3GuardCommitter.java | 150 ++---
.../commit/staging/DirectoryStagingCommitter.java | 8 +-
.../staging/PartitionedStagingCommitter.java | 41 +-
.../fs/s3a/commit/staging/StagingCommitter.java | 134 +++--
.../hadoop/fs/s3a/impl/CreateFileBuilder.java | 263 +++++++++
.../hadoop/fs/s3a/impl/InternalConstants.java | 10 +
.../apache/hadoop/fs/s3a/impl/MkdirOperation.java | 68 ++-
.../hadoop/fs/s3a/impl/PutObjectOptions.java | 104 ++++
.../hadoop/fs/s3a/impl/RequestFactoryImpl.java | 37 +-
.../hadoop/fs/s3a/impl/S3AMultipartUploader.java | 3 +-
.../statistics/BlockOutputStreamStatistics.java | 3 +-
.../fs/s3a/statistics/CommitterStatistics.java | 8 +
...erStatistics.java => PutTrackerStatistics.java} | 49 +-
.../statistics/impl/EmptyS3AStatisticsContext.java | 7 +
.../src/site/markdown/tools/hadoop-aws/auditing.md | 11 +-
.../site/markdown/tools/hadoop-aws/committers.md | 11 +-
.../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 63 ---
.../hadoop/fs/s3a/ITestS3AMiscOperations.java | 16 +-
.../apache/hadoop/fs/s3a/MockS3AFileSystem.java | 30 +-
.../apache/hadoop/fs/s3a/MultipartTestUtils.java | 3 +-
.../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 6 +
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 27 +-
.../hadoop/fs/s3a/TestS3ABlockOutputStream.java | 4 +-
.../fs/s3a/audit/ITestAuditAccessChecks.java | 4 +-
.../apache/hadoop/fs/s3a/auth/ITestAssumeRole.java | 39 +-
.../hadoop/fs/s3a/commit/AbstractCommitITest.java | 101 +++-
.../fs/s3a/commit/AbstractITCommitProtocol.java | 86 +--
.../hadoop/fs/s3a/commit/CommitterTestHelper.java | 148 +++++
.../fs/s3a/commit/ITestCommitOperationCost.java | 325 +++++++++++
.../fs/s3a/commit/ITestCommitOperations.java | 290 +++++-----
.../fs/s3a/commit/ITestS3ACommitterFactory.java | 2 +-
.../org/apache/hadoop/fs/s3a/commit/TestTasks.java | 569 -------------------
.../s3a/commit/magic/ITestMagicCommitProtocol.java | 27 +-
.../s3a/commit/magic/ITestS3AHugeMagicCommits.java | 29 +-
.../s3a/commit/staging/MockedStagingCommitter.java | 4 +-
.../fs/s3a/commit/staging/StagingTestBase.java | 39 +-
.../staging/TestDirectoryCommitterScale.java | 50 +-
.../s3a/commit/staging/TestStagingCommitter.java | 21 +-
.../TestStagingDirectoryOutputCommitter.java | 47 +-
.../staging/TestStagingPartitionedFileListing.java | 4 +-
.../staging/TestStagingPartitionedJobCommit.java | 17 +-
.../staging/TestStagingPartitionedTaskCommit.java | 10 +-
.../integration/ITestStagingCommitProtocol.java | 2 +-
.../fs/s3a/commit/terasort/ITestTerasortOnS3A.java | 2 +-
.../hadoop/fs/s3a/impl/StubContextAccessor.java | 82 +++
.../hadoop/fs/s3a/impl/TestCreateFileBuilder.java | 173 ++++++
.../hadoop/fs/s3a/impl/TestRequestFactory.java | 11 +-
.../fs/s3a/performance/AbstractS3ACostTest.java | 24 +-
.../fs/s3a/performance/ITestCreateFileCost.java | 248 +++++++++
.../fs/s3a/performance/ITestS3ADeleteCost.java | 11 +-
.../hadoop/fs/s3a/performance/OperationCost.java | 5 +-
.../fs/s3a/performance/OperationCostValidator.java | 18 +-
.../fs/s3a/scale/ITestS3ADirectoryPerformance.java | 5 +-
.../scale/ITestS3AMultipartUploadSizeLimits.java | 2 +-
.../AggregatedLogDeletionService.java | 4 +-
.../yarn/webapp/GenericExceptionHandler.java | 4 +-
.../logaggregation/LogAggregationTestUtils.java | 68 +++
.../TestAggregatedLogDeletionService.java | 514 +++++-------------
.../TestLogAggregationFileControllerFactory.java | 76 +--
.../AggregatedLogDeletionServiceForTest.java | 67 +++
.../logaggregation/testutils/FileStatusUtils.java | 76 +++
.../testutils/LogAggregationTestcase.java | 421 ++++++++++++++
.../testutils/LogAggregationTestcaseBuilder.java | 172 ++++++
.../testutils/MockRMClientUtils.java | 72 +++
.../testutils/PathWithFileStatus.java | 45 ++
.../scheduler/capacity/AbstractLeafQueue.java | 4 +-
.../webapp/CapacitySchedulerPage.java | 21 +-
.../server/resourcemanager/TestClientRMTokens.java | 17 +-
.../capacity/TestCapacitySchedulerApps.java | 77 ++-
.../hadoop/yarn/server/router/RouterMetrics.java | 33 ++
.../clientrm/FederationClientInterceptor.java | 38 +-
.../yarn/server/router/TestRouterMetrics.java | 33 ++
.../clientrm/TestFederationClientInterceptor.java | 45 ++
.../TestableFederationClientInterceptor.java | 11 +-
173 files changed, 6465 insertions(+), 3077 deletions(-)
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitterStatisticNames.java
delete mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/Tasks.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/AuditContextUpdater.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitContext.java
rename hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/{
=> impl}/CommitOperations.java (82%)
rename hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/{
=> impl}/CommitUtilsWithMR.java (74%)
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/{files =>
impl}/package-info.java (52%)
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CreateFileBuilder.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PutObjectOptions.java
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/statistics/{CommitterStatistics.java
=> PutTrackerStatistics.java} (50%)
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/CommitterTestHelper.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperationCost.java
delete mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/TestTasks.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/StubContextAccessor.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestCreateFileBuilder.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/LogAggregationTestUtils.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/AggregatedLogDeletionServiceForTest.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/FileStatusUtils.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/LogAggregationTestcase.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/LogAggregationTestcaseBuilder.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/MockRMClientUtils.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/testutils/PathWithFileStatus.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
