This is an automated email from the ASF dual-hosted git repository.
shv pushed a change to branch fgl
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
omit 7598a68 HDFS-16130. [FGL] Implement CREATE File with FGL. Contributed
by Renukaprasad C. (#3205)
omit 0a73bde HDFS-16128. [FGL] Added support for saving/loading an FS
Image for PartitionedGSet. Contributed by Xing Lin. (#3201)
omit d6efb60 HDFS-16125. [FGL] Fix the iterator for PartitionedGSet.
Contributed by Xing Lin. (#3197)
omit b784277 Add namespace key for INode. (shv)
omit 6c2b39e INodeMap with PartitionedGSet and per-partition locking.
add 05b6a1a YARN-10833. Set the X-FRAME-OPTIONS header for the default
contexts. (#3203)
add 4c35466 HADOOP-17317. [JDK 11] Upgrade dnsjava to remove illegal
access warnings (#2442)
add dd8e540 Addendum HADOOP-17770 WASB : Support disabling buffered reads
in positional reads - Added the invalid SpotBugs warning to
findbugs-exclude.xml (#3223)
add 2f2f822 HDFS-12920. HDFS default value change (with adding time unit)
breaks old version MR tarball work with new version (3.0) of hadoop. (#3227)
add b7431c3 [UI2] Bump http-proxy to 1.18.1 (#2891)
add 5d76549 HDFS-16131. Show storage type for failed volumes on namenode
web (#3211). Contributed by tomscut.
add d710ec8 HDFS-16140. TestBootstrapAliasmap fails by BindException.
(#3229)
add 97c88c9 HADOOP-17807. Use separate src dir for platform builds (#3210)
add b038042 HDFS-16139. Update BPServiceActor Scheduler's
nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani.
add f813554 HADOOP-13887. Support S3 client side encryption (S3-CSE)
using AWS-SDK (#2706)
add fa0289b YARN-6221. Entities missing from ATS when summary log file
info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
add aecfcf1 HDFS-16119. start balancer with parameters
-hotBlockTimeInterval xxx is invalid. (#3185)
add 10ba4cc HADOOP-17765. ABFS: Use Unique File Paths in Tests. (#3153)
add ae20516 HDFS-16111. Add a configuration to
RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175)
add b4a5247 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails
intermittent (#3212)
add dac10fc HDFS-16145. CopyListing fails with FNF exception with
snapshot diff. (#3234)
add fd13970 HDFS-16137.Improve the comments related to
FairCallQueue#queues. (#3226)
add 8d0297c YARN-10727. ParentQueue does not validate the queue on
removal. Contributed by Andras Gyori
add 4eae284 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer
see attribute provider permissions). Contributed by Stephen O'Donnell
add b19dae8 HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard
enabled (#3239)
add 1b9efe5 YARN-10790. CS Flexible AQC: Add separate parent and leaf
template property. Contributed by Andras Gyori
add f2b6c03 YARN-6272.
TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently.
Contributed by Andras Gyory & Prabhu Joseph
add e001f8e HADOOP-17814. Provide fallbacks for identity/cost providers
and backoff enable (#3230)
add 1d03c69 HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up
configuration values (#3221)
add 3c8a48e HADOOP-17819. Add extensions to ProtobufRpcEngine
RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242)
add 683feaa HDFS-15175. Multiple CloseOp shared block instance causes the
standby namenode to crash when rolling editlog. Contributed by Wan Chang.
add 6f730fd HDFS-15936.Solve SocketTimeoutException#sendPacket() does not
record SocketTimeout exception. (#2836)
add d78b300 YARN-10841. Fix token reset synchronization for UAM response
token. (#3194)
add 54f9fff YARN-10628. Add node usage metrics in SLS. Contributed by
Vadaga Ananyo Rao
add 74770c8 YARN-10663. Add runningApps stats in SLS. Contributed by
Vadaga Ananyo Rao
add ac0a4e7 YARN-10869. CS considers only the default
maximum-allocation-mb/vcore property as a maximum when it creates dynamic
queues (#3225)
add 8f750c5 YARN-10856. Prevent ATS v2 health check REST API call if the
ATS service itself is disabled. (#3236)
add 13467f4 HADOOP-17815. Run CI for Centos 7 (#3231)
add 798a083 YARN-10814. Fallback to RandomSecretProvider if the secret
file is empty (#3206)
add 6d77f3b HDFS-14529. SetTimes to throw FileNotFoundException if inode
is not found (#3243)
add a218038 HADOOP-17139 Re-enable optimized copyFromLocal implementation
in S3AFileSystem (#3101)
add 266b1bd HADOOP-17812. NPE in S3AInputStream read() after failure to
reconnect to store (#3222)
new 2e3eb2d INodeMap with PartitionedGSet and per-partition locking.
new e00c784 Add namespace key for INode. (shv)
new 6ff40a0 HDFS-16125. [FGL] Fix the iterator for PartitionedGSet.
Contributed by Xing Lin. (#3197)
new b135d26 HDFS-16128. [FGL] Added support for saving/loading an FS
Image for PartitionedGSet. Contributed by Xing Lin. (#3201)
new 0f1a9a8 HDFS-16130. [FGL] Implement CREATE File with FGL. Contributed
by Renukaprasad C. (#3205)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (7598a68)
\
N -- N -- N refs/heads/fgl (0f1a9a8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev-support/Jenkinsfile | 254 +++++++---
dev-support/docker/Dockerfile_centos_7 | 19 +-
.../{install-cmake.sh => install-git.sh} | 16 +-
.../{install-zstandard.sh => install-nodejs.sh} | 15 +-
dev-support/docker/pkg-resolver/packages.json | 13 +-
dev-support/jenkins.sh | 68 ++-
.../server/AuthenticationFilter.java | 4 +-
.../util/FileSignerSecretProvider.java | 27 +-
.../server/TestAuthenticationFilter.java | 28 ++
.../util/TestFileSignerSecretProvider.java | 27 ++
.../main/java/org/apache/hadoop/fs/FileUtil.java | 3 +
.../org/apache/hadoop/ipc/DecayRpcScheduler.java | 24 +
.../java/org/apache/hadoop/ipc/FairCallQueue.java | 9 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 33 +-
.../org/apache/hadoop/security/SecurityUtil.java | 7 +-
.../org/apache/hadoop/util/PartitionedGSet.java | 3 +
.../src/main/proto/ProtobufRpcEngine.proto | 3 +
.../src/main/proto/ProtobufRpcEngine2.proto | 3 +
.../src/main/resources/core-default.xml | 35 ++
.../src/site/markdown/filesystem/filesystem.md | 106 ++++
.../hadoop/conf/TestCommonConfigurationFields.java | 3 +
.../apache/hadoop/fs/TestLocalFSCopyFromLocal.java | 98 ++++
.../AbstractContractCopyFromLocalTest.java | 336 +++++++++++++
.../AbstractContractMultipartUploaderTest.java | 3 +
.../apache/hadoop/ipc/TestDecayRpcScheduler.java | 61 +++
.../hadoop/registry/server/dns/RegistryDNS.java | 77 ++-
.../hadoop/registry/server/dns/SecureableZone.java | 3 +-
.../registry/server/dns/TestRegistryDNS.java | 118 ++---
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 8 +-
.../hadoop/hdfs/server/balancer/Balancer.java | 10 +-
.../hdfs/server/balancer/BalancerParameters.java | 4 +
.../hdfs/server/datanode/BPServiceActor.java | 42 +-
.../hadoop/hdfs/server/datanode/BlockSender.java | 1 +
.../hadoop/hdfs/server/datanode/DataNode.java | 2 +-
.../fsdataset/RoundRobinVolumeChoosingPolicy.java | 29 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 4 +-
.../hadoop/hdfs/server/namenode/FSDirAttrOp.java | 13 +-
.../hadoop/hdfs/server/namenode/FSDirMkdirOp.java | 4 +-
.../hadoop/hdfs/server/namenode/FSDirectory.java | 73 +--
.../hadoop/hdfs/server/namenode/FSEditLogOp.java | 22 +-
.../hdfs/server/namenode/FSPermissionChecker.java | 44 +-
.../hadoop/hdfs/server/namenode/INodeMap.java | 54 +--
.../hadoop/hdfs/server/namenode/INodesInPath.java | 21 -
.../src/main/resources/hdfs-default.xml | 33 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 3 +
.../hadoop/hdfs/server/balancer/TestBalancer.java | 48 ++
.../datanode/TestBpServiceActorScheduler.java | 23 +-
.../server/datanode/TestDataNodeVolumeFailure.java | 4 +
.../TestDataNodeVolumeFailureReporting.java | 5 +-
.../TestRoundRobinVolumeChoosingPolicy.java | 48 +-
.../hdfs/server/namenode/TestFSDirAttrOp.java | 18 +-
.../namenode/TestINodeAttributeProvider.java | 155 ++----
.../server/namenode/ha/TestBootstrapAliasmap.java | 3 +
hadoop-project/pom.xml | 2 +-
.../java/org/apache/hadoop/fs/s3a/Constants.java | 7 +-
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 135 +++++-
.../java/org/apache/hadoop/fs/s3a/Listing.java | 9 +-
.../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 43 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 214 +++++---
.../org/apache/hadoop/fs/s3a/S3AInputStream.java | 29 +-
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 46 +-
.../java/org/apache/hadoop/fs/s3a/Statistic.java | 9 +-
.../hadoop/fs/s3a/impl/CopyFromLocalOperation.java | 540 +++++++++++++++++++++
.../hadoop/fs/s3a/impl/HeaderProcessing.java | 11 +-
.../hadoop/fs/s3a/impl/InternalConstants.java | 8 +
.../apache/hadoop/fs/s3a/impl/RenameOperation.java | 2 +-
.../apache/hadoop/fs/s3a/impl/StoreContext.java | 15 +-
.../hadoop/fs/s3a/impl/StoreContextBuilder.java | 16 +-
.../site/markdown/tools/hadoop-aws/encryption.md | 234 ++++++++-
.../tools/hadoop-aws/troubleshooting_s3a.md | 303 ++++++++++++
.../apache/hadoop/fs/s3a/AbstractS3ATestBase.java | 9 +
.../apache/hadoop/fs/s3a/EncryptionTestUtils.java | 2 +-
.../fs/s3a/ITestS3AClientSideEncryption.java | 301 ++++++++++++
.../fs/s3a/ITestS3AClientSideEncryptionKms.java | 95 ++++
.../hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java | 140 +-----
.../ITestS3AEncryptionSSEKMSUserDefinedKey.java | 9 +-
.../ITestS3AEncryptionWithDefaultS3Settings.java | 6 +-
.../hadoop/fs/s3a/ITestS3AInconsistency.java | 9 +
.../hadoop/fs/s3a/ITestS3AMiscOperations.java | 3 +-
.../hadoop/fs/s3a/ITestS3GuardListConsistency.java | 6 +-
.../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 13 +
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 12 +
.../hadoop/fs/s3a/TestS3AInputStreamRetry.java | 21 +-
.../apache/hadoop/fs/s3a/TestSSEConfiguration.java | 6 +-
.../hadoop/fs/s3a/auth/ITestCustomSigner.java | 34 +-
.../ITestSessionDelegationInFileystem.java | 23 +-
.../hadoop/fs/s3a/commit/AbstractCommitITest.java | 9 +-
.../fileContext/ITestS3AFileContextStatistics.java | 33 +-
.../fs/s3a/scale/ITestS3AHugeFilesEncryption.java | 8 +-
.../s3a/scale/ITestS3AHugeFilesSSECDiskBlocks.java | 2 +-
.../s3a/scale/ITestS3AInputStreamPerformance.java | 5 +
.../hadoop-azure/dev-support/findbugs-exclude.xml | 13 +
.../fs/azurebfs/AzureBlobFileSystemStore.java | 2 +-
.../azurebfs/services/ExponentialRetryPolicy.java | 11 +
.../hadoop-azure/src/site/markdown/abfs.md | 28 +-
.../src/site/markdown/testing_azure.md | 4 +-
.../fs/azurebfs/AbstractAbfsIntegrationTest.java | 21 +-
.../apache/hadoop/fs/azurebfs/ITestAbfsClient.java | 11 +-
.../ITestAbfsListStatusRemoteIterator.java | 6 +-
.../fs/azurebfs/ITestAbfsReadWriteAndSeek.java | 12 +-
.../hadoop/fs/azurebfs/ITestAbfsStatistics.java | 18 +-
.../fs/azurebfs/ITestAbfsStreamStatistics.java | 4 +-
.../azurebfs/ITestAzureBlobFileSystemAppend.java | 23 +-
.../ITestAzureBlobFileSystemAuthorization.java | 6 +-
.../ITestAzureBlobFileSystemBackCompat.java | 9 +-
.../ITestAzureBlobFileSystemCheckAccess.java | 3 +-
.../fs/azurebfs/ITestAzureBlobFileSystemCopy.java | 2 +-
.../azurebfs/ITestAzureBlobFileSystemCreate.java | 26 +-
.../ITestAzureBlobFileSystemDelegationSAS.java | 34 +-
.../azurebfs/ITestAzureBlobFileSystemDelete.java | 27 +-
.../fs/azurebfs/ITestAzureBlobFileSystemE2E.java | 59 +--
.../ITestAzureBlobFileSystemFileStatus.java | 28 +-
.../fs/azurebfs/ITestAzureBlobFileSystemFlush.java | 15 +-
.../ITestAzureBlobFileSystemListStatus.java | 31 +-
.../fs/azurebfs/ITestAzureBlobFileSystemMkDir.java | 8 +-
.../fs/azurebfs/ITestAzureBlobFileSystemOauth.java | 58 ++-
.../ITestAzureBlobFileSystemPermission.java | 3 +-
.../ITestAzureBlobFileSystemRandomRead.java | 19 +-
.../azurebfs/ITestAzureBlobFileSystemRename.java | 45 +-
.../ITestAzureBlobFileSystemRenameUnicode.java | 2 +-
.../fs/azurebfs/ITestAzureBlobFilesystemAcl.java | 5 +-
.../fs/azurebfs/ITestCustomerProvidedKey.java | 60 ++-
.../fs/azurebfs/ITestFileSystemProperties.java | 31 +-
.../fs/azurebfs/ITestWasbAbfsCompatibility.java | 20 +-
.../azurebfs/services/ITestAbfsOutputStream.java | 26 +-
.../fs/azurebfs/services/TestAbfsInputStream.java | 8 +-
.../services/TestExponentialRetryPolicy.java | 40 +-
.../java/org/apache/hadoop/tools/DistCpSync.java | 64 ++-
.../org/apache/hadoop/tools/TestDistCpSync.java | 170 +++++++
hadoop-tools/hadoop-sls/pom.xml | 5 +
.../src/main/html/showSimulationTrace.html | 21 +
.../src/main/html/simulate.html.template | 28 +-
.../hadoop/yarn/sls/appmaster/AMSimulator.java | 14 +
.../hadoop/yarn/sls/appmaster/DAGAMSimulator.java | 3 +-
.../hadoop/yarn/sls/appmaster/MRAMSimulator.java | 6 +-
.../yarn/sls/appmaster/StreamAMSimulator.java | 3 +-
.../hadoop/yarn/sls/nodemanager/NMSimulator.java | 21 +-
.../yarn/sls/resourcemanager/MockAMLauncher.java | 3 +-
.../yarn/sls/scheduler/SchedulerMetrics.java | 47 ++
.../hadoop/yarn/sls/utils/NodeUsageRanges.java | 68 +++
.../hadoop/yarn/sls/utils}/package-info.java | 5 +-
.../org/apache/hadoop/yarn/sls/web/SLSWebApp.java | 33 ++
.../hadoop/yarn/sls/appmaster/TestAMSimulator.java | 29 ++
.../yarn/sls/nodemanager/TestNMSimulator.java | 100 +++-
.../yarn/client/api/impl/TestAMRMClient.java | 55 ++-
.../org/apache/hadoop/yarn/webapp/WebApps.java | 31 +-
.../client/api/impl/TestTimelineClientV2Impl.java | 36 +-
.../amrmproxy/FederationInterceptor.java | 15 +-
.../scheduler/capacity/AbstractCSQueue.java | 8 +-
.../capacity/AutoCreatedQueueTemplate.java | 102 +++-
.../capacity/CapacitySchedulerConfiguration.java | 14 +
.../scheduler/capacity/ParentQueue.java | 15 +
.../webapp/dao/CapacitySchedulerInfo.java | 14 +-
.../webapp/dao/CapacitySchedulerQueueInfo.java | 16 +-
.../dao/helper/CapacitySchedulerInfoHelper.java | 5 +-
.../capacity/TestAutoCreatedQueueTemplate.java | 41 ++
.../TestCapacitySchedulerAutoCreatedQueueBase.java | 32 +-
.../TestCapacitySchedulerAutoQueueCreation.java | 33 +-
.../TestCapacitySchedulerNewQueueAutoCreation.java | 45 ++
.../webapp/TestRMWebServicesCapacitySched.java | 6 +-
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
.../hadoop/yarn/webapp/TestRMWithXFSFilter.java | 155 +++---
.../timeline/EntityGroupFSTimelineStore.java | 3 +-
.../timeline/TestEntityGroupFSTimelineStore.java | 25 +
.../src/main/webapp/app/initializers/loader.js | 8 +
.../src/main/webapp/app/routes/application.js | 18 +-
.../src/main/webapp/app/routes/yarn-app/logs.js | 14 +-
.../hadoop-yarn-ui/src/main/webapp/yarn.lock | 22 +-
168 files changed, 5156 insertions(+), 1333 deletions(-)
copy dev-support/docker/pkg-resolver/{install-cmake.sh => install-git.sh} (77%)
copy dev-support/docker/pkg-resolver/{install-zstandard.sh =>
install-nodejs.sh} (79%)
create mode 100644
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSCopyFromLocal.java
create mode 100644
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCopyFromLocalTest.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AClientSideEncryption.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AClientSideEncryptionKms.java
create mode 100644
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/utils/NodeUsageRanges.java
copy hadoop-tools/{hadoop-extras/src/main/java/org/apache/hadoop/tools =>
hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/utils}/package-info.java
(91%)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
