This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch HDDS-2939
in repository https://gitbox.apache.org/repos/asf/ozone.git.
discard ff04733 HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
omit f707bba HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
omit 155fd00 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
omit 0f9bc0e HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
add e800303 HDDS-4339. Allow AWSSignatureProcessor init when aws
signature is absent. (#1498)
add 5d9af79 HDDS-4410. Cache DatanodeDetails#getUuidString(). (#1538).
Contributed by Ayush Saxena.
add b4c839b HDDS-4418. TestContainerMetrics is flaky (#1543)
add 7d955be HDDS-4421. Fix throwing INTERNAL_ERROR out of
LOG.isDebugEnabled() check in OzoneClientProducer. (#1545)
add ca220f9 HDDS-4413. Container replication should fail in case of
import failure. (#1537)
add 046c2a8 HDDS-4431. Improve output message for key/bucket list
command. (#1553)
add 08151fa HDDS-4416. Update README.md after TLP separation (#1540)
add 0ad6aba HDDS-4434.Fix typo in Admin.md (#1556)
add 239d05c HDDS-4425. NM container does not wait for RM due to typo
(#1550)
add 8bcd859 HDDS-4420. Missing a container shouldn't log message at ERROR
level (#1554)
add 1f6b0d9 HDDS-4446. Update Jetty version to 9.4.34. (#1566)
add 2ef4cf9 HDDS-4445. Owner info is not passed to authorizer for
BUCKET/KEY crea… (#1563)
add 3b53f9f
HDDS-4436.TestReplicationManager#testUnderReplicatedQuasiClosedContainerWithUnhealthyReplica
fails intermittently. (#1560)
add 50e5765 HDDS-4438. Avoid counting skipped transactions belonging to
open containers. (#1565)
add fd9d0ce HDDS-4435. Fix can not insert the second entry when hive on
ozone (#1559)
add 90190f5 HDDS-4307.Start Trash Emptier in Ozone Manager (#1507)
add a28cda5 HDDS-4363. Add metric to track the number of RocksDB
open/close operations. (#1530)
add fdbd5da HDDS-4417. Simplify Ozone client code with configuration
object (#1542)
add d2b7135 HDDS-4437. Avoid unnecessary builder conversion in setting
volume Quota/Owner request (#1564)
add f3c8900 HDDS-4441. Add metrics for ACL related operations. (#1571)
add 8523775 HDDS-4455. Fix typo in README.md doc (#1578)
add e56d7bc HDDS-4315. Use Epoch to generate unique ObjectIDs (#1480)
add 7a52990 HDDS-4443. Recon: Using Mysql database throws exception and
fails startup (#1570)
add 277a589 HDDS-4408: terminate Datanode when Datanode State Machine
Thread got uncaught exception. (#1533)
add 5e8aaee HDDS-4453. Replicate closed container for random selected
datanodes. (#1574)
add 5ec98e1 HDDS-4083. Create ZH translation of Recon.md in doc (#1575)
add 760c1e8 HDDS-4337. Implement RocksDB options cache for new datanode
DB utilities. (#1544)
add 0e35176 HDDS-4338. Fix the issue that SCM web UI banner shows "HDFS
SCM". (#1583)
add b80d9b0 HDDS-4235. Ozone client FS path validation is not present in
OFS. (#1582)
add 5f43814 HDDS-4184. Add Features menu for Chinese document. (#1547)
add 45e30d6 HDDS-4079. Create ZH translation of Containers.md in doc.
(#1539)
add fcf0a4d HDDS-4080. Create ZH translation of OzoneManager.md in doc.
(#1541)
add 7df5dba HDDS-4081. Create ZH translation of
StorageContainerManager.md in doc. (#1558)
add 585984e HDDS-4441. Add metrics for ACL related operations.(Addendum
for HA). (#1584)
add d203b64 HDDS-4442. Disable the location information of audit logger
to reduce overhead (#1567)
add 787e5b6 HDDS-4458. Fix Max Transaction ID value in OM. (#1585)
add e738172 HDDS-4466. Update url in .asf.yaml to use TLP project (#1592)
add 9062c5b HDDS-4467. Acceptance test fails due to new Hadoop 3 image
(#1594)
add 1cd5863 HDDS-4464. Upgrade httpclient version due to CVE-2020-13956.
(#1590)
add d0aa34c HDDS-4082. Create ZH translation of HA.md in doc. (#1591)
add 04a757a HDDS-4462. Add --frozen-lockfile to pnpm install to prevent
ozone-recon-web/pnpm-lock.yaml from being updated automatically (#1589)
add 4e36d03 HDDS-4367. Configuration for deletion service intervals
should be different for OM, SCM and datanodes (#1573)
add 8b2c2eb HDDS-4399. Safe mode rule for piplelines should only consider
open pipelines. (#1526)
add 40c1944 HDDS-4426. SCM should create transactions using all blocks
received from OM (#1561)
add 0aca5c7 HDDS-4346.Ozone specific Trash Policy (#1535)
add 541ae9f HDDS-4450. Cannot run ozone if HADOOP_HOME points to Hadoop
install (#1572)
add bbeaf65 HDDS-4448. Duplicate refreshPipeline in listStatus (#1569)
add 58ec7f4 HDDS-4427. Avoid ContainerCache in ContainerReader at
Datanode startup (#1549)
add 417082c HDDS-4452. findbugs.sh couldn't be executed after a full
build (#1576)
add 8ae1408 HDDS-4478. Large deletedKeyset slows down OM via listStatus.
(#1598)
new fbd5707 HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
new 4124a34 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
new 8bd8e7a HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
new 28f2bbc HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ff04733)
\
N -- N -- N refs/heads/HDDS-2939 (28f2bbc)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 2 +-
README.md | 22 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 213 +++++++++++++
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 66 ++--
.../storage/TestBlockOutputStreamCorrectness.java | 17 +-
.../hdds/scm/storage/TestChunkInputStream.java | 11 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 5 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 23 ++
.../hdds/scm/storage/ContainerProtocolCalls.java | 4 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 73 +----
.../org/apache/hadoop/ozone/common/Checksum.java | 22 +-
.../common/src/main/resources/ozone-default.xml | 91 +-----
.../ratis/TestContainerCommandRequestMessage.java | 13 +-
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 8 +-
.../common/src/test/resources/auditlog.properties | 2 +-
.../hdds/conf/ConfigurationReflectionUtil.java | 14 +
.../hadoop/hdds/conf/ConfigurationTarget.java | 4 +
.../common/statemachine/DatanodeConfiguration.java | 24 ++
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../container/common/utils/ContainerCache.java | 52 +--
.../common/utils/ContainerCacheMetrics.java | 114 +++++++
.../container/keyvalue/helpers/BlockUtils.java | 54 ++++
.../keyvalue/helpers/KeyValueContainerUtil.java | 107 ++++---
.../container/metadata/AbstractDatanodeStore.java | 65 +++-
.../ozone/container/ozoneimpl/OzoneContainer.java | 12 +-
.../replication/DownloadAndImportReplicator.java | 12 +-
.../replication/GrpcReplicationClient.java | 23 +-
.../replication/SimpleContainerDownloader.java | 36 ++-
.../main/resources/webapps/hddsDatanode/index.html | 2 +-
.../ozone/container/ContainerTestHelper.java | 6 +-
.../container/common/TestBlockDeletingService.java | 9 +-
.../ozone/container/common/TestContainerCache.java | 8 +
.../common/impl/TestContainerPersistence.java | 53 ++--
.../container/keyvalue/TestKeyValueContainer.java | 54 +++-
.../container/ozoneimpl/TestContainerReader.java | 13 +-
.../container/ozoneimpl/TestOzoneContainer.java | 2 +
.../replication/TestReplicationSupervisor.java | 35 ++
.../replication/TestSimpleContainerDownloader.java | 85 +++++
hadoop-hdds/docs/content/concept/Containers.zh.md | 47 +++
.../docs/content/concept/OzoneManager.zh.md | 92 ++++--
.../content/concept/StorageContainerManager.zh.md | 69 +++-
hadoop-hdds/docs/content/feature/GDPR.md | 2 +-
hadoop-hdds/docs/content/feature/GDPR.zh.md | 5 +-
hadoop-hdds/docs/content/feature/HA.md | 4 +-
hadoop-hdds/docs/content/feature/HA.zh.md | 117 +++++++
hadoop-hdds/docs/content/feature/Quota.zh.md | 3 +
hadoop-hdds/docs/content/feature/Recon.zh.md | 49 +++
hadoop-hdds/docs/content/feature/_index.zh.md | 7 +-
hadoop-hdds/docs/content/interface/_index.zh.md | 2 +-
hadoop-hdds/docs/content/tools/Admin.md | 4 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 351 +++++++++++----------
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 21 +-
hadoop-hdds/pom.xml | 50 +--
.../apache/hadoop/hdds/scm/block/BlockManager.java | 4 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 50 +--
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 29 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 5 +-
.../IncrementalContainerReportHandler.java | 4 +
.../hadoop/hdds/scm/metadata/PipelineCodec.java | 3 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 8 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 42 ++-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 3 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 45 ++-
.../hdds/scm/server/SCMBlockProtocolServer.java | 85 +++--
.../src/main/resources/webapps/scm/index.html | 6 +-
.../hdds/scm/container/TestReplicationManager.java | 12 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 4 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 6 +
.../TestOneReplicaPipelineSafeModeRule.java | 74 +++--
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 34 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 135 ++------
.../client/io/BlockOutputStreamEntryPool.java | 89 ++----
.../hadoop/ozone/client/io/KeyOutputStream.java | 115 +++----
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 99 +-----
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 59 +++-
.../org/apache/hadoop/ozone/util/ExitManager.java | 2 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 5 +
hadoop-ozone/datanode/pom.xml | 23 ++
hadoop-ozone/dist/pom.xml | 48 +--
.../docker-image/docker-krb5/Dockerfile-krb5 | 2 +-
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 2 +-
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 2 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 4 +-
.../compose/ozonesecure-om-ha/docker-compose.yaml | 4 +-
.../main/compose/ozonesecure/docker-compose.yaml | 2 +
.../main/smoketest/security/ozone-secure-fs.robot | 19 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 11 +-
hadoop-ozone/dist/src/shell/ozone/stop-ozone.sh | 5 +
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 28 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 13 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 85 ++++-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 6 +
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 49 +--
.../hadoop/ozone/TestStorageContainerManager.java | 67 ++--
.../ozone/client/rpc/TestBlockOutputStream.java | 34 +-
.../rpc/TestBlockOutputStreamFlushDelay.java | 22 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 39 ++-
...estBlockOutputStreamWithFailuresFlushDelay.java | 34 +-
.../rpc/TestCloseContainerHandlingByClient.java | 35 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 7 +-
.../client/rpc/TestContainerStateMachine.java | 35 +-
.../rpc/TestContainerStateMachineFailures.java | 64 ++--
.../client/rpc/TestDeleteWithSlowFollower.java | 6 +-
.../client/rpc/TestDiscardPreallocatedBlocks.java | 35 +-
.../client/rpc/TestFailureHandlingByClient.java | 7 +-
.../ozone/client/rpc/TestKeyInputStream.java | 33 +-
.../rpc/TestOzoneClientRetriesOnException.java | 43 +--
...estOzoneClientRetriesOnExceptionFlushDelay.java | 31 +-
.../ozone/client/rpc/TestOzoneRpcClient.java | 7 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 23 +-
.../client/rpc/TestOzoneRpcClientWithRatis.java | 2 +
.../client/rpc/TestValidateBCSIDOnRestart.java | 41 ++-
.../ozone/client/rpc/TestWatchForCommit.java | 46 +--
.../commandhandler/TestBlockDeletion.java | 10 +
.../container/metrics/TestContainerMetrics.java | 43 ++-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 105 ++++++
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 100 +++++-
.../src/test/resources/contract/ozone.xml | 5 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 5 +
.../hadoop/ozone/om/ratis/OMTransactionInfo.java | 6 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 18 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 37 +++
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 18 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 111 ++++++-
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 7 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 235 ++++++++++++++
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 29 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 2 +-
.../om/request/bucket/OMBucketCreateRequest.java | 4 +-
.../request/bucket/acl/OMBucketAddAclRequest.java | 8 +
.../bucket/acl/OMBucketRemoveAclRequest.java | 9 +
.../request/bucket/acl/OMBucketSetAclRequest.java | 9 +
.../om/request/file/OMDirectoryCreateRequest.java | 26 +-
.../request/file/OMDirectoryCreateRequestV1.java | 27 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 1 +
.../om/request/file/OMFileCreateRequestV1.java | 5 +-
.../ozone/om/request/file/OMFileRequest.java | 31 --
.../ozone/om/request/key/OMKeyCreateRequest.java | 1 +
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 62 ++--
.../om/request/key/acl/OMKeyAddAclRequest.java | 9 +
.../om/request/key/acl/OMKeyRemoveAclRequest.java | 9 +
.../om/request/key/acl/OMKeySetAclRequest.java | 9 +
.../S3InitiateMultipartUploadRequest.java | 3 +-
.../om/request/volume/OMVolumeCreateRequest.java | 3 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 6 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 6 +-
.../request/volume/acl/OMVolumeAddAclRequest.java | 9 +
.../volume/acl/OMVolumeRemoveAclRequest.java | 9 +
.../request/volume/acl/OMVolumeSetAclRequest.java | 9 +
.../om/response/file/OMFileCreateResponse.java | 5 +
.../ozone/om/response/key/OMKeyCreateResponse.java | 3 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 9 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 75 ++++-
.../request/volume/TestOMVolumeCreateRequest.java | 5 +-
.../ozone/om/response/TestCleanupTableInfo.java | 328 ++++++++++++++++++-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 26 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 28 +-
hadoop-ozone/pom.xml | 50 +--
.../ozone/recon/codegen/ReconSqlDbConfig.java | 2 +-
.../recon/schema/ReconTaskSchemaDefinition.java | 2 +-
hadoop-ozone/recon/pom.xml | 2 +-
.../hadoop/ozone/s3/AWSSignatureProcessor.java | 43 ++-
.../hadoop/ozone/s3/OzoneClientProducer.java | 61 +++-
.../apache/hadoop/ozone/s3/SignatureProcessor.java | 2 +
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 5 +
.../hadoop/ozone/s3/TestOzoneClientProducer.java | 25 +-
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 5 +
.../hadoop/ozone/s3/endpoint/TestRootList.java | 5 +
.../hadoop/ozone/freon/DatanodeBlockPutter.java | 7 +-
.../java/org/apache/hadoop/ozone/shell/Shell.java | 24 +-
pom.xml | 6 +-
172 files changed, 3997 insertions(+), 1726 deletions(-)
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCacheMetrics.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestSimpleContainerDownloader.java
create mode 100644 hadoop-hdds/docs/content/concept/Containers.zh.md
create mode 100644 hadoop-hdds/docs/content/feature/HA.zh.md
create mode 100644 hadoop-hdds/docs/content/feature/Recon.zh.md
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashPolicyOzone.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
