This is an automated email from the ASF dual-hosted git repository.
avijayan pushed a change to branch HDDS-3698-upgrade
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 7a26250 HDDS-4342. Add DataNode state and transitions for a node
going through upgrade. (#1508)
add 8bcd859 HDDS-4420. Missing a container shouldn't log message at ERROR
level (#1554)
add 1f6b0d9 HDDS-4446. Update Jetty version to 9.4.34. (#1566)
add 2ef4cf9 HDDS-4445. Owner info is not passed to authorizer for
BUCKET/KEY crea… (#1563)
add 3b53f9f
HDDS-4436.TestReplicationManager#testUnderReplicatedQuasiClosedContainerWithUnhealthyReplica
fails intermittently. (#1560)
add 50e5765 HDDS-4438. Avoid counting skipped transactions belonging to
open containers. (#1565)
add fd9d0ce HDDS-4435. Fix can not insert the second entry when hive on
ozone (#1559)
add 90190f5 HDDS-4307.Start Trash Emptier in Ozone Manager (#1507)
add a28cda5 HDDS-4363. Add metric to track the number of RocksDB
open/close operations. (#1530)
add fdbd5da HDDS-4417. Simplify Ozone client code with configuration
object (#1542)
add d2b7135 HDDS-4437. Avoid unnecessary builder conversion in setting
volume Quota/Owner request (#1564)
add f3c8900 HDDS-4441. Add metrics for ACL related operations. (#1571)
add 8523775 HDDS-4455. Fix typo in README.md doc (#1578)
add e56d7bc HDDS-4315. Use Epoch to generate unique ObjectIDs (#1480)
add 7a52990 HDDS-4443. Recon: Using Mysql database throws exception and
fails startup (#1570)
add 277a589 HDDS-4408: terminate Datanode when Datanode State Machine
Thread got uncaught exception. (#1533)
add 5e8aaee HDDS-4453. Replicate closed container for random selected
datanodes. (#1574)
add 5ec98e1 HDDS-4083. Create ZH translation of Recon.md in doc (#1575)
add 760c1e8 HDDS-4337. Implement RocksDB options cache for new datanode
DB utilities. (#1544)
add 0e35176 HDDS-4338. Fix the issue that SCM web UI banner shows "HDFS
SCM". (#1583)
add b80d9b0 HDDS-4235. Ozone client FS path validation is not present in
OFS. (#1582)
add 5f43814 HDDS-4184. Add Features menu for Chinese document. (#1547)
add 45e30d6 HDDS-4079. Create ZH translation of Containers.md in doc.
(#1539)
add fcf0a4d HDDS-4080. Create ZH translation of OzoneManager.md in doc.
(#1541)
add 7df5dba HDDS-4081. Create ZH translation of
StorageContainerManager.md in doc. (#1558)
add 585984e HDDS-4441. Add metrics for ACL related operations.(Addendum
for HA). (#1584)
add d203b64 HDDS-4442. Disable the location information of audit logger
to reduce overhead (#1567)
add 787e5b6 HDDS-4458. Fix Max Transaction ID value in OM. (#1585)
add e738172 HDDS-4466. Update url in .asf.yaml to use TLP project (#1592)
add 9062c5b HDDS-4467. Acceptance test fails due to new Hadoop 3 image
(#1594)
add 1cd5863 HDDS-4464. Upgrade httpclient version due to CVE-2020-13956.
(#1590)
add d0aa34c HDDS-4082. Create ZH translation of HA.md in doc. (#1591)
add 04a757a HDDS-4462. Add --frozen-lockfile to pnpm install to prevent
ozone-recon-web/pnpm-lock.yaml from being updated automatically (#1589)
add 4e36d03 HDDS-4367. Configuration for deletion service intervals
should be different for OM, SCM and datanodes (#1573)
add 8b2c2eb HDDS-4399. Safe mode rule for piplelines should only consider
open pipelines. (#1526)
add 40c1944 HDDS-4426. SCM should create transactions using all blocks
received from OM (#1561)
add 0aca5c7 HDDS-4346.Ozone specific Trash Policy (#1535)
add 541ae9f HDDS-4450. Cannot run ozone if HADOOP_HOME points to Hadoop
install (#1572)
add bbeaf65 HDDS-4448. Duplicate refreshPipeline in listStatus (#1569)
add 58ec7f4 HDDS-4427. Avoid ContainerCache in ContainerReader at
Datanode startup (#1549)
add 417082c HDDS-4452. findbugs.sh couldn't be executed after a full
build (#1576)
new 4384789 Merge remote-tracking branch 'upstream/master' into
HDDS-3698-upgrade
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 2 +-
README.md | 2 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 213 +
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 66 +-
.../storage/TestBlockOutputStreamCorrectness.java | 17 +-
.../hdds/scm/storage/TestChunkInputStream.java | 11 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 23 +
.../hdds/scm/storage/ContainerProtocolCalls.java | 4 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 73 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 22 +-
.../common/src/main/resources/ozone-default.xml | 91 +-
.../ratis/TestContainerCommandRequestMessage.java | 13 +-
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 8 +-
.../common/src/test/resources/auditlog.properties | 2 +-
.../hdds/conf/ConfigurationReflectionUtil.java | 14 +
.../hadoop/hdds/conf/ConfigurationTarget.java | 4 +
.../common/statemachine/DatanodeConfiguration.java | 24 +
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../container/common/utils/ContainerCache.java | 52 +-
.../common/utils/ContainerCacheMetrics.java | 114 +
.../container/keyvalue/helpers/BlockUtils.java | 54 +
.../keyvalue/helpers/KeyValueContainerUtil.java | 107 +-
.../container/metadata/AbstractDatanodeStore.java | 65 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 12 +-
.../replication/GrpcReplicationClient.java | 23 +-
.../replication/SimpleContainerDownloader.java | 36 +-
.../main/resources/webapps/hddsDatanode/index.html | 2 +-
.../ozone/container/ContainerTestHelper.java | 6 +-
.../container/common/TestBlockDeletingService.java | 9 +-
.../ozone/container/common/TestContainerCache.java | 8 +
.../common/impl/TestContainerPersistence.java | 53 +-
.../container/keyvalue/TestKeyValueContainer.java | 54 +-
.../container/ozoneimpl/TestContainerReader.java | 13 +-
.../container/ozoneimpl/TestOzoneContainer.java | 2 +
.../replication/TestSimpleContainerDownloader.java | 85 +
hadoop-hdds/docs/content/concept/Containers.zh.md | 47 +
.../docs/content/concept/OzoneManager.zh.md | 92 +-
.../content/concept/StorageContainerManager.zh.md | 69 +-
hadoop-hdds/docs/content/feature/GDPR.md | 2 +-
hadoop-hdds/docs/content/feature/GDPR.zh.md | 5 +-
hadoop-hdds/docs/content/feature/HA.md | 4 +-
hadoop-hdds/docs/content/feature/HA.zh.md | 117 +
hadoop-hdds/docs/content/feature/Quota.zh.md | 3 +
hadoop-hdds/docs/content/feature/Recon.zh.md | 49 +
hadoop-hdds/docs/content/feature/_index.zh.md | 7 +-
hadoop-hdds/docs/content/interface/_index.zh.md | 2 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 351 +-
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 21 +-
hadoop-hdds/pom.xml | 50 +-
.../apache/hadoop/hdds/scm/block/BlockManager.java | 4 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 50 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 29 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 5 +-
.../IncrementalContainerReportHandler.java | 4 +
.../hadoop/hdds/scm/metadata/PipelineCodec.java | 3 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 8 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 42 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 3 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 45 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 85 +-
.../src/main/resources/webapps/scm/index.html | 6 +-
.../hdds/scm/container/TestReplicationManager.java | 12 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 4 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 6 +
.../TestOneReplicaPipelineSafeModeRule.java | 74 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 34 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 135 +-
.../client/io/BlockOutputStreamEntryPool.java | 89 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 115 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 99 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 59 +-
.../org/apache/hadoop/ozone/util/ExitManager.java | 2 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 5 +
hadoop-ozone/datanode/pom.xml | 23 +
hadoop-ozone/dist/pom.xml | 48 +-
.../compose/ozonesecure-om-ha/docker-compose.yaml | 4 +-
.../main/compose/ozonesecure/docker-compose.yaml | 2 +
.../main/smoketest/security/ozone-secure-fs.robot | 17 +
hadoop-ozone/dist/src/shell/ozone/ozone | 11 +-
hadoop-ozone/dist/src/shell/ozone/stop-ozone.sh | 5 +
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 28 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 13 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 85 +-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 6 +
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 49 +-
.../hadoop/ozone/TestStorageContainerManager.java | 67 +-
.../ozone/client/rpc/TestBlockOutputStream.java | 34 +-
.../rpc/TestBlockOutputStreamFlushDelay.java | 22 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 39 +-
...estBlockOutputStreamWithFailuresFlushDelay.java | 34 +-
.../rpc/TestCloseContainerHandlingByClient.java | 35 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 7 +-
.../client/rpc/TestContainerStateMachine.java | 35 +-
.../rpc/TestContainerStateMachineFailures.java | 64 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 6 +-
.../client/rpc/TestDiscardPreallocatedBlocks.java | 35 +-
.../client/rpc/TestFailureHandlingByClient.java | 7 +-
.../ozone/client/rpc/TestKeyInputStream.java | 33 +-
.../rpc/TestOzoneClientRetriesOnException.java | 43 +-
...estOzoneClientRetriesOnExceptionFlushDelay.java | 31 +-
.../ozone/client/rpc/TestOzoneRpcClient.java | 7 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 23 +-
.../client/rpc/TestOzoneRpcClientWithRatis.java | 2 +
.../client/rpc/TestValidateBCSIDOnRestart.java | 41 +-
.../ozone/client/rpc/TestWatchForCommit.java | 46 +-
.../commandhandler/TestBlockDeletion.java | 10 +
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 105 +
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 100 +-
.../src/test/resources/contract/ozone.xml | 5 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 5 +
.../hadoop/ozone/om/ratis/OMTransactionInfo.java | 6 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 18 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 37 +
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 18 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 112 +-
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 7 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 235 +
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 29 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 2 +-
.../om/request/bucket/OMBucketCreateRequest.java | 4 +-
.../request/bucket/acl/OMBucketAddAclRequest.java | 10 +
.../bucket/acl/OMBucketRemoveAclRequest.java | 10 +
.../request/bucket/acl/OMBucketSetAclRequest.java | 9 +
.../om/request/file/OMDirectoryCreateRequest.java | 23 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 1 +
.../ozone/om/request/file/OMFileRequest.java | 30 -
.../ozone/om/request/key/OMKeyCreateRequest.java | 1 +
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 18 +-
.../om/request/key/acl/OMKeyAddAclRequest.java | 9 +
.../om/request/key/acl/OMKeyRemoveAclRequest.java | 9 +
.../om/request/key/acl/OMKeySetAclRequest.java | 9 +
.../S3InitiateMultipartUploadRequest.java | 3 +-
.../om/request/volume/OMVolumeCreateRequest.java | 3 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 6 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 6 +-
.../request/volume/acl/OMVolumeAddAclRequest.java | 9 +
.../volume/acl/OMVolumeRemoveAclRequest.java | 9 +
.../request/volume/acl/OMVolumeSetAclRequest.java | 9 +
...OzoneManagerProtocolServerSideTranslatorPB.java | 9 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 75 +-
.../request/volume/TestOMVolumeCreateRequest.java | 5 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 26 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 28 +-
hadoop-ozone/pom.xml | 50 +-
.../ozone/recon/codegen/ReconSqlDbConfig.java | 2 +-
.../recon/schema/ReconTaskSchemaDefinition.java | 2 +-
hadoop-ozone/recon/pom.xml | 2 +-
.../webapps/recon/ozone-recon-web/pnpm-lock.yaml | 17167 +++++++++++++++++++
.../hadoop/ozone/freon/DatanodeBlockPutter.java | 7 +-
pom.xml | 6 +-
150 files changed, 20592 insertions(+), 1583 deletions(-)
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCacheMetrics.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestSimpleContainerDownloader.java
create mode 100644 hadoop-hdds/docs/content/concept/Containers.zh.md
create mode 100644 hadoop-hdds/docs/content/feature/HA.zh.md
create mode 100644 hadoop-hdds/docs/content/feature/Recon.zh.md
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/TrashPolicyOzone.java
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/pnpm-lock.yaml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
