This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git
omit eedd983155 HDDS-7438. Add a createStreamKey method to OzoneBucket.
(#3914)
omit 88e1d1ebb2 HDDS-7431. [Ozone-Streaming] Disable data steam by default.
(#3900)
omit 653a120236 HDDS-6955. [Ozone-streaming] Add explicit stream flag in
ozone shell (#3559)
omit 29a8c157fd HDDS-6867. [Ozone-Streaming] PutKeyHandler should not use
streaming to put EC key. (#3516)
omit a5980ce1a4 HDDS-6842. [Ozone-Streaming] Reduce the number of watch
requests in StreamCommitWatcher. (#3492)
omit 7805efb012 HDDS-6592. [Ozone-Streaming] Fix
ContainerStateMachine#applyTransaction assert error (#3315)
omit 4a5b68370b HDDS-5666. Add option to createKey via streaming api in
Freon (#2574)
omit f2d574ce3e HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at
the end of the stream. (#3229)
omit 99b414df25 HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput.
(#3195)
omit 8e6be787c9 HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters.
(#3207)
omit 8bbe943d23 HDDS-6388. [Ozone-Streaming] Streaming write support both
pipeline model and star model (#3145)
omit b2ec47d5d3 HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
omit 3dde9e27b9 HDDS-6229. [Ozone-Streaming] Data Channel abstraction on
datanode (#3023)
omit cca5e3b020 HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE
(#3060)
omit 434d1fc57d HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support
FlushDelay. (#3002)
omit 1a31d6beef HDDS-6298. Add XceiverServerRatis stream config (#3070)
omit 693eea7ae6 HDDS-6138.[Ozone-Streaming] Define a limit on the size of
the retry bufferList. (#2946)
omit b3ec54827f HDDS-6281. Update ratis version to 2.3.0-94db58b-SNAPSHOT
version (#3059)
omit 33c0260ab5 HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
omit 25c3495e58 HDDS-6139. [Ozone-Streaming] Fix incorrect computation of
totalAckDataLength. (#2978)
omit ac981a7722 HDDS-6130. [Ozone-Streaming] When releaseBuffers will get
“Couldn 't find the required future” (#2939)
omit 7e61b1f642 HDDS-6039. Define a minimum packet size during streaming
writes. (#2883)
omit eb441a8e82 HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer
fixed boundary for streaming writes. (#2866)
omit f3a245157a HDDS-5743. [Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
omit 8de50af361 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
omit 8cbadbcaf6 HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
omit 3cf08778de HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
omit 7cbb3fa89c HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
omit 11fb9bf708 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
omit 5b73b5148f HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
omit b1f06e8480 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
omit 279168014c HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
omit b2133a5bad HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
omit 81b9f3d7c1 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
omit ae462a6747 HDDS-5599. [Ozone-Streaming]drop BufferPool and
ChunkBuffer to avoid buffer copying (#2557)
omit 0952195d73 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
omit 0568725ac9 HDDS-5480. [Ozone-Streaming] Client and server should
support stream setup. (#2452)
omit 01beaf8f43 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
omit 5626566ba7 HDDS-5452. Add link method to ContainerStateMachine for
Ratis streaming (#2422)
omit e15e987a41 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add 2f1117539c HDDS-7284. JVM crash for rocksdb for read/write after close
(#3801)
add 74aef20c9e HDDS-7182. Add property to control RocksDB max open files
(#3843)
add b9a47f6bbf HDDS-7253. Fix exception when '/' in key name (#3774)
add 340f3a7876 HDDS-7381. Cleanup of VolumeManagerImpl (#3873)
add dfc13a0506 HDDS-7258. Cleanup the allocated but uncommitted blocks
(#3778)
add 5b7f448a53 HDDS-7121. Support namespace summaries (du, dist & counts)
for legacy FS buckets (#3746)
add 462f32dd69 HDDS-7396. Force close non-RATIS containers in
ReplicationManager (#3877)
add 561788ebe7 Revert "HDDS-7253. Fix exception when '/' in key name
(#3774)"
add 9449747ff7 HDDS-7413. Fix logging while marking container state
unhealthy (#3887)
add a664ccac6a HDDS-7342. Move encryption-related code from
MultipartCryptoKeyInputStream to OzoneCryptoInputStream (#3852)
add b5ecea61de HDDS-7349. Flaky integration test have memory leak for
RatisDropwizardExports (#3858)
add 0652ba433b HDDS-7422. Bump woodstox-core from 5.0.3 to 5.4.0 (#3886)
add 557d7f84ff HDDS-7402. Adapt CommandQueue to track the count of each
queued command type (#3891)
add 3fb0dfb464 HDDS-7407. EC: Block allocation should not be stripped
across the EC group (#3882)
add 901bcf2dcc HDDS-7424. Bump jetty to 9.4.49.v20220914 (#3894)
add a84baced7e HDDS-7220. SCM should use sub-ca certificate for token
signature without HA enabled (#3752)
add 880f87cd4e HDDS-7370. Add pending commands in SCM to Datanode command
count (#3867)
add f6cad7c55f HDDS-7328. Improve Deletion of FSO Paths (#3844)
add dd67faa0b0 HDDS-7406. Remove unused the exception & improve debug log
in KeyDeletingService (#3880)
add 3a52215bb7 HDDS-1157. TestOzoneContainerWithTLS is failing with
SSLHandshakeException (#3888)
add f0d41c5fcb HDDS-7316. Print stacktrace to identify the location of
RocksObject leaks. (#3827)
add bd8a1610b3 HDDS-7421. Respect OZONE_LOGLEVEL and OZONE_ROOT_LOGGER for
CLI commands (#3892)
add 30d1a4e359 HDDS-7420. Bump Spring framework from 5.2.20 to 5.3.23
(#3902)
add cc574c6c5f HDDS-7432. Move command summary into Commands object in
CommandQueue (#3901)
add 7394f2c2ae HDDS-7361. Add general metrics for queues in Datanode
(#3863)
add c187a8ff65 HDDS-7384. EC: ReplicationManager - implement deleting
container handler (#3881)
add 3294d28cc9 HDDS-7231. Integrate the GetKeyInfo API to key read flows
(#3800)
add 416c575685 HDDS-7146. Recon: Add button to trigger OM DB Sync
(Backend) (#3821)
add f98376b743 HDDS-7244. Fix multiple reports queued up from same DN and
using up heap (#3783)
add 9c6d6a9da5 HDDS-7290. provide a config to increase the list batch size
in OzoneFileSystem (#3853)
add 2e1de7b95f HDDS-7363. Changes to Ozone CLI to validate if the jars in
classpath files are present on an install (#3864)
add 03595deac6 HDDS-7429. Remove Tag from SCMMetadataStoreMetrics (#3897)
add c71e1da783 HDDS-7320. Enable Container SchemaV3 by default. (#3841)
add c7f1c7c9ee HDDS-7447. Leak of TableIterator in
DirectoryDeletingService. (#3922)
add 8dded6ce7e HDDS-6926. Add support for shaded protobufs used by
hadoop-client/spark. (#3915)
add a77cc1da90 HDDS-7435. ContainerOperationClient cleanup (#3910)
add 074d225f2a HDDS-7025. Add cache metrics in OM. (#3865)
add 891e9f57a9 HDDS-7090. EC: delete empty closed EC container (#3660)
add d004f69b98 HDDS-6757. Test "size.leaving.source.max" limit in
ContainerBalancer (#3904)
add b316e73095 HDDS-7423. DB Scanner should allow iteration from specific
key (#3893)
add 5dd389b7cb HDDS-5866. Discrepancy in Trash directory in ofs vs o3fs.
(#3906)
add 6fed1c9434 HDDS-7324. S3G: Avoid calling getS3VolumeContext by using
GetGeyInfo API. (#3911)
add 2037968a56 HDDS-7419. Integrate the GetKeyInfo API to OFS (#3912)
add 3c0ec81687 HDDS-7362. Bring back the mechanism to share LRU cache
between RocksDB in datanodes (#3921)
add 5453cd6aec HDDS-3419. Throw exception with correct code when available
data nodes are not sufficient (#3847)
add 4ec46bc102 HDDS-7460. Bump snakeyaml from 1.32 to 1.33 (#3935)
new d4bf89af1b HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new 92a8081d73 HDDS-5452. Add link method to ContainerStateMachine for
Ratis streaming (#2422)
new 28d30359e4 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new c69955ed74 HDDS-5480. [Ozone-Streaming] Client and server should
support stream setup. (#2452)
new e6be97e0c6 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new db89077fe4 HDDS-5599. [Ozone-Streaming]drop BufferPool and
ChunkBuffer to avoid buffer copying (#2557)
new 8853c8a72b HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new 72bc75006b HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new 8124a1cb41 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new a6968d57f2 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
new 39587086ef HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
new 483eae8883 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
new e4a0f00325 HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
new 029055b113 HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
new 7290ff9827 HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
new 2054fe4dad HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
new d81ca2ab63 HDDS-5743. [Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
new 9d5ccb3067 HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer
fixed boundary for streaming writes. (#2866)
new 28fdecd856 HDDS-6039. Define a minimum packet size during streaming
writes. (#2883)
new 58da69f0aa HDDS-6130. [Ozone-Streaming] When releaseBuffers will get
“Couldn 't find the required future” (#2939)
new 156cd46ad6 HDDS-6139. [Ozone-Streaming] Fix incorrect computation of
totalAckDataLength. (#2978)
new ddf9f84df0 HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
new dd626842f7 HDDS-6281. Update ratis version to 2.3.0-94db58b-SNAPSHOT
version (#3059)
new 340f9594e8 HDDS-6138.[Ozone-Streaming] Define a limit on the size of
the retry bufferList. (#2946)
new 20f1682834 HDDS-6298. Add XceiverServerRatis stream config (#3070)
new 9c6904ddfd HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support
FlushDelay. (#3002)
new fce0dfad3a HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE
(#3060)
new 4ce474c12a HDDS-6229. [Ozone-Streaming] Data Channel abstraction on
datanode (#3023)
new 328c389b0b HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
new 1f097e1c46 HDDS-6388. [Ozone-Streaming] Streaming write support both
pipeline model and star model (#3145)
new b42829cfc3 HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters.
(#3207)
new 06bc9689ca HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput.
(#3195)
new d0407c1ea6 HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at
the end of the stream. (#3229)
new d1b7a92221 HDDS-5666. Add option to createKey via streaming api in
Freon (#2574)
new a7eb40e6fd HDDS-6592. [Ozone-Streaming] Fix
ContainerStateMachine#applyTransaction assert error (#3315)
new 317053dbcc HDDS-6842. [Ozone-Streaming] Reduce the number of watch
requests in StreamCommitWatcher. (#3492)
new b3a6cfccc0 HDDS-6867. [Ozone-Streaming] PutKeyHandler should not use
streaming to put EC key. (#3516)
new 6319ce8ad6 HDDS-6955. [Ozone-streaming] Add explicit stream flag in
ozone shell (#3559)
new 53af9086bb HDDS-7431. [Ozone-Streaming] Disable data steam by default.
(#3900)
new cfdf9cd171 HDDS-7438. [Ozone-Streaming] Add a createStreamKey method
to OzoneBucket. (#3914)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (eedd983155)
\
N -- N -- N refs/heads/HDDS-4454 (cfdf9cd171)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 40 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/hdds/scm/storage/BlockInputStream.java | 45 +-
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 19 +-
.../hdds/scm/storage/TestBlockInputStream.java | 133 ++--
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 3 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 19 +
.../apache/hadoop/ozone/OzoneManagerVersion.java | 2 +
.../common/src/main/resources/ozone-default.xml | 34 +-
hadoop-hdds/container-service/pom.xml | 4 +
.../apache/hadoop/ozone/HddsDatanodeService.java | 2 +-
.../common/statemachine/DatanodeConfiguration.java | 32 +-
.../common/statemachine/DatanodeQueueMetrics.java | 179 ++++++
.../common/statemachine/DatanodeStateMachine.java | 11 +
.../common/statemachine/StateContext.java | 23 +
.../container/common/utils/DatanodeStoreCache.java | 29 +-
.../common/utils/db/DatanodeDBProfile.java | 16 +-
.../container/keyvalue/KeyValueContainer.java | 5 +-
.../container/keyvalue/helpers/BlockUtils.java | 2 +-
.../metadata/DatanodeSchemaThreeDBDefinition.java | 4 +
.../container/common/TestDatanodeStoreCache.java | 2 +-
.../TestSchemaOneBackwardsCompatibility.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 8 +
.../container/common/volume/TestHddsVolume.java | 10 +-
.../container/keyvalue/TestKeyValueContainer.java | 51 +-
.../TestKeyValueContainerMarkUnhealthy.java | 2 +
.../container/keyvalue/TestKeyValueHandler.java | 10 +-
.../keyvalue/impl/AbstractTestChunkManager.java | 5 +-
hadoop-hdds/framework/pom.xml | 5 +
.../hadoop/hdds/scm/metadata/SCMMetadataStore.java | 4 +-
.../hadoop/hdds/server/events/EventExecutor.java | 7 +
.../FixedThreadPoolWithAffinityExecutor.java | 164 +++--
.../hdds/server/http/RatisDropwizardExports.java | 19 +-
.../hadoop/hdds/utils/TableCacheMetrics.java | 92 +++
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 6 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 213 ++++++-
.../org/apache/hadoop/hdds/utils/db/Table.java | 12 +-
.../apache/hadoop/hdds/utils/db/TableConfig.java | 4 +-
.../apache/hadoop/hdds/utils/db/TypedTable.java | 8 +-
.../hadoop/hdds/utils/db/cache/CacheStats.java | 32 +-
.../hdds/utils/db/cache/CacheStatsRecorder.java} | 47 +-
.../hadoop/hdds/utils/db/cache/FullTableCache.java | 16 +-
.../hdds/utils/db/cache/PartialTableCache.java | 13 +-
.../hadoop/hdds/utils/db/cache/TableCache.java | 6 +
.../db/managed/ManagedColumnFamilyOptions.java | 13 +
.../hdds/utils/db/managed/ManagedObject.java | 22 +-
.../utils/db/managed/ManagedRocksObjectUtils.java | 23 +-
.../hadoop/hdds/server/events/TestEventQueue.java | 24 +-
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 31 +
.../hadoop/hdds/utils/db/cache/TestTableCache.java | 44 +-
.../java/org/apache/hadoop/hdds/scm/ScmUtils.java | 33 +
.../scm/block/DeletedBlockLogStateManager.java | 3 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 2 +-
.../container/replication/ReplicationManager.java | 22 +
.../health/ClosingContainerHandler.java | 5 +-
.../health/DeletingContainerHandler.java | 94 +++
.../replication/health/EmptyContainerHandler.java | 19 +-
.../hdds/scm/metadata/SCMMetadataStoreImpl.java | 3 +-
.../hdds/scm/metadata/SCMMetadataStoreMetrics.java | 11 -
.../apache/hadoop/hdds/scm/node/CommandQueue.java | 66 +-
.../hdds/scm/node/CommandQueueReportHandler.java | 3 +-
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 29 +-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 19 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 20 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 9 +-
.../scm/pipeline/WritableECContainerProvider.java | 11 +-
.../hdds/scm/server/ContainerReportQueue.java | 378 ++++++++++++
.../hdds/scm/server/SCMClientProtocolServer.java | 4 +-
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 58 +-
.../hdds/scm/server/StorageContainerManager.java | 80 ++-
.../hadoop/hdds/scm/container/MockNodeManager.java | 16 +-
.../hdds/scm/container/SimpleMockNodeManager.java | 15 +-
.../balancer/TestContainerBalancerTask.java | 41 ++
.../health/TestClosingContainerHandler.java | 103 ++--
.../health/TestDeletingContainerHandler.java | 238 ++++++++
.../health/TestEmptyContainerHandler.java | 20 +-
.../scm/metadata/TestSCMMetadataStoreImpl.java | 10 -
.../hadoop/hdds/scm/node/TestCommandQueue.java | 101 +++
.../scm/node/TestCommandQueueReportHandler.java | 30 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 61 +-
.../pipeline/TestWritableECContainerProvider.java | 5 +-
.../testutils/ReplicationNodeManagerMock.java | 15 +-
.../hdds/scm/cli/ContainerOperationClient.java | 169 +-----
.../apache/hadoop/ozone/client/ObjectStore.java | 13 -
.../org/apache/hadoop/ozone/client/OzoneKey.java | 8 +
.../client/io/MultipartCryptoKeyInputStream.java | 135 +----
.../ozone/client/io/OzoneCryptoInputStream.java | 152 ++++-
.../ozone/client/protocol/ClientProtocol.java | 21 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 204 +++++--
.../hadoop/ozone/client/MockOmTransport.java | 14 +
.../hadoop/ozone/om/helpers/BucketLayout.java | 13 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 106 +++-
.../ozone/om/protocol/OzoneManagerProtocol.java | 4 +
hadoop-ozone/dev-support/intellij/log4j.properties | 2 +
.../dist/dev-support/bin/dist-layout-stitching | 1 -
hadoop-ozone/dist/pom.xml | 4 +
.../{ozone-csi => ozone-legacy-bucket}/.env | 0
.../{ozonesecure => ozone-legacy-bucket}/README.md | 11 +-
.../docker-compose.yaml | 0
.../{ozone => ozone-legacy-bucket}/docker-config | 2 +
.../{failing1 => ozone-legacy-bucket}/test.sh | 7 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 11 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 7 +-
.../smoketest/compatibility/dn-one-rocksdb.robot | 2 +-
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 3 +-
...n-fso-nssummary.robot => recon-nssummary.robot} | 12 +-
.../src/shell/conf/ozone-shell-log4j.properties | 34 --
hadoop-ozone/dist/src/shell/ozone/ozone | 24 +-
.../dist/src/shell/ozone/ozone-functions.sh | 87 +++
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 107 ++--
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 6 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 57 +-
.../TestRatisPipelineCreateAndDestroy.java | 9 +-
.../hadoop/ozone/TestStorageContainerManager.java | 104 ++++
.../ozone/TestStorageContainerManagerHelper.java | 19 +-
.../ozone/client/CertificateClientTestImpl.java | 11 +-
.../ozone/client/rpc/read/TestKeyInputStream.java | 4 +-
.../metrics/TestDatanodeQueueMetrics.java | 111 ++++
.../ozoneimpl/TestOzoneContainerWithTLS.java | 4 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 36 +-
.../org/apache/hadoop/ozone/om/TestLDBCli.java | 6 +
.../ozone/om/TestObjectStoreWithLegacyFS.java | 4 +-
.../ozone/om/TestOmContainerLocationCache.java | 674 +++++++++++++++++++++
.../ozone/om/TestOzoneManagerHAKeyDeletion.java | 10 +-
.../hadoop/ozone/shell/TestNSSummaryAdmin.java | 6 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 19 +-
.../src/main/proto/OmClientProtocol.proto | 1 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 2 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 26 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 12 +-
.../apache/hadoop/ozone/om/OzoneConfigUtil.java | 35 ++
.../org/apache/hadoop/ozone/om/OzoneManager.java | 17 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 105 +++-
.../org/apache/hadoop/ozone/om/VolumeManager.java | 2 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 59 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 19 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 23 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 24 +
.../ozone/om/response/key/OMKeyCommitResponse.java | 6 +
.../ozone/om/service/DirectoryDeletingService.java | 196 +++---
.../ozone/om/service/KeyDeletingService.java | 27 +-
.../om/request/key/TestOMKeyCommitRequest.java | 96 ++-
.../ozone/om/service/TestKeyDeletingService.java | 136 ++++-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 1 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 21 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 1 +
.../fs/ozone/BasicRootedOzoneFileSystem.java | 21 +-
.../apache/hadoop/fs/ozone/OzoneClientUtils.java | 22 +
.../pom.xml | 83 ++-
hadoop-ozone/pom.xml | 8 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 4 +-
...atusService.java => TriggerDBSyncEndpoint.java} | 35 +-
.../ozone/recon/api/handlers/BucketHandler.java | 30 +-
.../recon/api/handlers/DirectoryEntityHandler.java | 20 +-
.../recon/api/handlers/LegacyBucketHandler.java | 325 ++++++++++
.../hadoop/ozone/recon/api/types/NSSummary.java | 4 +-
.../scm/ReconStorageContainerManagerFacade.java | 24 +-
.../recon/spi/OzoneManagerServiceProvider.java | 6 +
.../spi/impl/OzoneManagerServiceProviderImpl.java | 153 +++--
.../impl/ReconContainerMetadataManagerImpl.java | 34 +-
.../hadoop/ozone/recon/tasks/NSSummaryTask.java | 232 +++----
...yTask.java => NSSummaryTaskDbEventHandler.java} | 60 +-
.../ozone/recon/tasks/NSSummaryTaskWithFSO.java | 42 +-
.../ozone/recon/tasks/NSSummaryTaskWithLegacy.java | 307 ++++++++++
.../ozone/recon/OMMetadataManagerTestUtils.java | 24 +-
.../recon/api/TestNSSummaryEndpointWithFSO.java | 10 +-
...O.java => TestNSSummaryEndpointWithLegacy.java} | 297 +++++----
.../ozone/recon/api/TestTriggerDBSyncEndpoint.java | 143 +++++
.../ozone/recon/api/filters/TestAdminFilter.java | 2 +
.../ozone/recon/tasks/TestNSSummaryTask.java | 492 +++++++++++++++
.../recon/tasks/TestNSSummaryTaskWithFSO.java | 13 +-
...thFSO.java => TestNSSummaryTaskWithLegacy.java} | 328 +++++++---
.../ozone/recon/tasks/TestTableCountTask.java | 2 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 10 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 13 +
.../ozone/s3/endpoint/TestPermissionCheck.java | 3 +-
.../ozone/admin/nssummary/DiskUsageSubCommand.java | 7 +-
.../admin/nssummary/FileSizeDistSubCommand.java | 7 +-
.../ozone/admin/nssummary/NSSummaryAdmin.java | 57 ++
.../ozone/admin/nssummary/NSSummaryCLIUtils.java | 9 +-
.../admin/nssummary/QuotaUsageSubCommand.java | 7 +-
.../ozone/admin/nssummary/SummarySubCommand.java | 7 +-
.../org/apache/hadoop/ozone/debug/DBScanner.java | 33 +
pom.xml | 6 +-
182 files changed, 7014 insertions(+), 1889 deletions(-)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeQueueMetrics.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/TableCacheMetrics.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeStorageReport.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheStats.java
(59%)
copy
hadoop-hdds/{common/src/test/java/org/apache/hadoop/hdds/fs/MockSpaceUsagePersistence.java
=>
framework/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheStatsRecorder.java}
(54%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/DeletingContainerHandler.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/ContainerReportQueue.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/health/TestDeletingContainerHandler.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestCommandQueue.java
copy hadoop-ozone/dist/src/main/compose/{ozone-csi =>
ozone-legacy-bucket}/.env (100%)
copy hadoop-ozone/dist/src/main/compose/{ozonesecure =>
ozone-legacy-bucket}/README.md (65%)
copy hadoop-ozone/dist/src/main/compose/{ozone =>
ozone-legacy-bucket}/docker-compose.yaml (100%)
copy hadoop-ozone/dist/src/main/compose/{ozone =>
ozone-legacy-bucket}/docker-config (95%)
copy hadoop-ozone/dist/src/main/compose/{failing1 =>
ozone-legacy-bucket}/test.sh (90%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/smoketest/recon/{recon-fso-nssummary.robot
=> recon-nssummary.robot} (92%)
delete mode 100644
hadoop-ozone/dist/src/shell/conf/ozone-shell-log4j.properties
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/metrics/TestDatanodeQueueMetrics.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmContainerLocationCache.java
copy hadoop-ozone/{ozonefs-hadoop3 => ozonefs-hadoop3-client}/pom.xml (53%)
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{TaskStatusService.java
=> TriggerDBSyncEndpoint.java} (63%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/{NSSummaryTask.java
=> NSSummaryTaskDbEventHandler.java} (79%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java
copy
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/{TestNSSummaryEndpointWithFSO.java
=> TestNSSummaryEndpointWithLegacy.java} (88%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTriggerDBSyncEndpoint.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTask.java
copy
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/{TestNSSummaryTaskWithFSO.java
=> TestNSSummaryTaskWithLegacy.java} (68%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
