This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a change to branch
dependabot-maven-org.cyclonedx-cyclonedx-maven-plugin-2.9.3
in repository https://gitbox.apache.org/repos/asf/ozone.git
from a8fd12cceb6 Bump org.cyclonedx:cyclonedx-maven-plugin from 2.9.2 to
2.9.3
add 036745a326d HDDS-15962. Remove leader readiness check on the bootstrap
flow. (#10862)
add 54facbe255e HDDS-16056. non-varargs call of varargs method with
inexact argument type (#10914)
add 951ed5cb51b HDDS-16086. installCheckpoint sets OM transaction info
inconsistent with the checkpoint index (#10943)
add 3b8c05e635e HDDS-13217. Test that snapshot checkpoint content is
preserved after defrag (#10951)
add 80ffb62a841 HDDS-16066. replicas after re-replication should span at
least 2 racks (#10944)
add cb6f12408ec HDDS-15852. Allow CopyObject to the same key when the
metadata directive is REPLACE (#10752).
add fce67f62f35 HDDS-16096. Bump Ranger to 2.9.0 (#10957)
add c8fa8dafd5c HDDS-16101. Extract bucket lifecycle operations into
dedicated handler (#10969)
add af3e34377b0 HDDS-16076. Separate test class for
testInstallSnapshotDuringBootstrapping. (#10931)
add 0a9bd0d999a HDDS-16067. Intermittent timeout in
testNodeWithOpenPipelineCanBeDecommissionedAndRecommissioned (#10939)
add 5425ee3f86c HDDS-15781. Adding Spankind information to ozone sh
commands (#10695)
add 0110ff57df4 HDDS-15396. Grafana dashboard for disk balancer (#10609).
add bd0f23b49ba HDDS-16119. Fix DatanodeStorageMetrics metrics-system
deadlock and mini-cluster source leak (#10988)
add 5be6afae7af HDDS-15389. Top-level delete entry retention utilities for
snapshot diff. (#10750)
add 5e99e226bb0 HDDS-16081. TestOMRatisSnapshots fails due to leader
applied index taken before flush (#10959)
add 09a7d615dec HDDS-15982. Fix UnsupportedOperationException and add
idempotent deletion in RangerClientMultiTenantAccessController (#10872)
add 07f069b72fc HDDS-16015. Make listStatusLight skip SCM pipeline
refresh. (#10894)
add 6384c643fb3 HDDS-14742. Clean up orphan files from a terminated
compaction. (#10821)
add b4a4227cf4f HDDS-14876. Make snapshot diff progress more granular
(#10901)
add 776ce13a798 HDDS-16113. Add regression tests for snapshot
defragmentation cleanup (#10978)
add a1ce3b9d900 HDDS-15774. Make SCM Ratis roles string parsing IPv6-safe
(#10831)
add 5dfed2ce426 HDDS-15775. Support IPv6 literal authorities in OzoneFS
URI parsing (#10933)
add 752e96c2fb3 HDDS-15776. Make S3 Gateway VirtualHostStyleFilter
IPv6-safe (#10809)
add 0f812bcbb30 HDDS-15542. Closed-container reconciliation advances BCSID
past a hole, masking missing chunks (#10496)
add 9065c84364d HDDS-15761. Create test for OMMXBean (#10960)
add 59351d77aba HDDS-16114. Avoid misleading Ozone mount warning in
acceptance test results (#10979)
add 1db063b63ce HDDS-16072. Implement GetObjectAttributes: ETag,
ObjectSize, StorageClass, and ObjectParts count (#10930).
add 291633510ce HDDS-15961. Resolve linked bucket source properties
consistently (#10859)
add a7f5cbc832c HDDS-16117. Failed FSO multipart complete leaks bucket
namespace quota (#10984)
add 2fbc910aa29 HDDS-11620. Log SCM finalization completion (#10963)
add 6a4513ddaec HDDS-15826. Recon: add cycle guard to NSSummary /du tree
walks (#10723)
add 5672fc71c5d HDDS-8082. Check for unnecessary newlines in config
defaults (#10840)
add 7f68eb6f36d HDDS-16125. Copy bucket snapshot in quota repair response
to fix cache aliasing (#11004)
add bc21296f297 HDDS-16118. Release snapshot DB handles before
KeyDeletingService submits OM requests (#10987)
add 7693b1979f8 HDDS-15895. Make ServerNotLeaderException suggested leader
flow IPv6-safe (#10807)
add 0c349a3f541 HDDS-15356. Make multi-buffer chunk checksum
allocation-free (#10350)
add ce13cdece18 HDDS-15911. Recon AI Assistant: add conversation memory
for follow-up questions. (#10808)
add 749da668af1 HDDS-16158. Fix sendDeleteKeysRequestAndClearList
Authorization and FSO Trash Handling (#10996)
add fb2cc733688 HDDS-16169. Add old updateContainerState to handle upgrade
path. (#11012)
add 711476cb0c4 HDDS-16089. Improve MPU table evaluation efficiency in
KeyLifeCycleService (#10949)
add 50978e3ad7c HDDS-15422. Stream read seek should not close stream
(#10415)
add c0c3241ff57 HDDS-16115. Reject invalid lifecycle rule Status values
(#10980)
add 1a4486a271b HDDS-16088. Reject lifecycle
AbortIncompleteMultipartUpload rules whose daysAfterInitiation exceeds the MPU
cleanup threshold (#10982)
add 32b6c97db1f HDDS-16071. Add S3 Bucket Lifecycle CRUD integration tests
to AbstractS3SDKV2Tests (#10986)
add a552e060433 HDDS-16183. Avoid the per-group list copy in
OMKeyRequest.sumBlockLengths (#11026)
add db3fff45c51 HDDS-16198. Refactor the admin permission check for
LifecycleService. (#11034)
add 6a903990741 Improve listSnapshotDiffJobs
add b58885e7baf Adding OM tests.
add 54881878681 Add missing check for GetContainerChecksumInfo.
add afa7f0e667c HDDS-13242. getTrashRoots should not be bounded by
ozone.fs.listing.page.size.max (#11002)
add 3af0d141eb1 HDDS-16197. Remove all bucket tagging outstanding TODOs
(#11036)
add a9b67b11ccb HDDS-16150. Fix error code for invalid S3 Lifecycle
Expiration Days from InvalidRequest to InvalidArgument (#11003)
add 2c02f432d30 HDDS-12208. Give `ozone debug replicas verify` options for
OM container location cache behavior. (#11015)
add d9cdd2539a2 HDDS-8096. Redirect S3G Jersey logs to log file (#11005)
add d5ce83cbd0d Improve OmMetadataReader
add 888cefdffbc HDDS-16207. Streaming read hangs until read timeout on
out-of-range ReadBlock offsets (#11040)
add 208b031453c HDDS-15140. Piggyback S3 derived key during key creation
(#10652)
add a46d391d915 HDDS-16150. Fix error code in
testS3LifecycleConfigurationCreationFailed (#11045)
add 8f4595c082e HDDS-16193. Clean up ScmClient (#11035)
add b88a3752322 HDDS-16200. Flaky container balancer robot test (#11033)
add 4dfc1cb7f81 HDDS-16085. OM CPU Load metrics on grafana uses wrong
port. (#10942)
add 5d4972e025e HDDS-16209. KeyLifecycleService leaks inFlight entry when
a task runs while shouldRun() is false (#11050)
add 9ac58d9f16c HDDS-16142. Fix double-decrement of deleted-block summary
on Ratis timeout (#11043)
add a0e14ceb083 HDDS-16211. Avoid the throwaway list copy in
OMAllocateBlockRequest quota check (#11057)
add 5b4de65e33c HDDS-16204. Avoid the per-group list copy in
ContainerToKeyMapping.getContainers (#11047)
add 3fa99ad4718 HDDS-16227. Aborted LifecycleActionTask should not mark
scan state completed (#11069)
add 48d1c2f398c HDDS-16202. Avoid the list copy in
ContainerEndpoint.getBlocks (#11038)
add 8e55a2f27b2 HDDS-15390. Utility to generate dependency ordered
snapdiff report. (#10778)
add 9e7ba249136 HDDS-16130. Remove Pair usage from Safe-mode APIs and
callers. (#10993)
add a65fe53821f HDDS-16192. Refactor o.a.h.hdds.scm.protocol (#11070)
add 758883e7bc8 HDDS-16219. Add ExportJob and expand ExportFileManager
(#11052)
add c50d89a4550 HDDS-16162. Fail lifecycle task when KeyLifecycleService
cannot prepare trash root (#11021)
add c886eef8292 HDDS-16243. Remove redundant RPC metadata from
ScmBlockLocationProtocol (#11076)
add b264f854f1b Improve Ozone Manager test coverage
add f61e35dd390 HDDS-16249. Fix no-op assertThat in
TestOzoneClientRetriesOnExceptions (#11081)
add 8a81f47f661 HDDS-16250. Improve RocksDB read performance by reducing
unnecessary iterator positioning (#11084)
add 9f7dcd839e5 HDDS-16244. Remove redundant RPC metadata from
SCMSecurityProtocol (#11079)
add 78d5650c313 HDDS-16201. Rename getLocationList to createLocationList
(#11077)
add d8940a9b280 HDDS-16223. Remove racy pre-condition assertion in
TestRefreshVolumeUsageHandler (#11063)
add 9eb3d1b1294 HDDS-16212. Avoid the per-group list copy in Recon
ContainerKeyMapperHelper (#11058)
add 9143e0e78a7 HDDS-16215. Avoid the per-group list copy in
OmMetadataManagerImpl.getBlocksForKeyDeletetBlocksForKeyDelete (#11089)
add 6e57e9f05ea HDDS-16205. Avoid the duplicate list copy in
S3MultipartUploadCompleteRequest.getMultipartDataSize (#11064)
add 45dc8861454 HDDS-1159. Fix flaky
testGetMatchingContainerMultipleThreads (#11048)
add c379bcc4478 HDDS-16133. Make TestReconTasks EMPTY_MISSING check
deterministic (#11059)
add 8efa8ff66da HDDS-16252. Avoid the per-group list copy in
OmKeyInfo.verifyAndGetKeyLocations (#11091)
add f59c1b822fd HDDS-16260. Bump zizmorcore/zizmor-action to 0.6.2 (#10977)
add 79464b60d76 HDDS-16213. Avoid the per-group list copy in
KeyManagerImpl.getPendingDeletionKeys (#11082)
add 91f20fba0ab HDDS-16261. Bump
apache/infrastructure-actions/allowlist-check to
69afc125e535c4c41e7f1b7470f583087e0f344b (#11087)
add 1fe18604720 HDDS-10478. Store full block data as ByteString in
MockDatanodeStorage (#11093)
add 66c9e5e9793 HDDS-15657. excludeContainersDueToFailure resets every
iteration, repeatedly failing containers keep being retried (#10892)
add fea5ee0c7f3 HDDS-16094. Bump ratis-thirdparty to 1.1.0 (#10956)
add 402e459b87e HDDS-16171. Tool to download OM metadata with snapshots.
(#11017)
add e80d51efdb7 HDDS-16170. Intermittent failure in
TestSnapshotDiffManager#testLoadJobsOnStartUp (#11014)
add 31b4e332b3e HDDS-16155. Fix OM checkpoint installation deadlock
between state machine pause and double-buffer flush (#10995)
add bb4a35077c2 HDDS-15333. Expose volume quota and namespace usage
metrics in OM Prometheus endpoint. (#11011)
add f944e929719 HDDS-16018. Use configured DNS-to-switch mapping directly
in SCM and OM (#10911)
add e6dfa38dbdf HDDS-15889. Show failed MoveResult breakdown in ozone
admin containerbalancer status -v (#10938)
add e62ad5b37af HDDS-16248. Refactor
PutBucketLifecycleConfigurationUnmarshaller to extend MessageUnmarshaller
(#11075)
add 4fe6e1f24c0 HDDS-16175. Do not change usedNamespace on hsync re-commit
(#11092)
add 949e43b3bea HDDS-16216. Avoid flatten-copy in
KeyInputStream.getBlockLocationInfo (#11099)
add 77f7df311df HDDS-11355. Remove Flaky tag from
testMultiBlockWritesWithIntermittentDnFailures (#11101)
add 21cdfda6081 HDDS-9377. Replace MutableStat with lock-free
ConcurrentMutableStat in OMLockMetrics and PerformanceMetrics (#11085)
add 81c61b5b44d HDDS-16224. Wait for replica with container and block
before read in TestFailureHandlingByClient (#11065)
add 7f6ea8b1ac7 HDDS-16134. Fix flaky
testPipelineExclusionWithPipelineFailure (#11062)
add 8dad48e88f2 HDDS-16285. Bump netty to 4.1.137 (#11110)
add 524660937a5 HDDS-12865. Handle automatic container cache refresh for
replica verification. (#11044)
add 4a85b7354e7 HDDS-16283. Fix flaky
testMoveToTrashAbortTaskWhenTrashRootPrepareFails (#11109)
add 9cbf98ce138 HDDS-16291. Bump commons-codec to 1.22.1 (#10971)
add d0a0620e393 HDDS-16293. Bump actions/stale to 11.0.0 (#10976)
add a16612c2b93 HDDS-16296. Bump common-custom-user-data-maven-extension
to 2.4.0 (#10972)
add da5697a19bf HDDS-16294. Bump zstd-jni to 1.5.7-12 (#10973)
add 2e0a51f7a95 Merge remote-tracking branch 'origin/master' into
dependabot-maven-org.cyclonedx-cyclonedx-maven-plugin-2.9.3
No new revisions were added by this update.
Summary of changes:
.github/workflows/asf-allowlist-check.yaml | 2 +-
.github/workflows/close-stale-prs.yaml | 2 +-
.github/workflows/zizmor.yml | 2 +-
.mvn/extensions.xml | 2 +-
hadoop-hdds/client/pom.xml | 4 +
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 5 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 3 +-
.../hdds/scm/storage/MultipartInputStream.java | 4 +-
.../hdds/scm/storage/StreamBlockInputStream.java | 221 ++--
.../scm/storage/TestStreamBlockInputStream.java | 114 ++
.../java/org/apache/hadoop/hdds/HddsUtils.java | 64 ++
.../hdds/ratis/ServerNotLeaderException.java | 5 +-
.../apache/hadoop/hdds/scm/net/HostAndPort.java | 3 +-
.../hadoop/hdds/tracing/GrpcServerInterceptor.java | 4 +-
.../{SkipTracing.java => InternalSpanKind.java} | 6 +-
.../apache/hadoop/hdds/tracing/TraceAllMethod.java | 9 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 56 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 186 ++--
.../apache/hadoop/ozone/common/ChecksumCache.java | 117 ++-
.../hadoop/ozone/common/utils/BufferUtils.java | 23 +
.../hadoop/ozone/util/ConcurrentMutableStat.java | 134 +++
.../hadoop/ozone/util/PerformanceMetrics.java | 5 +-
.../common/src/main/resources/ozone-default.xml | 16 +-
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 62 ++
...TestServerNotLeaderExceptionMessageParsing.java | 22 +
.../hadoop/hdds/scm/net/TestHostAndPort.java | 6 +
.../hdds/security/x509/CertificateTestUtils.java | 64 ++
.../hadoop/hdds/tracing/TestTraceAllMethod.java | 20 +
.../hdds/tracing/TestTracingAnnotations.java | 88 ++
.../hadoop/hdds/tracing/TestTracingUtil.java | 77 +-
.../apache/hadoop/ozone/common/TestChecksum.java | 17 +
.../hadoop/ozone/common/TestChecksumCache.java | 73 +-
.../ozone/common/TestChecksumMultiBuffer.java | 97 ++
.../hadoop/ozone/common/utils/TestBufferUtils.java | 81 ++
.../ozone/util/ConcurrentMutableStatBenchmark.java | 286 +++++
.../ozone/util/TestConcurrentMutableStat.java | 172 +++
.../checksum/ContainerChecksumTreeManager.java | 12 +-
.../container/checksum/ContainerDiffReport.java | 13 +
.../common/transport/server/XceiverServerGrpc.java | 4 +-
.../transport/server/ratis/XceiverServerRatis.java | 4 +-
.../common/volume/DatanodeStorageMetrics.java | 39 +-
.../container/common/volume/MutableVolumeSet.java | 37 +
.../ozone/container/keyvalue/KeyValueHandler.java | 188 +++-
.../container/keyvalue/impl/BlockManagerImpl.java | 10 +
.../keyvalue/interfaces/BlockManager.java | 10 +
.../common/volume/TestDatanodeStorageMetrics.java | 52 +-
.../container/common/volume/TestVolumeSet.java | 27 +
.../container/keyvalue/TestKeyValueHandler.java | 176 ++++
.../TestReconcileChunksPerBlockHoleBcsId.java | 522 ++++++++++
.../keyvalue/impl/TestFilePerBlockStrategy.java | 10 +-
.../docs/content/feature/ContainerBalancer.md | 4 +
.../docs/content/tools/debug/DebugReplicas.md | 6 +
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 10 -
.../apache/hadoop/hdds/scm/client/ScmClient.java | 74 +-
.../scm/protocol/ScmBlockLocationProtocol.java | 10 -
.../protocol/StorageContainerLocationProtocol.java | 16 +-
...inerLocationProtocolClientSideTranslatorPB.java | 29 +-
.../scm/proxy/SCMFailoverProxyProviderBase.java | 95 +-
.../server/OzoneProtocolMessageDispatcher.java | 3 +-
.../hdds/utils/db/RDBStoreAbstractIterator.java | 18 +
.../hdds/utils/db/RDBStoreByteArrayIterator.java | 1 -
.../hdds/utils/db/RDBStoreCodecBufferIterator.java | 1 -
.../hdds/protocolPB/TestSCMSecurityProtocolPB.java | 45 +-
.../protocolPB/TestScmBlockLocationProtocolPB.java | 42 +-
.../TestStorageContainerLocationProtocolPB.java | 42 +-
.../TestSCMFailoverProxyProviderRefreshWired.java | 167 ++-
.../hdds/upgrade/TestHDDSLayoutVersionManager.java | 8 +-
.../utils/db/TestRDBStoreByteArrayIterator.java | 29 +-
.../utils/db/TestRDBStoreCodecBufferIterator.java | 34 +-
.../src/main/proto/ScmAdminProtocol.proto | 14 +
.../interface-admin/src/main/resources/proto.lock | 58 ++
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 48 +-
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 55 +-
.../SCMDeletedBlockTransactionStatusManager.java | 22 +
.../hdds/scm/container/ContainerStateManager.java | 6 +
.../scm/container/ContainerStateManagerImpl.java | 30 +
.../ContainerBalancerSelectionCriteria.java | 47 +-
.../container/balancer/ContainerBalancerTask.java | 78 +-
.../ContainerBalancerTaskIterationStatusInfo.java | 42 +
.../balancer/ContainerMoveFailureDetail.java | 62 ++
.../balancer/ContainerMoveFailureTracker.java | 93 ++
.../scm/container/balancer/ContainerMoveInfo.java | 13 +-
.../scm/container/export/ExportFileManager.java | 120 ++-
.../hdds/scm/container/export/ExportJob.java | 41 +-
.../hdds/scm/container/export/ExportScope.java | 8 +-
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 2 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 31 +-
.../ha/invoker/ContainerStateManagerInvoker.java | 25 +-
...inerLocationProtocolServerSideTranslatorPB.java | 24 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 14 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 13 +-
.../hdds/scm/server/StorageContainerManager.java | 35 +-
.../scm/server/upgrade/SCMUpgradeFinalizer.java | 1 +
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 168 +++
.../TestContainerBalancerMoveFailureBreakdown.java | 278 +++++
.../balancer/TestContainerBalancerTask.java | 111 +-
.../container/export/TestExportFileManager.java | 103 +-
.../hdds/scm/container/export/TestExportJob.java | 67 ++
.../scm/ha/TestInterSCMGrpcProtocolService.java | 4 +-
.../hadoop/hdds/scm/ha/TestSCMRatisServerImpl.java | 62 ++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 25 +-
.../scm/server/TestStorageContainerManager.java | 56 +
.../hdds/scm/upgrade/TestScmFinalization.java | 50 +-
.../scm/cli/ContainerBalancerStatusSubcommand.java | 43 +
.../hdds/scm/cli/ContainerOperationClient.java | 127 +--
.../hdds/scm/cli/SafeModeCheckSubcommand.java | 66 +-
.../cli/datanode/DiskBalancerReportSubcommand.java | 2 +-
.../cli/datanode/DiskBalancerStatusSubcommand.java | 2 +-
.../admin/scm/GetScmRatisRolesSubcommand.java | 51 +-
.../datanode/TestContainerBalancerSubCommand.java | 109 ++
.../ozone/scm/TestGetScmRatisRolesSubcommand.java | 65 ++
.../ozone/debug/om/ContainerToKeyMapping.java | 15 +-
.../debug/replicas/BlockExistenceVerifier.java | 6 +
.../debug/replicas/BlockVerificationResult.java | 15 +
.../ozone/debug/replicas/ChecksumVerifier.java | 18 +
.../ozone/debug/replicas/ReplicasVerify.java | 122 ++-
.../apache/hadoop/ozone/fsck/ContainerMapper.java | 2 +-
.../ozone/debug/replicas/TestReplicasVerify.java | 275 +++++
.../hadoop/ozone/repair/om/DownloadOMDB.java | 205 ++++
.../apache/hadoop/ozone/repair/om/OMRepair.java | 3 +-
hadoop-ozone/cli-shell/pom.xml | 4 +
.../java/org/apache/hadoop/ozone/shell/Shell.java | 3 +-
.../ozone/shell/snapshot/SnapshotDiffHandler.java | 7 +
hadoop-ozone/client/pom.xml | 9 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 73 ++
.../hadoop/ozone/client/OzoneClientFactory.java | 5 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 20 +-
.../ozone/client/io/OzoneDataStreamOutput.java | 8 +
.../hadoop/ozone/client/io/OzoneOutputStream.java | 10 +
.../ozone/client/protocol/ClientProtocol.java | 76 ++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 112 +-
.../hadoop/ozone/client/MockDatanodeStorage.java | 15 +-
.../hadoop/ozone/client/TestOzoneECClient.java | 19 +-
hadoop-ozone/common/pom.xml | 4 +
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 6 +-
.../hadoop/ozone/om}/AWSV4AuthValidator.java | 35 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 11 +-
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 13 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 16 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 2 +-
.../hadoop/ozone/om/helpers/OpenKeySession.java | 10 +
.../hadoop/ozone/om/helpers/SnapshotDiffJob.java | 3 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 21 +-
.../ozone/snapshot/SnapshotDiffResponse.java | 28 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 19 +
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 21 +
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 10 +-
.../om/helpers/TestOmKeyLocationInfoGroup.java | 2 +-
.../ozone/snapshot/TestSnapshotDiffResponse.java | 40 +-
.../grafana/dashboards/Ozone - Disk Balancer.json | 1096 ++++++++++++++++++++
.../grafana/dashboards/Ozone - JVM Metrics.json | 2 +-
.../dashboards/Ozone - ReadKey Metrics.json | 6 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 5 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 5 +-
.../src/main/smoketest/balancer/testBalancer.robot | 2 +-
hadoop-ozone/dist/src/main/smoketest/testlib.sh | 3 +-
.../apache/ozone/fs/http/server/HttpFSServer.java | 2 +-
.../ozone/recon/TestReconAndAdminContainerCLI.java | 2 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 10 +-
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 4 +-
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 292 +++++-
.../dev-support/findbugsExcludeFile.xml | 12 -
.../hadoop/conf/ConfigurationFieldsTests.java | 1 -
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 44 +
.../hadoop/hdds/scm/TestContainerOperations.java | 16 +-
.../hadoop/hdds/scm/TestRackAwarePlacement.java | 58 +-
.../hdds/scm/TestStorageContainerManager.java | 4 +-
.../TestContainerStateManagerIntegration.java | 11 +-
.../scm/node/TestDecommissionAndMaintenance.java | 7 +-
.../org/apache/hadoop/ozone/OzoneTestUtils.java | 2 +-
.../org/apache/hadoop/ozone/TestBlockTokens.java | 6 +-
.../ozone/TestContainerBalancerOperations.java | 4 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 58 +-
.../ozone/client/rpc/OzoneRpcClientTests.java | 2 +-
.../client/rpc/TestDeleteWithInAdequateDN.java | 2 +-
.../client/rpc/TestFailureHandlingByClient.java | 58 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 4 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 2 -
.../client/rpc/TestOzoneAtRestEncryption.java | 2 +-
.../rpc/TestOzoneClientMultipartUploadWithFSO.java | 58 ++
.../rpc/TestOzoneClientRetriesOnExceptions.java | 2 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 2 +-
.../ozone/client/rpc/read/TestKeyInputStream.java | 2 +-
.../rpc/read/TestStreamBlockInputStream.java | 56 +-
.../ozone/client/rpc/read/TestStreamRead.java | 3 +-
.../ozone/container/TestContainerReplication.java | 2 +-
.../container/TestContainerReportHandling.java | 2 +-
.../commandhandler/TestBlockDeletion.java | 6 +-
.../commandhandler/TestFinalizeBlock.java | 8 +-
.../TestRefreshVolumeUsageHandler.java | 4 -
.../dn/TestDatanodeStorageMetricsIntegration.java | 9 +-
.../org/apache/hadoop/ozone/om/OMMXBeanTests.java | 88 ++
.../ozone/om/TestGetDBUpdatesAuthorization.java | 137 +++
.../ozone/om/TestKeyLookupAclNormalization.java | 206 ++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 40 +-
.../ozone/om/TestOMAdminRpcAuthorization.java | 143 +++
.../TestOMInstallSnapshotDuringBootstrapping.java | 207 ++++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 221 ++--
.../hadoop/ozone/om/TestOmBlockVersioning.java | 12 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 4 +-
.../hadoop/ozone/om/snapshot/OmSnapshotTests.java | 26 +
.../TestOmSnapshotCheckpointDbContent.java | 511 +++++++++
.../ozone/repair/om/TestDownloadOMDBTool.java | 260 +++++
.../ozone/shell/TestOzoneDebugReplicasVerify.java | 6 +-
.../java/org/apache/ozone/test/NonHATests.java | 9 +
.../src/main/proto/OmClientProtocol.proto | 23 +-
.../interface-client/src/main/resources/proto.lock | 18 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 3 +
.../apache/hadoop/ozone/om/lock/OMLockMetrics.java | 29 +-
.../ozone/om/helpers/TestOmKeyInfoCodec.java | 2 +-
.../om/helpers/TestRepeatedOmKeyInfoCodec.java | 4 +-
.../RangerClientMultiTenantAccessController.java | 98 +-
...lientMultiTenantAccessControllerMockClient.java | 153 +++
hadoop-ozone/ozone-manager/pom.xml | 4 -
.../hadoop/ozone/om/BucketUtilizationMetrics.java | 8 +-
.../org/apache/hadoop/ozone/om/KeyManager.java | 28 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 57 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 6 +-
.../om/OMDBCheckpointServletInodeBasedXfer.java | 6 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 11 +-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 79 +-
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 32 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 98 +-
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 34 +-
.../hadoop/ozone/om/VolumeUtilizationMetrics.java | 105 ++
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 2 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 97 +-
.../om/request/file/OMRecoverLeaseRequest.java | 4 +-
.../om/request/key/OMAllocateBlockRequest.java | 2 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 6 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 5 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 40 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 8 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 16 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 2 +-
.../OMLifecycleConfigurationSetRequest.java | 50 +-
.../lifecycle/OMLifecycleSaveScanStateRequest.java | 2 +-
.../OMLifecycleSetServiceStatusRequest.java | 2 +-
.../S3MultipartUploadCompleteRequest.java | 16 +-
.../om/request/volume/OMQuotaRepairRequest.java | 9 +-
.../ozone/om/service/KeyDeletingService.java | 15 +-
.../ozone/om/service/KeyLifecycleService.java | 230 ++--
.../ozone/om/snapshot/SnapshotDiffManager.java | 168 ++-
.../hadoop/ozone/om/snapshot/SnapshotUtils.java | 4 +-
.../om/snapshot/diff/SnapDiffDependencyEntry.java | 164 +++
.../om/snapshot/diff/SnapDiffDependencyGraph.java | 738 +++++++++++++
.../diff/delta/CompositeDeltaDiffComputer.java | 3 +
.../ozone/om/snapshot/diff}/package-info.java | 4 +-
.../om/snapshot/filter/ReclaimableFilter.java | 15 +-
.../protocolPB/OMAdminProtocolServerSideImpl.java | 12 +
.../protocolPB/OzoneManagerRequestHandler.java | 8 +-
.../OzoneDelegationTokenSecretManager.java | 1 +
.../hadoop/ozone/security/S3SecurityUtil.java | 2 +-
.../hadoop/ozone/om/TestBucketManagerImpl.java | 132 +++
.../ozone/om/TestBucketUtilizationMetrics.java | 12 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 33 +
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 55 +-
...DBCheckpointServletInodeBasedXferNonLeader.java | 86 +-
.../hadoop/ozone/om/TestOMMetadataReader.java | 2 +-
.../hadoop/ozone/om/TestTrashOzoneFileSystem.java | 133 +++
.../ozone/om/TestVolumeUtilizationMetrics.java | 114 ++
.../om/ratis/TestOzoneManagerStateMachine.java | 89 ++
.../om/request/file/TestOMFileCreateRequest.java | 4 +-
.../om/request/key/TestOMAllocateBlockRequest.java | 4 +-
.../om/request/key/TestOMKeyCommitRequest.java | 48 +-
.../om/request/key/TestOMKeyCreateRequest.java | 107 +-
.../om/request/key/TestOMKeysDeleteRequest.java | 53 +
.../TestOMLifecycleConfigurationRequest.java | 45 +-
.../TestOMLifecycleConfigurationSetRequest.java | 63 ++
.../TestOMLifecycleSaveScanStateRequest.java | 8 +-
...=> TestOMLifecycleSetServiceStatusRequest.java} | 61 +-
.../TestS3MultipartUploadCommitPartRequest.java | 8 +-
.../request/volume/TestOMQuotaRepairRequest.java | 95 ++
.../ozone/om/service/TestKeyDeletingService.java | 245 +++++
.../ozone/om/service/TestKeyLifecycleService.java | 375 ++++++-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 179 +++-
.../om/snapshot/TestSnapshotDiffManagerMXBean.java | 4 +-
.../snapshot/defrag/TestSnapshotDefragService.java | 38 +-
.../snapshot/diff/TestSnapDiffDependencyGraph.java | 489 +++++++++
.../protocolPB/TestOzoneManagerRequestHandler.java | 65 ++
.../ozone/security/TestAWSV4AuthValidator.java | 1 +
.../security/acl/TestOzoneNativeAuthorizer.java | 2 +-
.../hadoop/ozone/security/acl/TestParentAcl.java | 2 +-
.../hadoop/ozone/security/acl/TestVolumeOwner.java | 2 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 29 +-
.../fs/ozone/LeaseRecoveryClientDNHandler.java | 4 +-
.../hadoop/fs/ozone/TestBasicOzoneFileSystems.java | 49 +
.../recon/schema/ContainerSchemaDefinition.java | 12 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 10 +
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 55 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 17 +-
.../ozone/recon/api/handlers/EntityHandler.java | 122 ++-
.../ozone/recon/chatbot/ChatbotConfigKeys.java | 15 +
.../recon/chatbot/agent/ChatHistoryBuilder.java | 175 ++++
.../ozone/recon/chatbot/agent/ChatbotAgent.java | 90 +-
.../ozone/recon/chatbot/api/ChatbotEndpoint.java | 63 +-
.../ozone/recon/metrics/NSSummaryMetrics.java | 91 ++
.../recon/spi/ReconNamespaceSummaryManager.java | 6 +
.../impl/ReconContainerMetadataManagerImpl.java | 2 +-
.../spi/impl/ReconNamespaceSummaryManagerImpl.java | 21 +-
.../impl/StorageContainerServiceProviderImpl.java | 3 +-
.../recon/tasks/ContainerKeyMapperHelper.java | 86 +-
.../upgrade/InitialConstraintUpgradeAction.java | 10 +-
.../UnhealthyContainerReplicaMismatchAction.java | 10 +-
.../recon-tool-selection-prompt-preamble.txt | 7 +
.../ozone-recon-web/src/v2/hooks/useChat.hook.tsx | 28 +-
.../ozone-recon-web/src/v2/types/chatbot.types.ts | 8 +
.../apache/hadoop/ozone/recon/TestReconUtils.java | 40 +
.../api/handlers/TestEntityHandlerCycleGuard.java | 196 ++++
.../chatbot/agent/TestChatHistoryBuilder.java | 289 ++++++
.../agent/TestChatbotAgentExecutionPolicy.java | 62 ++
.../recon/chatbot/api/TestChatbotEndpoint.java | 14 +-
.../ozone/recon/metrics/TestNSSummaryMetrics.java | 60 ++
hadoop-ozone/s3gateway/pom.xml | 9 +
.../org/apache/hadoop/ozone/audit/S3GAction.java | 3 +-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 8 +
.../hadoop/ozone/s3/VirtualHostStyleFilter.java | 37 +-
.../ozone/s3/endpoint/BucketCrudHandler.java | 126 +--
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 1 +
...rudHandler.java => BucketLifecycleHandler.java} | 119 +--
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 19 +
.../s3/endpoint/GetObjectAttributesResponse.java | 206 ++++
.../ozone/s3/endpoint/ObjectAttributesHandler.java | 173 +++
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 38 +-
...utBucketLifecycleConfigurationUnmarshaller.java | 56 +-
.../s3/endpoint/S3LifecycleConfiguration.java | 16 +-
.../hadoop/ozone/s3/metrics/S3GatewayMetrics.java | 22 +
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 4 +
.../org/apache/hadoop/ozone/s3/TestGateway.java | 81 ++
.../ozone/s3/TestVirtualHostStyleFilter.java | 58 ++
.../ozone/s3/endpoint/EndpointTestUtils.java | 14 +
.../ozone/s3/endpoint/TestObjectAttributesGet.java | 176 ++++
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 42 +
...utBucketLifecycleConfigurationUnmarshaller.java | 86 ++
.../endpoint/TestS3LifecycleConfigurationPut.java | 110 +-
pom.xml | 16 +-
336 files changed, 16822 insertions(+), 2219 deletions(-)
copy
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/{SkipTracing.java
=> InternalSpanKind.java} (86%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ConcurrentMutableStat.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/tracing/TestTracingAnnotations.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumMultiBuffer.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/utils/TestBufferUtils.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/util/ConcurrentMutableStatBenchmark.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/util/TestConcurrentMutableStat.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestReconcileChunksPerBlockHoleBcsId.java
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/protocolPB/TestSCMSecurityProtocolPB.java
(50%)
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocolPB/TestScmBlockLocationProtocolPB.java
(53%)
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocolPB/TestStorageContainerLocationProtocolPB.java
(51%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerMoveFailureDetail.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerMoveFailureTracker.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerMoveFailureBreakdown.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/export/TestExportJob.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestStorageContainerManager.java
create mode 100644
hadoop-ozone/cli-debug/src/test/java/org/apache/hadoop/ozone/debug/replicas/TestReplicasVerify.java
create mode 100644
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java
rename
hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/security =>
common/src/main/java/org/apache/hadoop/ozone/om}/AWSV4AuthValidator.java (78%)
create mode 100644
hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk
Balancer.json
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/OMMXBeanTests.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestGetDBUpdatesAuthorization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyLookupAclNormalization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMAdminRpcAuthorization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMInstallSnapshotDuringBootstrapping.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotCheckpointDbContent.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/repair/om/TestDownloadOMDBTool.java
create mode 100644
hadoop-ozone/multitenancy-ranger/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestRangerClientMultiTenantAccessControllerMockClient.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeUtilizationMetrics.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffDependencyEntry.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffDependencyGraph.java
copy {hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage =>
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff}/package-info.java
(85%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestTrashOzoneFileSystem.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestVolumeUtilizationMetrics.java
copy
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/lifecycle/{TestOMLifecycleSaveScanStateRequest.java
=> TestOMLifecycleSetServiceStatusRequest.java} (55%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/volume/TestOMQuotaRepairRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/TestSnapDiffDependencyGraph.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/chatbot/agent/ChatHistoryBuilder.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/NSSummaryMetrics.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/handlers/TestEntityHandlerCycleGuard.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/chatbot/agent/TestChatHistoryBuilder.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/metrics/TestNSSummaryMetrics.java
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/{BucketCrudHandler.java
=> BucketLifecycleHandler.java} (63%)
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/GetObjectAttributesResponse.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectAttributesHandler.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestGateway.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectAttributesGet.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPutBucketLifecycleConfigurationUnmarshaller.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
