This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a change to branch
dependabot-maven-com.github.luben-zstd-jni-1.5.7-12
in repository https://gitbox.apache.org/repos/asf/ozone.git
from a536aaf4f6a Bump com.github.luben:zstd-jni from 1.5.7-11 to 1.5.7-12
add 036745a326d HDDS-15962. Remove leader readiness check on the bootstrap
flow. (#10862)
add 54facbe255e HDDS-16056. non-varargs call of varargs method with
inexact argument type (#10914)
add 951ed5cb51b HDDS-16086. installCheckpoint sets OM transaction info
inconsistent with the checkpoint index (#10943)
add 3b8c05e635e HDDS-13217. Test that snapshot checkpoint content is
preserved after defrag (#10951)
add 80ffb62a841 HDDS-16066. replicas after re-replication should span at
least 2 racks (#10944)
add cb6f12408ec HDDS-15852. Allow CopyObject to the same key when the
metadata directive is REPLACE (#10752).
add fce67f62f35 HDDS-16096. Bump Ranger to 2.9.0 (#10957)
add c8fa8dafd5c HDDS-16101. Extract bucket lifecycle operations into
dedicated handler (#10969)
add af3e34377b0 HDDS-16076. Separate test class for
testInstallSnapshotDuringBootstrapping. (#10931)
add 0a9bd0d999a HDDS-16067. Intermittent timeout in
testNodeWithOpenPipelineCanBeDecommissionedAndRecommissioned (#10939)
add 5425ee3f86c HDDS-15781. Adding Spankind information to ozone sh
commands (#10695)
add 0110ff57df4 HDDS-15396. Grafana dashboard for disk balancer (#10609).
add bd0f23b49ba HDDS-16119. Fix DatanodeStorageMetrics metrics-system
deadlock and mini-cluster source leak (#10988)
add 5be6afae7af HDDS-15389. Top-level delete entry retention utilities for
snapshot diff. (#10750)
add 5e99e226bb0 HDDS-16081. TestOMRatisSnapshots fails due to leader
applied index taken before flush (#10959)
add 09a7d615dec HDDS-15982. Fix UnsupportedOperationException and add
idempotent deletion in RangerClientMultiTenantAccessController (#10872)
add 07f069b72fc HDDS-16015. Make listStatusLight skip SCM pipeline
refresh. (#10894)
add 6384c643fb3 HDDS-14742. Clean up orphan files from a terminated
compaction. (#10821)
add b4a4227cf4f HDDS-14876. Make snapshot diff progress more granular
(#10901)
add 776ce13a798 HDDS-16113. Add regression tests for snapshot
defragmentation cleanup (#10978)
add a1ce3b9d900 HDDS-15774. Make SCM Ratis roles string parsing IPv6-safe
(#10831)
add 5dfed2ce426 HDDS-15775. Support IPv6 literal authorities in OzoneFS
URI parsing (#10933)
add 752e96c2fb3 HDDS-15776. Make S3 Gateway VirtualHostStyleFilter
IPv6-safe (#10809)
add 0f812bcbb30 HDDS-15542. Closed-container reconciliation advances BCSID
past a hole, masking missing chunks (#10496)
add 9065c84364d HDDS-15761. Create test for OMMXBean (#10960)
add 59351d77aba HDDS-16114. Avoid misleading Ozone mount warning in
acceptance test results (#10979)
add 1db063b63ce HDDS-16072. Implement GetObjectAttributes: ETag,
ObjectSize, StorageClass, and ObjectParts count (#10930).
add 291633510ce HDDS-15961. Resolve linked bucket source properties
consistently (#10859)
add a7f5cbc832c HDDS-16117. Failed FSO multipart complete leaks bucket
namespace quota (#10984)
add 2fbc910aa29 HDDS-11620. Log SCM finalization completion (#10963)
add 6a4513ddaec HDDS-15826. Recon: add cycle guard to NSSummary /du tree
walks (#10723)
add 5672fc71c5d HDDS-8082. Check for unnecessary newlines in config
defaults (#10840)
add 7f68eb6f36d HDDS-16125. Copy bucket snapshot in quota repair response
to fix cache aliasing (#11004)
add bc21296f297 HDDS-16118. Release snapshot DB handles before
KeyDeletingService submits OM requests (#10987)
add 7693b1979f8 HDDS-15895. Make ServerNotLeaderException suggested leader
flow IPv6-safe (#10807)
add 0c349a3f541 HDDS-15356. Make multi-buffer chunk checksum
allocation-free (#10350)
add ce13cdece18 HDDS-15911. Recon AI Assistant: add conversation memory
for follow-up questions. (#10808)
add 749da668af1 HDDS-16158. Fix sendDeleteKeysRequestAndClearList
Authorization and FSO Trash Handling (#10996)
add fb2cc733688 HDDS-16169. Add old updateContainerState to handle upgrade
path. (#11012)
add 711476cb0c4 HDDS-16089. Improve MPU table evaluation efficiency in
KeyLifeCycleService (#10949)
add 50978e3ad7c HDDS-15422. Stream read seek should not close stream
(#10415)
add c0c3241ff57 HDDS-16115. Reject invalid lifecycle rule Status values
(#10980)
add 1a4486a271b HDDS-16088. Reject lifecycle
AbortIncompleteMultipartUpload rules whose daysAfterInitiation exceeds the MPU
cleanup threshold (#10982)
add 32b6c97db1f HDDS-16071. Add S3 Bucket Lifecycle CRUD integration tests
to AbstractS3SDKV2Tests (#10986)
add a552e060433 HDDS-16183. Avoid the per-group list copy in
OMKeyRequest.sumBlockLengths (#11026)
add db3fff45c51 HDDS-16198. Refactor the admin permission check for
LifecycleService. (#11034)
add 6a903990741 Improve listSnapshotDiffJobs
add b58885e7baf Adding OM tests.
add 54881878681 Add missing check for GetContainerChecksumInfo.
add afa7f0e667c HDDS-13242. getTrashRoots should not be bounded by
ozone.fs.listing.page.size.max (#11002)
add 3af0d141eb1 HDDS-16197. Remove all bucket tagging outstanding TODOs
(#11036)
add a9b67b11ccb HDDS-16150. Fix error code for invalid S3 Lifecycle
Expiration Days from InvalidRequest to InvalidArgument (#11003)
add 2c02f432d30 HDDS-12208. Give `ozone debug replicas verify` options for
OM container location cache behavior. (#11015)
add d9cdd2539a2 HDDS-8096. Redirect S3G Jersey logs to log file (#11005)
add d5ce83cbd0d Improve OmMetadataReader
add 888cefdffbc HDDS-16207. Streaming read hangs until read timeout on
out-of-range ReadBlock offsets (#11040)
add 208b031453c HDDS-15140. Piggyback S3 derived key during key creation
(#10652)
add a46d391d915 HDDS-16150. Fix error code in
testS3LifecycleConfigurationCreationFailed (#11045)
add 8f4595c082e HDDS-16193. Clean up ScmClient (#11035)
add b88a3752322 HDDS-16200. Flaky container balancer robot test (#11033)
add 4dfc1cb7f81 HDDS-16085. OM CPU Load metrics on grafana uses wrong
port. (#10942)
add 5d4972e025e HDDS-16209. KeyLifecycleService leaks inFlight entry when
a task runs while shouldRun() is false (#11050)
add 9ac58d9f16c HDDS-16142. Fix double-decrement of deleted-block summary
on Ratis timeout (#11043)
add a0e14ceb083 HDDS-16211. Avoid the throwaway list copy in
OMAllocateBlockRequest quota check (#11057)
add 5b4de65e33c HDDS-16204. Avoid the per-group list copy in
ContainerToKeyMapping.getContainers (#11047)
add 3fa99ad4718 HDDS-16227. Aborted LifecycleActionTask should not mark
scan state completed (#11069)
add 48d1c2f398c HDDS-16202. Avoid the list copy in
ContainerEndpoint.getBlocks (#11038)
add 8e55a2f27b2 HDDS-15390. Utility to generate dependency ordered
snapdiff report. (#10778)
add 9e7ba249136 HDDS-16130. Remove Pair usage from Safe-mode APIs and
callers. (#10993)
add a65fe53821f HDDS-16192. Refactor o.a.h.hdds.scm.protocol (#11070)
add 758883e7bc8 HDDS-16219. Add ExportJob and expand ExportFileManager
(#11052)
add c50d89a4550 HDDS-16162. Fail lifecycle task when KeyLifecycleService
cannot prepare trash root (#11021)
add c886eef8292 HDDS-16243. Remove redundant RPC metadata from
ScmBlockLocationProtocol (#11076)
add b264f854f1b Improve Ozone Manager test coverage
add f61e35dd390 HDDS-16249. Fix no-op assertThat in
TestOzoneClientRetriesOnExceptions (#11081)
add 8a81f47f661 HDDS-16250. Improve RocksDB read performance by reducing
unnecessary iterator positioning (#11084)
add 9f7dcd839e5 HDDS-16244. Remove redundant RPC metadata from
SCMSecurityProtocol (#11079)
add 78d5650c313 HDDS-16201. Rename getLocationList to createLocationList
(#11077)
add d8940a9b280 HDDS-16223. Remove racy pre-condition assertion in
TestRefreshVolumeUsageHandler (#11063)
add 9eb3d1b1294 HDDS-16212. Avoid the per-group list copy in Recon
ContainerKeyMapperHelper (#11058)
add 9143e0e78a7 HDDS-16215. Avoid the per-group list copy in
OmMetadataManagerImpl.getBlocksForKeyDeletetBlocksForKeyDelete (#11089)
add 6e57e9f05ea HDDS-16205. Avoid the duplicate list copy in
S3MultipartUploadCompleteRequest.getMultipartDataSize (#11064)
add 45dc8861454 HDDS-1159. Fix flaky
testGetMatchingContainerMultipleThreads (#11048)
add c379bcc4478 HDDS-16133. Make TestReconTasks EMPTY_MISSING check
deterministic (#11059)
add 8efa8ff66da HDDS-16252. Avoid the per-group list copy in
OmKeyInfo.verifyAndGetKeyLocations (#11091)
add f59c1b822fd HDDS-16260. Bump zizmorcore/zizmor-action to 0.6.2 (#10977)
add 79464b60d76 HDDS-16213. Avoid the per-group list copy in
KeyManagerImpl.getPendingDeletionKeys (#11082)
add 91f20fba0ab HDDS-16261. Bump
apache/infrastructure-actions/allowlist-check to
69afc125e535c4c41e7f1b7470f583087e0f344b (#11087)
add 1fe18604720 HDDS-10478. Store full block data as ByteString in
MockDatanodeStorage (#11093)
add 66c9e5e9793 HDDS-15657. excludeContainersDueToFailure resets every
iteration, repeatedly failing containers keep being retried (#10892)
add fea5ee0c7f3 HDDS-16094. Bump ratis-thirdparty to 1.1.0 (#10956)
add 402e459b87e HDDS-16171. Tool to download OM metadata with snapshots.
(#11017)
add e80d51efdb7 HDDS-16170. Intermittent failure in
TestSnapshotDiffManager#testLoadJobsOnStartUp (#11014)
add 31b4e332b3e HDDS-16155. Fix OM checkpoint installation deadlock
between state machine pause and double-buffer flush (#10995)
add bb4a35077c2 HDDS-15333. Expose volume quota and namespace usage
metrics in OM Prometheus endpoint. (#11011)
add f944e929719 HDDS-16018. Use configured DNS-to-switch mapping directly
in SCM and OM (#10911)
add e6dfa38dbdf HDDS-15889. Show failed MoveResult breakdown in ozone
admin containerbalancer status -v (#10938)
add e62ad5b37af HDDS-16248. Refactor
PutBucketLifecycleConfigurationUnmarshaller to extend MessageUnmarshaller
(#11075)
add 4fe6e1f24c0 HDDS-16175. Do not change usedNamespace on hsync re-commit
(#11092)
add 949e43b3bea HDDS-16216. Avoid flatten-copy in
KeyInputStream.getBlockLocationInfo (#11099)
add 77f7df311df HDDS-11355. Remove Flaky tag from
testMultiBlockWritesWithIntermittentDnFailures (#11101)
add 21cdfda6081 HDDS-9377. Replace MutableStat with lock-free
ConcurrentMutableStat in OMLockMetrics and PerformanceMetrics (#11085)
add 81c61b5b44d HDDS-16224. Wait for replica with container and block
before read in TestFailureHandlingByClient (#11065)
add 7f6ea8b1ac7 HDDS-16134. Fix flaky
testPipelineExclusionWithPipelineFailure (#11062)
add 8dad48e88f2 HDDS-16285. Bump netty to 4.1.137 (#11110)
add 524660937a5 HDDS-12865. Handle automatic container cache refresh for
replica verification. (#11044)
add 4a85b7354e7 HDDS-16283. Fix flaky
testMoveToTrashAbortTaskWhenTrashRootPrepareFails (#11109)
add 9cbf98ce138 HDDS-16291. Bump commons-codec to 1.22.1 (#10971)
add d0a0620e393 HDDS-16293. Bump actions/stale to 11.0.0 (#10976)
add a16612c2b93 HDDS-16296. Bump common-custom-user-data-maven-extension
to 2.4.0 (#10972)
new cc4ecd1e11b Merge remote-tracking branch 'origin/master' into
dependabot-maven-com.github.luben-zstd-jni-1.5.7-12
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/asf-allowlist-check.yaml | 2 +-
.github/workflows/close-stale-prs.yaml | 2 +-
.github/workflows/zizmor.yml | 2 +-
.mvn/extensions.xml | 2 +-
hadoop-hdds/client/pom.xml | 4 +
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 5 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 3 +-
.../hdds/scm/storage/MultipartInputStream.java | 4 +-
.../hdds/scm/storage/StreamBlockInputStream.java | 221 ++--
.../scm/storage/TestStreamBlockInputStream.java | 114 ++
.../java/org/apache/hadoop/hdds/HddsUtils.java | 64 ++
.../hdds/ratis/ServerNotLeaderException.java | 5 +-
.../apache/hadoop/hdds/scm/net/HostAndPort.java | 3 +-
.../hadoop/hdds/tracing/GrpcServerInterceptor.java | 4 +-
.../{SkipTracing.java => InternalSpanKind.java} | 6 +-
.../apache/hadoop/hdds/tracing/TraceAllMethod.java | 9 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 56 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 186 ++--
.../apache/hadoop/ozone/common/ChecksumCache.java | 117 ++-
.../hadoop/ozone/common/utils/BufferUtils.java | 23 +
.../hadoop/ozone/util/ConcurrentMutableStat.java | 134 +++
.../hadoop/ozone/util/PerformanceMetrics.java | 5 +-
.../common/src/main/resources/ozone-default.xml | 16 +-
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 62 ++
...TestServerNotLeaderExceptionMessageParsing.java | 22 +
.../hadoop/hdds/scm/net/TestHostAndPort.java | 6 +
.../hdds/security/x509/CertificateTestUtils.java | 64 ++
.../hadoop/hdds/tracing/TestTraceAllMethod.java | 20 +
.../hdds/tracing/TestTracingAnnotations.java | 88 ++
.../hadoop/hdds/tracing/TestTracingUtil.java | 77 +-
.../apache/hadoop/ozone/common/TestChecksum.java | 17 +
.../hadoop/ozone/common/TestChecksumCache.java | 73 +-
.../ozone/common/TestChecksumMultiBuffer.java | 97 ++
.../hadoop/ozone/common/utils/TestBufferUtils.java | 81 ++
.../ozone/util/ConcurrentMutableStatBenchmark.java | 286 +++++
.../ozone/util/TestConcurrentMutableStat.java | 172 +++
.../checksum/ContainerChecksumTreeManager.java | 12 +-
.../container/checksum/ContainerDiffReport.java | 13 +
.../common/transport/server/XceiverServerGrpc.java | 4 +-
.../transport/server/ratis/XceiverServerRatis.java | 4 +-
.../common/volume/DatanodeStorageMetrics.java | 39 +-
.../container/common/volume/MutableVolumeSet.java | 37 +
.../ozone/container/keyvalue/KeyValueHandler.java | 188 +++-
.../container/keyvalue/impl/BlockManagerImpl.java | 10 +
.../keyvalue/interfaces/BlockManager.java | 10 +
.../common/volume/TestDatanodeStorageMetrics.java | 52 +-
.../container/common/volume/TestVolumeSet.java | 27 +
.../container/keyvalue/TestKeyValueHandler.java | 176 ++++
.../TestReconcileChunksPerBlockHoleBcsId.java | 522 ++++++++++
.../keyvalue/impl/TestFilePerBlockStrategy.java | 10 +-
.../docs/content/feature/ContainerBalancer.md | 4 +
.../docs/content/tools/debug/DebugReplicas.md | 6 +
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 10 -
.../apache/hadoop/hdds/scm/client/ScmClient.java | 74 +-
.../scm/protocol/ScmBlockLocationProtocol.java | 10 -
.../protocol/StorageContainerLocationProtocol.java | 16 +-
...inerLocationProtocolClientSideTranslatorPB.java | 29 +-
.../scm/proxy/SCMFailoverProxyProviderBase.java | 95 +-
.../server/OzoneProtocolMessageDispatcher.java | 3 +-
.../hdds/utils/db/RDBStoreAbstractIterator.java | 18 +
.../hdds/utils/db/RDBStoreByteArrayIterator.java | 1 -
.../hdds/utils/db/RDBStoreCodecBufferIterator.java | 1 -
.../hdds/protocolPB/TestSCMSecurityProtocolPB.java | 45 +-
.../protocolPB/TestScmBlockLocationProtocolPB.java | 42 +-
.../TestStorageContainerLocationProtocolPB.java | 42 +-
.../TestSCMFailoverProxyProviderRefreshWired.java | 167 ++-
.../hdds/upgrade/TestHDDSLayoutVersionManager.java | 8 +-
.../utils/db/TestRDBStoreByteArrayIterator.java | 29 +-
.../utils/db/TestRDBStoreCodecBufferIterator.java | 34 +-
.../src/main/proto/ScmAdminProtocol.proto | 14 +
.../interface-admin/src/main/resources/proto.lock | 58 ++
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 48 +-
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 55 +-
.../SCMDeletedBlockTransactionStatusManager.java | 22 +
.../hdds/scm/container/ContainerStateManager.java | 6 +
.../scm/container/ContainerStateManagerImpl.java | 30 +
.../ContainerBalancerSelectionCriteria.java | 47 +-
.../container/balancer/ContainerBalancerTask.java | 78 +-
.../ContainerBalancerTaskIterationStatusInfo.java | 42 +
.../balancer/ContainerMoveFailureDetail.java | 62 ++
.../balancer/ContainerMoveFailureTracker.java | 93 ++
.../scm/container/balancer/ContainerMoveInfo.java | 13 +-
.../scm/container/export/ExportFileManager.java | 120 ++-
.../hdds/scm/container/export/ExportJob.java | 41 +-
.../hdds/scm/container/export/ExportScope.java | 8 +-
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 2 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 31 +-
.../ha/invoker/ContainerStateManagerInvoker.java | 25 +-
...inerLocationProtocolServerSideTranslatorPB.java | 24 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 14 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 13 +-
.../hdds/scm/server/StorageContainerManager.java | 35 +-
.../scm/server/upgrade/SCMUpgradeFinalizer.java | 1 +
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 168 +++
.../TestContainerBalancerMoveFailureBreakdown.java | 278 +++++
.../balancer/TestContainerBalancerTask.java | 111 +-
.../container/export/TestExportFileManager.java | 103 +-
.../hdds/scm/container/export/TestExportJob.java | 67 ++
.../scm/ha/TestInterSCMGrpcProtocolService.java | 4 +-
.../hadoop/hdds/scm/ha/TestSCMRatisServerImpl.java | 62 ++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 25 +-
.../scm/server/TestStorageContainerManager.java | 56 +
.../hdds/scm/upgrade/TestScmFinalization.java | 50 +-
.../scm/cli/ContainerBalancerStatusSubcommand.java | 43 +
.../hdds/scm/cli/ContainerOperationClient.java | 127 +--
.../hdds/scm/cli/SafeModeCheckSubcommand.java | 66 +-
.../cli/datanode/DiskBalancerReportSubcommand.java | 2 +-
.../cli/datanode/DiskBalancerStatusSubcommand.java | 2 +-
.../admin/scm/GetScmRatisRolesSubcommand.java | 51 +-
.../datanode/TestContainerBalancerSubCommand.java | 109 ++
.../ozone/scm/TestGetScmRatisRolesSubcommand.java | 65 ++
.../ozone/debug/om/ContainerToKeyMapping.java | 15 +-
.../debug/replicas/BlockExistenceVerifier.java | 6 +
.../debug/replicas/BlockVerificationResult.java | 15 +
.../ozone/debug/replicas/ChecksumVerifier.java | 18 +
.../ozone/debug/replicas/ReplicasVerify.java | 122 ++-
.../apache/hadoop/ozone/fsck/ContainerMapper.java | 2 +-
.../ozone/debug/replicas/TestReplicasVerify.java | 275 +++++
.../hadoop/ozone/repair/om/DownloadOMDB.java | 205 ++++
.../apache/hadoop/ozone/repair/om/OMRepair.java | 3 +-
hadoop-ozone/cli-shell/pom.xml | 4 +
.../java/org/apache/hadoop/ozone/shell/Shell.java | 3 +-
.../ozone/shell/snapshot/SnapshotDiffHandler.java | 7 +
hadoop-ozone/client/pom.xml | 9 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 73 ++
.../hadoop/ozone/client/OzoneClientFactory.java | 5 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 20 +-
.../ozone/client/io/OzoneDataStreamOutput.java | 8 +
.../hadoop/ozone/client/io/OzoneOutputStream.java | 10 +
.../ozone/client/protocol/ClientProtocol.java | 76 ++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 112 +-
.../hadoop/ozone/client/MockDatanodeStorage.java | 15 +-
.../hadoop/ozone/client/TestOzoneECClient.java | 19 +-
hadoop-ozone/common/pom.xml | 4 +
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 6 +-
.../hadoop/ozone/om}/AWSV4AuthValidator.java | 35 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 11 +-
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 13 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 16 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 2 +-
.../hadoop/ozone/om/helpers/OpenKeySession.java | 10 +
.../hadoop/ozone/om/helpers/SnapshotDiffJob.java | 3 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 21 +-
.../ozone/snapshot/SnapshotDiffResponse.java | 28 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 19 +
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 21 +
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 10 +-
.../om/helpers/TestOmKeyLocationInfoGroup.java | 2 +-
.../ozone/snapshot/TestSnapshotDiffResponse.java | 40 +-
.../grafana/dashboards/Ozone - Disk Balancer.json | 1096 ++++++++++++++++++++
.../grafana/dashboards/Ozone - JVM Metrics.json | 2 +-
.../dashboards/Ozone - ReadKey Metrics.json | 6 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 5 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 5 +-
.../src/main/smoketest/balancer/testBalancer.robot | 2 +-
hadoop-ozone/dist/src/main/smoketest/testlib.sh | 3 +-
.../apache/ozone/fs/http/server/HttpFSServer.java | 2 +-
.../ozone/recon/TestReconAndAdminContainerCLI.java | 2 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 10 +-
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 4 +-
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 292 +++++-
.../dev-support/findbugsExcludeFile.xml | 12 -
.../hadoop/conf/ConfigurationFieldsTests.java | 1 -
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 44 +
.../hadoop/hdds/scm/TestContainerOperations.java | 16 +-
.../hadoop/hdds/scm/TestRackAwarePlacement.java | 58 +-
.../hdds/scm/TestStorageContainerManager.java | 4 +-
.../TestContainerStateManagerIntegration.java | 11 +-
.../scm/node/TestDecommissionAndMaintenance.java | 7 +-
.../org/apache/hadoop/ozone/OzoneTestUtils.java | 2 +-
.../org/apache/hadoop/ozone/TestBlockTokens.java | 6 +-
.../ozone/TestContainerBalancerOperations.java | 4 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 58 +-
.../ozone/client/rpc/OzoneRpcClientTests.java | 2 +-
.../client/rpc/TestDeleteWithInAdequateDN.java | 2 +-
.../client/rpc/TestFailureHandlingByClient.java | 58 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 4 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 2 -
.../client/rpc/TestOzoneAtRestEncryption.java | 2 +-
.../rpc/TestOzoneClientMultipartUploadWithFSO.java | 58 ++
.../rpc/TestOzoneClientRetriesOnExceptions.java | 2 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 2 +-
.../ozone/client/rpc/read/TestKeyInputStream.java | 2 +-
.../rpc/read/TestStreamBlockInputStream.java | 56 +-
.../ozone/client/rpc/read/TestStreamRead.java | 3 +-
.../ozone/container/TestContainerReplication.java | 2 +-
.../container/TestContainerReportHandling.java | 2 +-
.../commandhandler/TestBlockDeletion.java | 6 +-
.../commandhandler/TestFinalizeBlock.java | 8 +-
.../TestRefreshVolumeUsageHandler.java | 4 -
.../dn/TestDatanodeStorageMetricsIntegration.java | 9 +-
.../org/apache/hadoop/ozone/om/OMMXBeanTests.java | 88 ++
.../ozone/om/TestGetDBUpdatesAuthorization.java | 137 +++
.../ozone/om/TestKeyLookupAclNormalization.java | 206 ++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 40 +-
.../ozone/om/TestOMAdminRpcAuthorization.java | 143 +++
.../TestOMInstallSnapshotDuringBootstrapping.java | 207 ++++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 221 ++--
.../hadoop/ozone/om/TestOmBlockVersioning.java | 12 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 4 +-
.../hadoop/ozone/om/snapshot/OmSnapshotTests.java | 26 +
.../TestOmSnapshotCheckpointDbContent.java | 511 +++++++++
.../ozone/repair/om/TestDownloadOMDBTool.java | 260 +++++
.../ozone/shell/TestOzoneDebugReplicasVerify.java | 6 +-
.../java/org/apache/ozone/test/NonHATests.java | 9 +
.../src/main/proto/OmClientProtocol.proto | 23 +-
.../interface-client/src/main/resources/proto.lock | 18 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 3 +
.../apache/hadoop/ozone/om/lock/OMLockMetrics.java | 29 +-
.../ozone/om/helpers/TestOmKeyInfoCodec.java | 2 +-
.../om/helpers/TestRepeatedOmKeyInfoCodec.java | 4 +-
.../RangerClientMultiTenantAccessController.java | 98 +-
...lientMultiTenantAccessControllerMockClient.java | 153 +++
hadoop-ozone/ozone-manager/pom.xml | 4 -
.../hadoop/ozone/om/BucketUtilizationMetrics.java | 8 +-
.../org/apache/hadoop/ozone/om/KeyManager.java | 28 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 57 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 6 +-
.../om/OMDBCheckpointServletInodeBasedXfer.java | 6 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 11 +-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 79 +-
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 32 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 98 +-
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 34 +-
.../hadoop/ozone/om/VolumeUtilizationMetrics.java | 105 ++
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 2 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 97 +-
.../om/request/file/OMRecoverLeaseRequest.java | 4 +-
.../om/request/key/OMAllocateBlockRequest.java | 2 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 6 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 5 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 40 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 8 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 16 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 2 +-
.../OMLifecycleConfigurationSetRequest.java | 50 +-
.../lifecycle/OMLifecycleSaveScanStateRequest.java | 2 +-
.../OMLifecycleSetServiceStatusRequest.java | 2 +-
.../S3MultipartUploadCompleteRequest.java | 16 +-
.../om/request/volume/OMQuotaRepairRequest.java | 9 +-
.../ozone/om/service/KeyDeletingService.java | 15 +-
.../ozone/om/service/KeyLifecycleService.java | 230 ++--
.../ozone/om/snapshot/SnapshotDiffManager.java | 168 ++-
.../hadoop/ozone/om/snapshot/SnapshotUtils.java | 4 +-
.../om/snapshot/diff/SnapDiffDependencyEntry.java | 164 +++
.../om/snapshot/diff/SnapDiffDependencyGraph.java | 738 +++++++++++++
.../diff/delta/CompositeDeltaDiffComputer.java | 3 +
.../ozone/om/snapshot/diff}/package-info.java | 4 +-
.../om/snapshot/filter/ReclaimableFilter.java | 15 +-
.../protocolPB/OMAdminProtocolServerSideImpl.java | 12 +
.../protocolPB/OzoneManagerRequestHandler.java | 8 +-
.../OzoneDelegationTokenSecretManager.java | 1 +
.../hadoop/ozone/security/S3SecurityUtil.java | 2 +-
.../hadoop/ozone/om/TestBucketManagerImpl.java | 132 +++
.../ozone/om/TestBucketUtilizationMetrics.java | 12 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 33 +
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 55 +-
...DBCheckpointServletInodeBasedXferNonLeader.java | 86 +-
.../hadoop/ozone/om/TestOMMetadataReader.java | 2 +-
.../hadoop/ozone/om/TestTrashOzoneFileSystem.java | 133 +++
.../ozone/om/TestVolumeUtilizationMetrics.java | 114 ++
.../om/ratis/TestOzoneManagerStateMachine.java | 89 ++
.../om/request/file/TestOMFileCreateRequest.java | 4 +-
.../om/request/key/TestOMAllocateBlockRequest.java | 4 +-
.../om/request/key/TestOMKeyCommitRequest.java | 48 +-
.../om/request/key/TestOMKeyCreateRequest.java | 107 +-
.../om/request/key/TestOMKeysDeleteRequest.java | 53 +
.../TestOMLifecycleConfigurationRequest.java | 45 +-
.../TestOMLifecycleConfigurationSetRequest.java | 63 ++
.../TestOMLifecycleSaveScanStateRequest.java | 8 +-
...=> TestOMLifecycleSetServiceStatusRequest.java} | 61 +-
.../TestS3MultipartUploadCommitPartRequest.java | 8 +-
.../request/volume/TestOMQuotaRepairRequest.java | 95 ++
.../ozone/om/service/TestKeyDeletingService.java | 245 +++++
.../ozone/om/service/TestKeyLifecycleService.java | 375 ++++++-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 179 +++-
.../om/snapshot/TestSnapshotDiffManagerMXBean.java | 4 +-
.../snapshot/defrag/TestSnapshotDefragService.java | 38 +-
.../snapshot/diff/TestSnapDiffDependencyGraph.java | 489 +++++++++
.../protocolPB/TestOzoneManagerRequestHandler.java | 65 ++
.../ozone/security/TestAWSV4AuthValidator.java | 1 +
.../security/acl/TestOzoneNativeAuthorizer.java | 2 +-
.../hadoop/ozone/security/acl/TestParentAcl.java | 2 +-
.../hadoop/ozone/security/acl/TestVolumeOwner.java | 2 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 29 +-
.../fs/ozone/LeaseRecoveryClientDNHandler.java | 4 +-
.../hadoop/fs/ozone/TestBasicOzoneFileSystems.java | 49 +
.../recon/schema/ContainerSchemaDefinition.java | 12 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 10 +
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 55 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 17 +-
.../ozone/recon/api/handlers/EntityHandler.java | 122 ++-
.../ozone/recon/chatbot/ChatbotConfigKeys.java | 15 +
.../recon/chatbot/agent/ChatHistoryBuilder.java | 175 ++++
.../ozone/recon/chatbot/agent/ChatbotAgent.java | 90 +-
.../ozone/recon/chatbot/api/ChatbotEndpoint.java | 63 +-
.../ozone/recon/metrics/NSSummaryMetrics.java | 91 ++
.../recon/spi/ReconNamespaceSummaryManager.java | 6 +
.../impl/ReconContainerMetadataManagerImpl.java | 2 +-
.../spi/impl/ReconNamespaceSummaryManagerImpl.java | 21 +-
.../impl/StorageContainerServiceProviderImpl.java | 3 +-
.../recon/tasks/ContainerKeyMapperHelper.java | 86 +-
.../upgrade/InitialConstraintUpgradeAction.java | 10 +-
.../UnhealthyContainerReplicaMismatchAction.java | 10 +-
.../recon-tool-selection-prompt-preamble.txt | 7 +
.../ozone-recon-web/src/v2/hooks/useChat.hook.tsx | 28 +-
.../ozone-recon-web/src/v2/types/chatbot.types.ts | 8 +
.../apache/hadoop/ozone/recon/TestReconUtils.java | 40 +
.../api/handlers/TestEntityHandlerCycleGuard.java | 196 ++++
.../chatbot/agent/TestChatHistoryBuilder.java | 289 ++++++
.../agent/TestChatbotAgentExecutionPolicy.java | 62 ++
.../recon/chatbot/api/TestChatbotEndpoint.java | 14 +-
.../ozone/recon/metrics/TestNSSummaryMetrics.java | 60 ++
hadoop-ozone/s3gateway/pom.xml | 9 +
.../org/apache/hadoop/ozone/audit/S3GAction.java | 3 +-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 8 +
.../hadoop/ozone/s3/VirtualHostStyleFilter.java | 37 +-
.../ozone/s3/endpoint/BucketCrudHandler.java | 126 +--
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 1 +
...rudHandler.java => BucketLifecycleHandler.java} | 119 +--
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 19 +
.../s3/endpoint/GetObjectAttributesResponse.java | 206 ++++
.../ozone/s3/endpoint/ObjectAttributesHandler.java | 173 +++
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 38 +-
...utBucketLifecycleConfigurationUnmarshaller.java | 56 +-
.../s3/endpoint/S3LifecycleConfiguration.java | 16 +-
.../hadoop/ozone/s3/metrics/S3GatewayMetrics.java | 22 +
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 4 +
.../org/apache/hadoop/ozone/s3/TestGateway.java | 81 ++
.../ozone/s3/TestVirtualHostStyleFilter.java | 58 ++
.../ozone/s3/endpoint/EndpointTestUtils.java | 14 +
.../ozone/s3/endpoint/TestObjectAttributesGet.java | 176 ++++
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 42 +
...utBucketLifecycleConfigurationUnmarshaller.java | 86 ++
.../endpoint/TestS3LifecycleConfigurationPut.java | 110 +-
pom.xml | 14 +-
336 files changed, 16821 insertions(+), 2218 deletions(-)
copy
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/{SkipTracing.java
=> InternalSpanKind.java} (86%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ConcurrentMutableStat.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/tracing/TestTracingAnnotations.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumMultiBuffer.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/utils/TestBufferUtils.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/util/ConcurrentMutableStatBenchmark.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/util/TestConcurrentMutableStat.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestReconcileChunksPerBlockHoleBcsId.java
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/protocolPB/TestSCMSecurityProtocolPB.java
(50%)
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocolPB/TestScmBlockLocationProtocolPB.java
(53%)
copy
hadoop-ozone/interface-storage/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3SecretValueCodec.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocolPB/TestStorageContainerLocationProtocolPB.java
(51%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerMoveFailureDetail.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerMoveFailureTracker.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerMoveFailureBreakdown.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/export/TestExportJob.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestStorageContainerManager.java
create mode 100644
hadoop-ozone/cli-debug/src/test/java/org/apache/hadoop/ozone/debug/replicas/TestReplicasVerify.java
create mode 100644
hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/om/DownloadOMDB.java
rename
hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/security =>
common/src/main/java/org/apache/hadoop/ozone/om}/AWSV4AuthValidator.java (78%)
create mode 100644
hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk
Balancer.json
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/OMMXBeanTests.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestGetDBUpdatesAuthorization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyLookupAclNormalization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMAdminRpcAuthorization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMInstallSnapshotDuringBootstrapping.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotCheckpointDbContent.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/repair/om/TestDownloadOMDBTool.java
create mode 100644
hadoop-ozone/multitenancy-ranger/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestRangerClientMultiTenantAccessControllerMockClient.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeUtilizationMetrics.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffDependencyEntry.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapDiffDependencyGraph.java
copy {hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage =>
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff}/package-info.java
(85%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestTrashOzoneFileSystem.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestVolumeUtilizationMetrics.java
copy
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/lifecycle/{TestOMLifecycleSaveScanStateRequest.java
=> TestOMLifecycleSetServiceStatusRequest.java} (55%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/volume/TestOMQuotaRepairRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/TestSnapDiffDependencyGraph.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/chatbot/agent/ChatHistoryBuilder.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/NSSummaryMetrics.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/handlers/TestEntityHandlerCycleGuard.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/chatbot/agent/TestChatHistoryBuilder.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/metrics/TestNSSummaryMetrics.java
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/{BucketCrudHandler.java
=> BucketLifecycleHandler.java} (63%)
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/GetObjectAttributesResponse.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectAttributesHandler.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestGateway.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectAttributesGet.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPutBucketLifecycleConfigurationUnmarshaller.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
