This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-5713
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 4b52037fdeb HDDS-14102. [DiskBalancer] Fix incorrect default port
documentation in DiskBalancer(9858 -> 19864) (#9459)
add 86ee18ff2bb HDDS-13673. Bump commons-validator to 1.10.0 (#9020)
add 058c65565c9 HDDS-13706. Limit max-uploads at 1000 (#9060)
add 34abace0b41 HDDS-13694. Container Balancer Stop Command Fails with
Error as Already Stopped (#9047)
add 449996c9855 HDDS-13701. Rename StorageVolumeScannerMetrics to
BackgroundVolumeScannerMetrics (#9053)
add 2ff58b3195d HDDS-13709. Fix freon streaming command with duration.
(#9065)
add 259f39db01c HDDS-13710. Ozone Recon - "Metrics source userTableCache
already exists" error closing RocksDB. (#9067)
add 26c066108fd HDDS-13713. Bump opentelemetry to 1.54.1 (#9071)
add 5dd35225f10 HDDS-13714. Bump assertj-core to 3.27.6 (#9074)
add 61fa8a8e00c HDDS-13715. Bump frontend-maven-plugin to 1.15.4 (#9072)
add f3ec68800a4 HDDS-13661. Mark
TestKeyDeletingService#testPurgeKeysRequestBatching as flaky
add 35abb0ac819 HDDS-13573. Mark
TestNSSummaryUnifiedControl#testMultipleConcurrentAttempts as flaky
add d33d3460753 HDDS-13711. Handle null failedEntry in notifyLogFailed to
avoid NPE. (#9070)
add fb3e9bd9912 HDDS-13716. Bump awssdk to 2.34.5 (#9073)
add 81f9d645cd1 HDDS-13718. Improve ASCII logo and startup message (#9075)
add 3afffe8061e HDDS-13224. Support CodecBuffer for
KeyPrefixContainerCodec (#9061)
add be0849276c1 HDDS-12484. Allow ini files to configure
ColumnFamilyOptions for rocksDB (#8173)
add e10b59352db HDDS-13245. Container scanner needs to account for deleted
blocks when building the merkle tree. (#9015)
add 0fda8b28859 HDDS-13544. DN Decommission Fails When Other Datanodes Are
Offline Due to Invalid Affinity Node in Ratis Replication (#8934)
add f7e04eaf3fe HDDS-13719. Replace HadoopIllegalArgumentException (#9077)
add 12a43fe212a HDDS-13144. Fix mvn javadoc:aggregate goal. (#8891)
add 75f32a1383f HDDS-13720. Move OzoneFsShell to ozone-tools (#9078)
add f4c8a12edb0 HDDS-13635. Move ACL check in OMVolumeCreateRequest and
OMBucketCreateRequest to pre execute (#9045)
add b186c321b14 HDDS-13727. Use TextFormat.shortDebugString(..) in
processForDebug(..) methods. (#9085)
add 1468af02067 HDDS-13731. Extra parent dir appeared in CI artifacts
(#9088)
add b1ccf4881e9 HDDS-13735. DBConfigFromFile warns about trying to read
from "" (#9093)
add 4d184ac976b HDDS-13743. Bump junit to 5.14.0 (#9095)
add 23a62e48843 HDDS-13729. Acquire Bulk Bucket locks in order to prevent
deadlock in OmDirectoryPurgeRequest (#9084)
add 9f56748274e HDDS-13741. Invalid OTEL_EXPORTER_OTLP_ENDPOINT, must be
full Jaeger URL (#9098)
add d6a161e201c HDDS-13744. Bump zstd-jni to 1.5.7-5 (#9096)
add d389ff74596 HDDS-13742. Upgrade Jaeger to v2 (#9099)
add 2e6cb323149 HDDS-13746. Bump axios to 0.30.2 (#9083)
add 1985a901ef9 HDDS-13745. Bump license-maven-plugin to 2.7.0 (#9097)
add cac77caa8c5 HDDS-13749. Bump awssdk to 2.35.0 (#9094)
add 3157f705d5c HDDS-13626. Update hadoop to 3.4.2 (#8980)
add fbbfa798698 HDDS-5410. Improve protobuf compilation (#9081)
add a309eca5915 HDDS-13751. Rename all occurrences of "compact" related to
HDDS-13003 into "defrag" (#9105)
add 5dbf8bfd400 HDDS-13681. Add docs for ozone tenant user
get-secret/set-secret commands (#9106)
add bb98cc20b1a HDDS-12998. Bring real container size in pb message when
exporting/importing containers (#8915)
add c4f47c5ff7c HDDS-10298. Replace estimated count with actual count
while listing openkeys. (#9043)
add 8b7579ef838 HDDS-13733. RepeatedOmKeyInfo in deletedKeyTable should
track bucket id (#9089)
add 9bed52bb5b3 HDDS-13754. Add a word of caution in
validateAndUpdateCache javadoc (#9108)
add 2ae1f655d13 HDDS-13722. Refactor duplicate code in
OMDirectoriesPurgeRequestWithFSO (#9102)
add 01a7a3b3d9e HDDS-13707. Update docker quick start section in README
(#9080)
add 584ebb95d35 HDDS-13692. Clarify --all option usage in ListLimitOptions
(#9109)
add c351de99147 HDDS-13756. Introduce Bucket Snapshot Used Bytes and
SnapshotUsedNamespace in BucketInfo (#9115)
add b2a4cb3b375 HDDS-13632. Fix impersonation with SPNEGO for Recon UI
(#9114)
add 527d30a2f09 HDDS-13760. Track last purge transactionInfo in memory to
track inflight purge request in KDS and DDS (#9116)
add ee9cf3139dc HDDS-13765. SnapshotLocalData yaml should also track
snapshotId (#9123)
add 2e8f924a8aa HDDS-13739. Make error messages less verbose for failed
PutBlock or WriteChunk client requests (#9101)
add 528ce916611 HDDS-8203. Log OM Garbage Collection logs to a OM System
audit log file. (#8906)
add 91e0187fbac HDDS-10782. Replace GSON with Jackson in EventQueue (#8949)
add f6bb53671fe HDDS-13763. Ozone Recon - Return empty path when NSSummary
tree parent Id not found while constructing full path of FSO key. (#9121)
add 434f10caca2 HDDS-13658. Bump snakeyaml to 2.5 (#9014)
add c3dc0e975d5 HDDS-13669. Move OM related metadata of OM tasks from SQL
derby DB to rocksDB (#9040)
add 56062d39061 HDDS-13764. KeyDeletingService and
DirectoryDeletingService should reduce snapshot bucket quota usage (#9122)
add d9e91bd9b46 HDDS-13721. Move admin interface usage out of hdds-common
(#9082)
add 22fa6f5ea2e HDDS-13036. Bucket Quota usage should be a total of AOS
usage + Snapshot Usage (#8587)
add c937649d46e HDDS-13677. Update Axios to 1.9.0 and improve error
handling (#9025).
add eea504c5977 HDDS-13570. Reduce the code duplicate between Ratis and EC
ContainerSafeModeRule. (#8940)
add 92483818515 HDDS-13778. Use lean multi-arch image for Hadoop (#9134)
add 9a164854f99 HDDS-11314. OM system audit for internal request and
leader change (#9092)
add dfe4c75df9d HDDS-13637. Add metrics in recon OM sync for staging and
queue based implementation. (#9103)
add 6acff6873f9 HDDS-13688. Remove console log from Recon task status
acceptance tests (#9145)
add bf50f0fa29b HDDS-13617. Avoid immediate ICR for close container (#9064)
add 2ffd6565e12 HDDS-13787. [Docs] Update the instructions for ARM silicon
builds further (#9148)
add 812739ec8b2 HDDS-13690. Move Ozone snapshot configuration properties
to its own page (#9119)
add d2e7b23e570 HDDS-13767. Refactor SnapshotLocalDataYaml related code
into OmSnapshotLocalDataManager (#9124)
add 196bdf89a86 HDDS-13790. Bump exec-maven-plugin to 3.6.1 (#9137)
add 4a46cb6d7dd HDDS-13793. Bump awssdk to 2.35.5 (#9138)
add 720f12303f6 HDDS-13786. Flush BucketInfo changes corresponding in case
an empty directory is purged (#9149)
add bf5ae5e11cd HDDS-13603. [Snapshot] Do not warn a snapshot does not
exist after it is purged. (#9136)
add 4e5e5a0e0ad HDDS-13779. Correctly initialize the bootstrap lock in
OMDBCheckpointServletInodeBasedXfer. (#9142)
add 96ce0b647e0 HDDS-13780. Skeleton of background snapshot defrag service
(#9133)
add 74d4bd220f7 HDDS-13627. In-memory Manager for Snapshot Local Data
(#9141)
add b27282ef3f4 HDDS-13791. Improve Recon bootup times in case of any
failed delta tasks in previous run. (#9151)
add f4337adef54 HDDS-13382. Add RocksDB documentation page (#8740)
add a9371f638fd HDDS-13515. Recon - Staging DB based processing by Recon
during full OM DB snapshot based bootstrap. (#8872)
add d3b9e3343fd HDDS-13657. min free space default configuration to 2
percent (#9153)
add 47f3f315357 HDDS-13792. Move container related metadata from Derby to
OM DB. (#9154)
add 5e715aa072f HDDS-13797. Refactor OzoneManagerLock Resource class to
handle hierarchical resource locking (#9159)
add 5fdf2d2d0bd HDDS-13679. Design doc for OpenTelemetry integration
(#9051)
add 1e7067a55de HDDS-13798. Implement
PoolBasedHierarchicalResourceLockManager for Hierarchical Resource (#9160)
add a65f72a0fe3 HDDS-13784. [Ozone 2.1] Update proto.lock files (#9158)
add 615db22b5fa HDDS-13732. Move off of macos-13 runner (#9163)
add 098bbd23d91 HDDS-13651. Remove delete transaction from memory in
abnormal case. (#9144)
add 33d4744f0ce HDDS-13758. Add replicatedSizeOfFiles to NSSummary to
Calculate DiskUsage (#9127)
add a2bcd5815ed HDDS-13762. Mis replication bug when there is a dead
maintenance node (#9156)
add 3daf9f89a9b HDDS-13808. Revert of HDDS-13792. (#9168)
add 155c0284384 HDDS-13805. Ozone Recon - Correct the open keys and delete
pending and summary APIs to load OM metadata from new globalStats rocksDB
table. (#9167)
add 3fee4c21bd5 HDDS-13799. Revert "HDDS-12982. Reduce log level for
snapshot validation failure (#8851)" (#9162)
add cfcfcd501f5 HDDS-13811. Bump awssdk to 2.35.10 (#9172)
add 54b9e44435b HDDS-13813. Bump jacoco to 0.8.14 (#9171)
add 9cf566525f0 HDDS-13814. Bump opentelemetry to 1.55.0 (#9170)
add aee5aa31a73 Revert "HDDS-13814. Bump opentelemetry to 1.55.0 (#9170)"
add 819e9cf088c HDDS-13757. Create Builder for
OMSnapshotMoveDeletedKeysResponse instead of suppressing checkstyle warning
(#9161)
add e3fbb4f4b4f HDDS-13815. Test for repairing ratis transaction is flaky
(#9178)
add 317aa5f2a75 HDDS-13816. Intermittent failure in Put Object with
STANDARD_IA (#9179)
add d243267793d HDDS-13466. Disable flaky TestOmSnapshotObjectStore,
TestOmSnapshotWithoutBucketLinkingLegacy
add 6db557e8687 HDDS-13686. Remove duplicate config
`ozone.om.ratis.server.pending.write.byte-limit` (#9177)
add 8278977995f HDDS-13807. Fix 404 error on switching between Namespace
Usage page (#9183).
add 63809d6a2c3 HDDS-13821. Fix Container Size distribution graph in new
UI (#9182)
add 43357ff1aca HDDS-13769. Orphan files can be copied as hard link file
list has files added outside the lock. (#9128)
add ea45c40eee1 HDDS-13820. Fix indentation in
RunningDatanodeState#buildEndPointTask (#9186)
add 411f25ce861 HDDS-13814. Bump opentelemetry to 1.55.0 (#9170)
add 423a7f568cd HDDS-13096. Counter for total volumes and healthy volumes
per node in datanode list command. (#9023)
add 62682aca8a5 HDDS-13776. Fail DirectoryPurge requests if previous
snapshot ID validation fails. (#9130)
add 00c935b7291 Revert "HDDS-13776. Fail DirectoryPurge requests if
previous snapshot ID validation fails. (#9130)"
add c33f8e013e7 HDDS-13460. [Docs] S3 secret storage. (#8824)
add cc730871b27 HDDS-13801. [Docs] ozone s3 getsecret command and REST API
supports user parameter (#9164)
add 1b595f3cf72 HDDS-13770. SstBackup Directory can have orphan files
after bootstrap not present in compaction log table (#9169)
add 65fb295a320 HDDS-13639. Optimize container iterator for frequent
operation (#9147)
add b067b7051fd HDDS-13834. [Ozone 2.1] Update master branch version
number (#9196)
add ddef96ecf9f HDDS-13837. Bump jnr-posix to 3.1.21 (#9200)
add 87dbdcce535 HDDS-13838. Bump zstd-jni to 1.5.7-6 (#9197)
add 7e8e0addca8 HDDS-13836. Bump exec-maven-plugin to 3.6.2 (#9199)
add 1390527aa1b HDDS-13717. Bump Bouncy Castle to 1.82 (#9054)
add 94ea4320602 HDDS-13839. Bump awssdk to 2.36.2 (#9198)
add 798c4aa42af HDDS-13843. Add Open Containers card in new UI (#9204).
add d5be9866648 HDDS-13840. Reset Namespace metadata pagination when path
changes (#9203)
add b39bac05eca HDDS-13783. Implement locks for OmSnapshotLocalDataManager
(#9140)
add 388f3d2ac4c HDDS-13400. S3g has accumulated memory pressure due to
unlimited ElasticByteBufferPool in RpcClient (#9166)
add 2806bae1812 HDDS-13004. Snapshot Cache lock on a specific snapshotId
(#9210)
add d4e7d94239a HDDS-13167. Add example for uploading file via HttpFS
(#9175)
add 8a5c4e86753 HDDS-12749. Use EnumCounters instead Map<Type, Integer>
for command counts (#9211)
add f30870f2a22 HDDS-13841. Namespace summary API gives wrong count of
directories and keys. (#9213)
add fb706e783d3 HDDS-13831. Refine set role logic in getServicelist (#9191)
add 8bd70b7974c HDDS-13833. Add transactionInfo field in SnapshotLocalData
and update the value on SnapshotPurgeRequest (#9193)
add a8b86079a8d HDDS-13856. Change SstFileInfo to track fileName as the
name of the file without sst extension (#9221)
add e2e862e4b3c HDDS-13859. OmSnapshotLocalDataManager should handle
needsDefrag flag in the yaml file (#9224)
add 72167cf86c3 HDDS-13860. RocksDatabase#open leaks column family handles
when failing to read cfOptions from file (#9225)
add c21ec5ddafe HDDS-13847. Introduce Snapshot Content Lock to lock table
contents (#9212)
add 833e955bdc2 HDDS-13822. Add regression testing for OM epoch and txId
calculation (#9188)
add 4d6f3a5a769 HDDS-13772. Snapshot Paths to be re read from om
checkpoint db inside lock again. (#9131)
add 1de5c2f77aa HDDS-13755. Add doc for ozone sh snapshot listDiff command
(#9238)
add 29a9d0f77ea HDDS-13871. Bump awssdk to 2.37.3 (#9233)
add 55bd1f14b89 HDDS-13851. Remove extra OzoneConfiguration#of from
OzoneFileSystem#initialize (#9231)
add 25cceefa0a3 HDDS-13872. Bump junit to 5.14.1 (#9232)
add 991a291ee37 HDDS-13485. Reduce duplication between
ContainerSafeModeRule tests (#9194)
add 5c35ebb54f0 HDDS-13830. Snapshot Rocks DB directory path computation
based on local version of snapshot (#9208)
add be8567ed916 HDDS-13858. Add permission check and test in getFileStatus
(#9237)
add af123a5e5ca HDDS-13640. Add CLI that allows manually triggering
snapshot defrag (#9155)
add be3b8282a26 HDDS-13823. Initial s3v volume cache entry will not be
evicted until OM restart (#9241)
add bc577ae42c2 HDDS-13868. Add unit test coverage for OMNodeDetails
(#9245)
add 51deb3c9004 HDDS-13826. Move ACL check in OMKeySetTimesRequest (#9192)
add 5ab59c952b5 HDDS-13737. S3 ETag JSON should be quoted (#9248)
add 61cf1f7c719 HDDS-13178. Include block size in delete request and pass
it to SCM. (#9246)
add d85440b3292 HDDS-13785. Remove orphan versions from SnapshotLocalData
Yaml file (#9150)
add a8206e027a6 HDDS-13768. OM should acquire snapshot cache lock before
taking checkpoint. (#9129)
add a40d1296a2f HDDS-13863. RocksDBCheckpointDiffer Compaction Tracker
erroneously logs warn logs in absence of snapshot (#9230)
add dce2e5e8c0d HDDS-13187. Extend Recon events handling to
MultipartInfoTable (#9250)
add 1fa7e569f7c HDDS-13861. Intermittent failure in
testOverWriteKeyWithAndWithOutVersioning (#9234)
add c6d7bd1f79f HDDS-13873. Block ozone repair if service is running and
skip check in dry run mode (#9249)
add 3f54b1452c5 HDDS-13892. Allow flaky-test-check to be cancelled through
the GitHub UI (#9260)
add c5da9e2a86d HDDS-13137. Include services to be stopped for `ozone
repair` in command description (#9217)
add 2e323b4b92c HDDS-13258. Refactor HttpServletResponse (Part1 -
HddsConfServlet) (#9126)
add 4ff1e7fbdbf HDDS-13861. Intermittent failure in
testOverWriteKeyWithAndWithOutVersioning [2/2] (#9261)
add 272544aa95c HDDS-13889. Mark
testSnapshotAndKeyDeletionBackgroundServices as flaky
add 312ad3cfb2a HDDS-13879. Mark testBucketUsedNamespace as flaky
add 7787c27dea7 HDDS-13880. Mark testSnapshotDeepClean as flaky
add 64c5806fced HDDS-13898. Bump sqlite-jdbc to 3.51.0.0 (#9267)
add 1445ba23476 HDDS-13899. Bump opentelemetry to 1.56.0 (#9265)
add 4db764a673b HDDS-13900. Bump awssdk to 2.38.2 (#9266)
add 78490e99592 HDDS-11370. Use GitHub's stale action to auto-close
inactive pull requests (#9259)
add ea808b3d07b HDDS-13903. Remove all MiniDFSCluster comments in tests
(#9274)
add faf39ee3995 HDDS-13901. OmSnaphshotLocalDataManager should throw
IOException if unable to resolve to a previous snapshot id (#9269)
add ff41ce6092a HDDS-13849. Refactor getTablePrefix function in
SnapshotDiff flow (#9235)
add 598cbc9529a HDDS-13893. SCMBlockDeletingService fails to work when
container in tx not found in SCM ContainerManager (#9262)
add 5322ff18c85 HDDS-13442. Replace OMKeyInfo with light-weight
ReconBasicOmKeyInfo for MultipartInfoInsightHandler (#9251)
add 94a301641f7 HDDS-13781. Certificate expiry date should consider time
zone daylight saving impact (#9209)
add 6d0288bea7f HDDS-13641. [Docs] Include configurations and metrics
related to DataNode decommission (#9004)
add 8c3de469139 HDDS-11370. Addendum: increase operations-per-run for
stale action (#9280)
add 4ae6be0381e HDDS-13466. Mark TestOmSnapshotFsoWithNativeLib as
unhealthy
add ffa63e0cb6f HDDS-13509. Have single constructor in
ContainerReplicaPendingOps and ContainerReplicaOp (#9118)
add 576d5cfa97e HDDS-13907. Prune ACLs from OmKeyInfo before building
purge requests in DirectoryDeletingService (#9275)
add 13075df2c78 HDDS-13910. Remove unused POM profile: clover (#9278)
add 20471a74e6c HDDS-13905. Bootstrap lock acquired in background services
can lead to deadlock (#9273)
add f0f2a0a2959 HDDS-13904. Exception handling correctly should release
snapshot read lock (#9271)
add b255f110aeb HDDS-13794. Use synchronized logger for auditing tasks
(#9229)
add 18d8ae1ccb6 HDDS-13917. Refactor Recon test helper class CommonUtils
(#9287)
add d82cfaa160b HDDS-13918. Bump Ratis to 3.2.1 (#9288)
add 47473dc5a28 HDDS-13452. Prevent snapshot defrag from happening before
upgrade finalization (#9240)
add d480dd2dde0 HDDS-13923. Remove leftover references to custom pending
label (#9290)
add 28fe3437a5f HDDS-13916. Remove setModificationTime and setOwner from
OmBucketInfo (#9291)
add 367e6252d8b HDDS-13931. Fix protobuf install instructions for Apple
ARM (#9295)
add 3ad3d1691ba HDDS-13934. Bump awssdk to 2.38.7 (#9299)
add 36c28b37411 HDDS-13935. Bump common-custom-user-data-maven-extension
to 2.1.0 (#9300)
add f4660a254a1 HDDS-10388. Make WithMetadata immutable (#9293)
add dfb564b7702 HDDS-13938. Extract builderFromProtobuf methods (#9303)
add cbad1202d09 HDDS-13406. Standardize option for OM address for `ozone
admin om` (#8918)
add 77680bda7d5 HDDS-12094. OzoneKey isFile returns true for directories
(#8838)
add beac3271cc1 HDDS-10178. Shaded Jar build fails on case-insensitive
filesystem (#6042)
add 200a2430458 HDDS-13939. Potential channel leak in
StreamingClient.stream() method (#9304)
add a5658eebf7f HDDS-9279. Basic implementation of OM Follower read (#9222)
add dc2b5585818 HDDS-13867. SnapshotDiff delta file computation should
happen based on LocalDataYaml file (#9268)
add f9facd848ae HDDS-13946. Wrong fields used to determine deleted block
size in OM. (#9308)
add 902dd43589e HDDS-13944. S3G Metrics support displaying Netty off-heap
memory (#9309)
add f093e68af3c HDDS-13466. Mark
TestOmSnapshotObjectStoreWithLinkedBuckets as unhealthy
add 2b74301d228 HDDS-13936. Replace Thread.sleep with waitFor in
TestContainerBalancerOperations (#9302)
add dc57ebb07ab HDDS-13947. Stop managing plexus-archiver version (#9311)
add 9bb1898c576 HDDS-13957. Snapshot needsDefrag should be set on upgrade
when creating missing yaml files (#9321)
add a337a777909 HDDS-9855. Clarify error message for invalid volume or
bucket name (#9301)
add e51a88bfa76 HDDS-13943. Improve error message for malformed input in
DirstreamClientHandler (#9316)
add b6c742e162f HDDS-13184. Persist Block Size in Delete Transaction for
SCM (#9272)
add 9ca907c37b1 HDDS-13883. Support HTTPS with ozone insight command
(#9285)
add 39e4673787c HDDS-499. Display descriptions for properties on the
configuration page (#9152)
add 00dd115a9a2 HDDS-13912. Modularise Snapshot Delta file computer (Full
Diff) (#9283)
add b7241c0d297 HDDS-13829. Audit file rollover does not happen at the
start of a new day (#9190)
add aefe07a4d6c HDDS-13467. Introduce pending deletion block bytes of
container in DN. (#9325)
add 4382cb33287 HDDS-13929. Modularise Snapshot Delta file computer
(Efficient Diff) (#9312)
add 83f4b3be713 HDDS-13959. Flaky check fails at
testPurgeKeysRequestBatching despite passing repeated run (#9329)
add 9c932b01221 HDDS-13664. Persist pendingDeleteBytes only when feature
is finalized. (#9330)
add 03816462c89 HDDS-13949. Move dbTxSequenceNumber from SnapshotInfo to
LocalDataYaml file (#9313)
add 63ee00df7a6 HDDS-13844. Decouple DirectoryDeletingService delete
batching from Ratis request size. (#9270)
add d7e68784840 HDDS-12133. Define replication metrics task names and
descriptions as constants to reuse in CommandHandler (#9215)
add 56e7988f0f5 HDDS-13937. Make WithObjectID immutable (#9327)
add fd3cb676477 HDDS-13930. Snapshot diff can use rocksdb iterator instead
of using multiple gets (#9317)
add acc8a868642 HDDS-13723. Add detailed lock information for
omMetadataManager locks (#9157)
add 95ef871d9ce HDDS-13979. Bump awssdk to 2.39.2 (#9347)
add 4eb883f8707 HDDS-13927. Recon - OM Db delta updates handler resource
cleanup. (#9307)
add c1a606ebc1d HDDS-13965. Code cleanup for deleted block transaction.
(#9337)
add fe5cd611ddf HDDS-13970. Remove HttpFS site (#9341)
add d6d5c4e977f HDDS-13993. Refactor and move SnapshotDefragService to
snapshot package (#9355)
add cc479a8679d HDDS-13977. Bump Netty to 4.1.128, gRPC to 1.77.0 (#9349)
add c8bcf7f7d64 HDDS-13978. OMLockDetails should not be used as the object
returns a ThreadLocal Object (#9346)
add 299b2948095 HDDS-13009. Background snapshot defrag service (#9324)
add 7c53356a5e4 HDDS-13964. Validate objectID/updateID in
WithObjectID.build() (#9356)
add cd78b4190a9 HDDS-13940. Make OmVolumeArgs owner/timestamps/quotas
immutable (#9305)
add bc06db9ca76 HDDS-13960. Convert FlatResource Lock into DAG based lock
ordering for the snapshot lock (#9328)
add c42fc3aae47 HDDS-13962. Make Bootstrap Lock acquire
DAGBasedLeveledResourceLock (#9331)
add bad6cfdb6fe HDDS-13996. Skip Hugo build in unrelated checks (#9357)
add d5664258c4a HDDS-14002. Refactor DBStore interface to return Table
instead TypedTable (#9360)
add 92de1ca728f HDDS-13995. Improve summary for installation failure
(#9359)
add f52c16620fe HDDS-13973. The ground work to support stream read block
(#9342)
add f66c87e2ec2 HDDS-13984. Reuse ResourceLockTracker for
HierarchicalResourceLockManager (#9352)
add 58d1b6595ff HDDS-13974. Use the same gRPC steam for reading the same
block. (#9369) Contributed by Stephen O'Donnell
add 26da8d4272e HDDS-14001. Reduce custom client RPC timeout in tests
(#9361)
add 8f0c6e3d522 HDDS-13809. Close checkpoint OMMetadataManager properly
(#9173)
add 39e26d762c6 HDDS-13862. Show datanode reserved space in ozone admin
datanode usageinfo command (#9338)
add f52e9832312 HDDS-13956. Code reliability and data integrity
improvements in Recon (#9322)
add c59a08d9de8 HDDS-14013. Release stream resources in
StreamBlockInputStream properly (#9374)
add 40304b69ae5 HDDS-13983. Support full resource lock in
PoolBasedHierarchicalLockManager (#9353)
add 76b146c0301 HDDS-13967. OM bootstrap lock should acquire a lock on
SnapshotLocalDataManager before copying the local data files (#9335)
add 0a9df7bc1e8 HDDS-14015. Delete older snapshot checkpoint dirs under
the snapshot content lock (#9380)
add 25a1f804f34 HDDS-13969. Add SCM metric for number of datanodes out of
space (#9339)
add 2da06967135 HDDS-14021. Read-only RDBStore should not flush on close
(#9384)
add a29e2b186af HDDS-14023. Extract utility for resizing thread pool
(#9387)
add ae799bd73c2 HDDS-14016. Reconfigure function should not manually set
configuration (#9383)
add 01ca7ae1cf0 HDDS-13999. Optionally ignore ACL from client during
object creation (#9386)
add ffa50c5313e HDDS-14028. Bump awssdk to 2.39.5 (#9395)
add 8a6ed86b385 HDDS-13752. Copy Hadoop RPC code to Ozone (#9107)
add 98b938157ae HDDS-13865. Intermittent failure in
TestDirectoryDeletingServiceWithFSO (#9236)
add 8737837d5c8 HDDS-12301. Move ozone.om.volume.listall.allowed into
OmConfig (#9391)
add 4d284b88199 HDDS-14032. Remove unused RemovalListener from
OmSnapshotManager (#9397)
add 02f90379a66 HDDS-13753. Use forked Hadoop RPC (#9112)
add e5c1119b340 HDDS-13998. Make SCM replication manager report sample
size dynamically configurable (#9373)
add 4f2f67c65c9 HDDS-14014. Refactor dir event handlers in
NSSummaryTaskDbEventHandler (#9389).
add ccbfed5f82b HDDS-14026. Close OutputStream properly in
TestBlockOutputStream (#9392)
add 9c36cc01d33 HDDS-14042. Intermittent testcase failure in
TestScmDataDistributionFinalization (#9405)
add 75e7ca64999 HDDS-13982. OmKeyInfo fileName possible inconsistency
between cache and DB (#9381)
add 227fb6aa4f4 HDDS-13842. Exit safemode rule quickly at follower for
idle SCM (#9202)
add d673fbf634c HDDS-13975. Limit the number of responses in stream read
block. (#9375)
add fe33021ff06 HDDS-4267. Set limit for coredump only if allowed (#9396)
add 2930ecc69c5 HDDS-10791. Duplicated instanceof checks in
OzoneOutputStream (#9370)
add 0fdf91470bd HDDS-14052. Auto-detect arch for RPM builds (#9379)
add 173fd61f7a8 HDDS-14033. Remove leftover TODO from
OMKeyDeleteRequestWithFSO (#9408)
add 5ee1bf483dd HDDS-14046. ReconStorageContainerManagerFacade is not
initialized properly after upgrade (#9407).
add a3a3286d129 HDDS-13980. SCM start DN protocol server during startup
(#9350)
add 2702db5ca6f HDDS-13465. [Recon] Create API Endpoint to Return
Consolidated Storage Report. (#9326)
add c374145c50a HDDS-14055. StreamRead: Fix bugs for arbitrary block
length. (#9410)
add e0d972cc16f HDDS-14056. Relocate protobuf in ozone-filesystem shaded
jars (#9412)
add 65e2bb312cf HDDS-14065. populate-cache fails at
ozone-filesystem-hadoop2 (#9419)
add 509c5f54dc4 HDDS-11645. Intermittent timeout in
TestReconScmSnapshot#testExplicitRemovalOfNode (#9414)
add cef8b2ad320 HDDS-13953. Optionally allow skipping ReadIndex for Leader
(#9319)
add 85feb8813ea HDDS-13981. DN quickly reregister to SCM (#9351)
add a8f208c952b HDDS-13761. Remove hadoop-thirdparty protobuf compilation
(#9125)
add f3b60487651 HDDS-14060. Replace Preconditions.checkNotNull in
ozone-manager (#9422)
add 13464ec416a HDDS-5407. Upgrade protobuf to 3.25.8 (#9113)
add 7508da8c99e HDDS-14024. Clean up handles when dropping column family
(#9388)
add 04d337952a3 HDDS-14000. Require existing build for
acceptance/kubernetes check (#9377)
add bad3e166aa3 HDDS-14035. StreamRead: Positioned-read should not do
pre-read. (#9425)
add 13f8a6767e8 HDDS-14051. Update k3s version to v1.34.2+k3s1 (#9428)
add ebacba1a49f HDDS-14040. Ozone client hang for data write in failure
scenario (#9401)
add ce65c678496 HDDS-14048. Optimise NodesOutOfSpace metric logic (#9418)
add 641730dbd2b HDDS-13827. Increase the number of retained audit log
files (#9189)
add 5eb975777bd HDDS-13835. Fix Replication columns in Open Keys panel
(#9429)
add 923bd2b0522 HDDS-14029. Bump Bouncy Castle to 1.83, maven-shade-plugin
to 3.6.1 (#9433)
add 59932f6d7bb HDDS-14058. Replace Preconditions.checkNotNull in tests
(#9437)
add 0e86035a4a5 HDDS-14081. Docker logs missing from failed xcompat bundle
(#9438)
add 190d043bc90 HDDS-14059. Replace Preconditions.checkNotNull in
hdds-container-service (#9411)
add a92ac05e4db HDDS-14080. Intermittent disk out of space in EC
acceptance tests (#9440)
add 5c904bb2628 HDDS-12360. Disk usage vs overview Cluster summary
improvement (#8636)
add 0d4f6bbb2b0 HDDS-14027. Use CodecBuffer instead of byte array in
defrag to reduce GC (#9393)
add fb80176568d HDDS-14093. Timeout in
xcompat-cluster-2.0.0-client-2.0.0-write (#9444)
add c39fd4a3559 HDDS-14096. Bump opentelemetry to 1.57.0 (#9447)
add 85209f13e8f HDDS-13966. Clean up TestKeyManagerImpl integration test
(#9424)
add 36f80fd722e HDDS-14095. Bump sqlite-jdbc to 3.51.1.0 (#9448)
add 9437ffa6a36 HDDS-14085. Bump Hugo to 0.152.2 (#9441)
add 407779a9ce3 HDDS-14088. Replace Preconditions.checkNotNull in
hdds-server-scm (#9450)
add 48facbdc391 HDDS-14086. Replace Preconditions.checkNotNull in
hdds-server-framework (#9449)
add 18b27411128 HDDS-14087. Replace Preconditions.checkNotNull in
hdds-client and -common (#9452)
add aa21f1141f0 HDDS-14089. Replace Preconditions.checkNotNull in
ozone-client and -common (#9453)
add 7deb6f67c00 HDDS-14050. Mark
testFinalizationEmptyClusterDataDistribution as flaky
add 55562bd9ac9 HDDS-14101. Mark
testContainerStateMachine(Single/Dual)FailureRetry as flaky
add 73898fafe00 HDDS-13848. [STS] Artifacts for Ranger to authorize STS
token (#9434)
add 73b63ae2878 HDDS-14079. Fix Recon startup failures during schema
upgrades due to race conditions. (#9436).
add d1aecfd3a82 HDDS-13954. Add localLease for followerRead (#9320)
add eaa7b65723c HDDS-14030. Add ConfigGroup prefix to all configs where
missing (#9460)
add 4121f519f6d HDDS-14097. Bump awssdk to 2.40.3 (#9446)
add 79b59457479 HDDS-13941. Introduce builder for list of ACLs (#9430)
add d646f60c0df HDDS-14061. Log OM request type if response size exceeds
threshold. (#9417)
add 41539924d51 HDDS-14090. Replace remaining Preconditions.checkNotNull
usage (#9458)
add 9036e1ab6e8 HDDS-13535. Show under/over-replication in `replicas
verify --container-state` results (#9135)
add 0d1ceb35ec2 HDDS-13771. Hard link list file structure should have all
paths relative to metadata directory (#9132)
add a534ac2f388 Merge branch 'master' of github.com:apache/ozone into
HDDS-5713-upstream
No new revisions were added by this update.
Summary of changes:
.github/ci.md | 7 +-
.github/close-pending.sh | 41 -
.github/closing-message.txt | 7 -
.github/comment-commands/close.sh | 35 -
.github/comment-commands/pending.sh | 39 -
.github/comment-commands/ready.sh | 35 -
.github/workflows/check.yml | 1 +
.github/workflows/ci.yml | 15 +-
.../{close-pending.yaml => close-stale-prs.yaml} | 30 +-
.github/workflows/intermittent-test-check.yml | 4 +-
.mvn/extensions.xml | 2 +-
README.md | 23 +-
dev-support/byteman/hadooprpc.btm | 2 +-
dev-support/pmd/pmd-ruleset.xml | 1 +
dev-support/pom.xml | 2 +-
hadoop-hdds/annotations/pom.xml | 4 +-
hadoop-hdds/client/pom.xml | 4 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 19 +-
.../hadoop/hdds/scm/XceiverClientCreator.java | 4 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 117 +-
.../hadoop/hdds/scm/XceiverClientManager.java | 13 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 3 +-
.../hadoop/hdds/scm/client/HddsClientUtils.java | 4 +-
.../hdds/scm/storage/AbstractDataStreamOutput.java | 2 +-
.../hdds/scm/storage/BlockDataStreamOutput.java | 9 +-
.../hdds/scm/storage/BlockExtendedInputStream.java | 97 +
.../hadoop/hdds/scm/storage/BlockInputStream.java | 93 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 9 +-
.../hadoop/hdds/scm/storage/ByteBufferReader.java | 21 +-
.../hdds/scm/storage/ExtendedInputStream.java | 26 +-
.../hdds/scm/storage/MultipartInputStream.java | 39 +-
.../hdds/scm/storage/StreamBlockInputStream.java | 531 +++
.../ozone/client/io/BadDataLocationException.java | 2 +-
.../client/io/BlockInputStreamFactoryImpl.java | 18 +
.../client/io/BoundedElasticByteBufferPool.java | 148 +
.../hdds/scm/storage/TestBlockInputStream.java | 4 +-
.../client/io/TestBlockInputStreamFactoryImpl.java | 15 +-
.../io/TestBoundedElasticByteBufferPool.java | 121 +
.../common/dev-support/findbugsExcludeFile.xml | 6 +
hadoop-hdds/common/pom.xml | 20 +-
hadoop-hdds/common/src/main/conf/ozone-env.sh | 19 +-
.../protobuf/{Proto2Utils.java => ProtoUtils.java} | 8 +-
.../org/apache/hadoop/hdds/DatanodeVersion.java | 2 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 3 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 96 +-
.../java/org/apache/hadoop/hdds/StringUtils.java | 7 +
.../apache/hadoop/hdds/cli/AbstractSubcommand.java | 11 +
.../org/apache/hadoop/hdds/cli/GenericCli.java | 1 +
.../hadoop/hdds/cli/GenericParentCommand.java | 5 +
.../hdds/client/ReplicationConfigValidator.java | 2 +-
.../hadoop/hdds/conf/HddsPrometheusConfig.java | 2 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 34 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 3 +-
.../hdds/ratis/ContainerCommandRequestMessage.java | 3 +-
.../hadoop/hdds/ratis/conf/RatisClientConfig.java | 26 +-
.../org/apache/hadoop/hdds/recon/ReconConfig.java | 6 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 6 +-
.../hadoop/hdds/scm/ScmRatisServerConfig.java | 2 +-
.../hadoop/hdds/scm/StreamingReadResponse.java | 55 +
.../StreamingReaderSpi.java} | 15 +-
.../apache/hadoop/hdds/scm/XceiverClientSpi.java | 13 +
.../hdds/scm/container/ContainerException.java | 2 +-
.../scm/container/ContainerNotFoundException.java | 2 +-
.../ContainerReplicaNotFoundException.java | 2 +-
.../scm/container/ReplicationManagerReport.java | 16 +-
.../balancer/ContainerBalancerConfiguration.java | 28 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 2 +-
.../hadoop/hdds/scm/ha/NonRetriableException.java | 2 +-
.../apache/hadoop/hdds/scm/net/InnerNodeImpl.java | 3 +-
.../org/apache/hadoop/hdds/scm/net/NodeSchema.java | 3 +-
.../ContainerCommandResponseBuilders.java | 21 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 33 +
.../hadoop/hdds/security/SecurityConfig.java | 13 +-
.../certificate/utils/CertificateSignRequest.java | 10 +-
.../certificate/utils/SelfSignedCertificate.java | 28 +-
.../org/apache/hadoop/hdds/server/YamlUtils.java | 6 +-
.../hadoop/hdds/upgrade/HDDSLayoutFeature.java | 3 +-
.../hadoop/hdds/utils/BackgroundService.java | 4 +
.../org/apache/hadoop/hdds/utils/GlobPattern.java | 165 +
.../apache/hadoop/hdds/utils/HddsVersionInfo.java | 4 +-
.../java/org/apache/hadoop/hdds/utils/IOUtils.java | 22 +
.../apache/hadoop/hdds/utils/ResourceCache.java | 12 +-
.../org/apache/hadoop/hdds/utils/VersionInfo.java | 16 +-
.../hdds/utils/io/RandomAccessFileChannel.java | 92 +
.../org/apache/hadoop/ipc_/AlignmentContext.java | 92 +
.../ipc_/AsyncCallLimitExceededException.java | 36 +
.../org/apache/hadoop/ipc_/CallQueueManager.java | 456 +++
.../java/org/apache/hadoop/ipc_/CallerContext.java | 144 +
.../main/java/org/apache/hadoop/ipc_/Client.java | 1911 ++++++++++
.../java/org/apache/hadoop/ipc_/ClientCache.java | 121 +
.../main/java/org/apache/hadoop/ipc_/ClientId.java | 94 +
.../java/org/apache/hadoop/ipc_/CostProvider.java | 46 +
.../org/apache/hadoop/ipc_/DecayRpcScheduler.java | 1029 ++++++
.../hadoop/ipc_/DecayRpcSchedulerMXBean.java | 32 +
.../apache/hadoop/ipc_/DefaultCostProvider.java | 43 +
.../apache/hadoop/ipc_/DefaultRpcScheduler.java | 49 +
.../java/org/apache/hadoop/ipc_/ExternalCall.java | 94 +
.../java/org/apache/hadoop/ipc_/FairCallQueue.java | 453 +++
.../apache/hadoop/ipc_/FairCallQueueMXBean.java | 27 +
.../apache/hadoop/ipc_/GenericRefreshProtocol.java | 49 +
.../org/apache/hadoop/ipc_/IdentityProvider.java | 36 +
.../java/org/apache/hadoop/ipc_/IpcException.java | 33 +
.../ipc_/ObserverRetryOnActiveException.java | 31 +
.../org/apache/hadoop/ipc_/ProcessingDetails.java | 92 +
.../java/org/apache/hadoop/ipc_/ProtoUtil.java | 194 +
.../org/apache/hadoop/ipc_/ProtobufHelper.java | 47 +
.../org/apache/hadoop/ipc_/ProtobufRpcEngine.java | 605 +++
.../hadoop/ipc_/ProtobufRpcEngineCallback.java | 29 +
.../java/org/apache/hadoop/ipc_/ProtocolInfo.java | 39 +
.../org/apache/hadoop/ipc_/ProtocolMetaInfoPB.java | 34 +
.../ProtocolMetaInfoServerSideTranslatorPB.java | 121 +
.../apache/hadoop/ipc_/ProtocolMetaInterface.java | 38 +
.../java/org/apache/hadoop/ipc_/ProtocolProxy.java | 118 +
.../org/apache/hadoop/ipc_/ProtocolSignature.java | 253 ++
.../org/apache/hadoop/ipc_/ProtocolTranslator.java | 33 +
.../java/org/apache/hadoop/ipc_/ProxyCombiner.java | 151 +
.../src/main/java/org/apache/hadoop/ipc_/RPC.java | 1168 ++++++
.../hadoop/ipc_/RefreshCallQueueProtocol.java | 44 +
.../org/apache/hadoop/ipc_/RefreshHandler.java | 32 +
.../org/apache/hadoop/ipc_/RefreshRegistry.java | 134 +
.../org/apache/hadoop/ipc_/RefreshResponse.java | 76 +
.../org/apache/hadoop/ipc_/RemoteException.java | 140 +
.../org/apache/hadoop/ipc_/ResponseBuffer.java | 102 +
.../org/apache/hadoop/ipc_/RetriableException.java | 39 +
.../java/org/apache/hadoop/ipc_/RetryCache.java | 391 ++
.../org/apache/hadoop/ipc_/RpcClientException.java | 47 +
.../java/org/apache/hadoop/ipc_/RpcClientUtil.java | 241 ++
.../java/org/apache/hadoop/ipc_/RpcConstants.java | 69 +
.../java/org/apache/hadoop/ipc_/RpcEngine.java | 138 +
.../java/org/apache/hadoop/ipc_/RpcException.java | 49 +
.../apache/hadoop/ipc_/RpcInvocationHandler.java | 36 +
.../org/apache/hadoop/ipc_/RpcMultiplexer.java | 32 +
.../hadoop/ipc_/RpcNoSuchMethodException.java | 47 +
.../hadoop/ipc_/RpcNoSuchProtocolException.java | 46 +
.../java/org/apache/hadoop/ipc_/RpcScheduler.java | 81 +
.../org/apache/hadoop/ipc_/RpcServerException.java | 62 +
.../java/org/apache/hadoop/ipc_/RpcWritable.java | 192 +
.../java/org/apache/hadoop/ipc_/Schedulable.java | 45 +
.../main/java/org/apache/hadoop/ipc_/Server.java | 3904 ++++++++++++++++++++
.../org/apache/hadoop/ipc_/StandbyException.java | 32 +
.../hadoop/ipc_/UnexpectedServerException.java | 48 +
.../apache/hadoop/ipc_/UserIdentityProvider.java | 36 +
.../org/apache/hadoop/ipc_/VersionedProtocol.java | 54 +
.../hadoop/ipc_/WeightedRoundRobinMultiplexer.java | 151 +
.../hadoop/ipc_/WeightedTimeCostProvider.java | 110 +
.../org/apache/hadoop/ipc_/WritableRpcEngine.java | 630 ++++
.../hadoop/ipc_/metrics/RetryCacheMetrics.java | 92 +
.../hadoop/ipc_/metrics/RpcDetailedMetrics.java | 86 +
.../org/apache/hadoop/ipc_/metrics/RpcMetrics.java | 321 ++
.../apache/hadoop/ipc_/metrics/package-info.java | 22 +
.../java/org/apache/hadoop/ipc_/package-info.java | 22 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 13 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 16 +-
.../org/apache/hadoop/ozone/OzoneSecurityUtil.java | 40 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 9 +
.../hadoop/ozone/conf/OzoneServiceConfig.java | 2 +-
.../ozone/container/common/helpers/ChunkInfo.java | 4 +-
.../hadoop/ozone/lock/BootstrapStateHandler.java | 29 +-
.../apache/hadoop/ozone/util/ObjectSerializer.java | 73 +
.../apache/hadoop/ozone/util/ProtobufUtils.java | 4 +
.../apache/hadoop/ozone/util/WithChecksum.java} | 13 +-
.../apache/hadoop/ozone/util/YamlSerializer.java | 159 +
.../org/apache/hadoop/security_/SaslRpcClient.java | 715 ++++
.../org/apache/hadoop/security_/SaslRpcServer.java | 366 ++
.../main/resources/hdds-version-info.properties | 4 +-
.../common/src/main/resources/ozone-default.xml | 86 +-
.../hadoop/hdds/conf/SimpleConfiguration.java | 16 +-
.../container/TestReplicationManagerReport.java | 31 +-
.../ozone/container/ContainerTestHelper.java | 4 +-
hadoop-hdds/config/pom.xml | 4 +-
.../hadoop/hdds/conf/ConfigurationExample.java | 20 +-
hadoop-hdds/container-service/pom.xml | 4 +-
.../ozone/HddsDatanodeClientProtocolServer.java | 8 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 12 +-
.../org/apache/hadoop/ozone/audit/DNAction.java | 3 +-
.../checksum/ContainerChecksumTreeManager.java | 273 +-
.../container/checksum/ContainerDiffReport.java | 42 +-
.../checksum/ContainerMerkleTreeMetrics.java | 7 +
.../checksum/ContainerMerkleTreeWriter.java | 147 +-
.../container/checksum/ReconcileContainerTask.java | 6 +-
.../helpers/BlockDeletingServiceMetrics.java | 12 +
.../container/common/helpers/ContainerUtils.java | 25 +-
.../common/impl/BlockDeletingService.java | 3 +
.../ozone/container/common/impl/ContainerData.java | 34 +-
.../container/common/impl/ContainerDataYaml.java | 4 +-
.../ozone/container/common/impl/ContainerSet.java | 47 +-
.../container/common/impl/HddsDispatcher.java | 91 +-
.../common/impl/OpenContainerBlockMap.java | 148 -
.../common/impl/StorageLocationReport.java | 21 +
.../ContainerDeletionChoosingPolicyTemplate.java | 9 +-
.../common/interfaces/ContainerDispatcher.java | 13 +
.../ozone/container/common/interfaces/Handler.java | 26 +-
.../common/report/IncrementalReportSender.java | 11 +-
.../container/common/report/ReportManager.java | 4 +-
.../common/statemachine/DatanodeConfiguration.java | 84 +-
.../common/statemachine/DatanodeQueueMetrics.java | 9 +-
.../common/statemachine/DatanodeStateMachine.java | 33 +-
.../common/statemachine/SCMConnectionManager.java | 4 +-
.../common/statemachine/StateContext.java | 22 +-
.../commandhandler/CommandDispatcher.java | 29 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 37 +-
.../ReconcileContainerCommandHandler.java | 19 +-
.../ReconstructECContainersCommandHandler.java | 19 +-
.../ReplicateContainerCommandHandler.java | 19 +-
.../SetNodeOperationalStateCommandHandler.java | 4 +-
.../states/datanode/RunningDatanodeState.java | 44 +-
.../states/endpoint/HeartbeatEndpointTask.java | 17 +-
.../states/endpoint/VersionEndpointTask.java | 8 +-
.../transport/server/GrpcXceiverService.java | 31 +-
.../common/transport/server/XceiverServerGrpc.java | 4 +-
.../server/ratis/ContainerStateMachine.java | 24 +-
.../transport/server/ratis/DispatcherContext.java | 7 +
.../server/ratis/RatisServerConfiguration.java | 2 +-
...cs.java => BackgroundVolumeScannerMetrics.java} | 14 +-
.../ozone/container/common/volume/HddsVolume.java | 20 +
.../container/common/volume/StorageVolume.java | 14 +-
.../common/volume/StorageVolumeChecker.java | 6 +-
.../ECReconstructionCoordinatorTask.java | 6 +-
.../container/keyvalue/KeyValueContainerData.java | 28 +-
.../KeyValueContainerMetadataInspector.java | 84 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 225 +-
.../ozone/container/keyvalue/PendingDelete.java} | 36 +-
.../container/keyvalue/helpers/BlockUtils.java | 28 +-
.../helpers/KeyValueContainerLocationUtil.java | 17 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 151 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 27 +-
.../keyvalue/impl/ChunkManagerDispatcher.java | 8 +-
.../keyvalue/impl/ChunkManagerDummyImpl.java | 4 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 8 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 5 +-
.../statemachine/background/BlockDeletingTask.java | 67 +-
.../ozone/container/metadata/AbstractRDBStore.java | 44 +-
.../metadata/DatanodeSchemaThreeDBDefinition.java | 43 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 7 +-
.../ozoneimpl/ContainerScannerConfiguration.java | 16 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 53 +-
.../container/replication/ContainerImporter.java | 38 +-
.../replication/DownloadAndImportReplicator.java | 4 +-
.../replication/GrpcContainerUploader.java | 22 +-
.../replication/GrpcReplicationClient.java | 16 +-
.../container/replication/ReplicationServer.java | 28 +-
.../container/replication/ReplicationTask.java | 6 +-
.../replication/SendContainerOutputStream.java | 17 +-
.../replication/SendContainerRequestHandler.java | 16 +-
.../container/stream/DirstreamClientHandler.java | 15 +-
.../ozone/container/stream/StreamingClient.java | 18 +-
.../ScmHAFinalizeUpgradeActionDatanode.java | 5 +-
.../protocol/commands/CloseContainerCommand.java | 4 +-
.../protocol/commands/ClosePipelineCommand.java | 10 +-
.../protocol/commands/CreatePipelineCommand.java | 4 +-
.../protocol/commands/DeleteContainerCommand.java | 4 +-
.../commands/FinalizeNewLayoutVersionCommand.java | 4 +-
.../commands/ReconcileContainerCommand.java | 3 +-
.../commands/ReconstructECContainersCommand.java | 3 +-
.../commands/RefreshVolumeUsageCommand.java | 4 +-
.../commands/ReplicateContainerCommand.java | 4 +-
.../commands/SetNodeOperationalStateCommand.java | 4 +-
.../ozone/protocolPB/ReconDatanodeProtocolPB.java | 2 +-
...inerDatanodeProtocolClientSideTranslatorPB.java | 6 +-
.../StorageContainerDatanodeProtocolPB.java | 2 +-
.../hadoop/ozone/TestHddsSecureDatanodeInit.java | 12 +-
.../checksum/ContainerMerkleTreeTestUtils.java | 72 +-
.../checksum/TestContainerChecksumTreeManager.java | 378 +-
.../container/checksum/TestContainerDiff.java | 310 ++
.../checksum/TestContainerMerkleTreeWriter.java | 467 ++-
.../ozone/container/common/ContainerTestUtils.java | 4 +-
.../ozone/container/common/SCMTestUtils.java | 4 +-
.../hadoop/ozone/container/common/ScmTestMock.java | 4 +-
.../container/common/TestBlockDeletingService.java | 122 +-
.../container/common/TestDatanodeStateMachine.java | 2 +-
.../common/TestKeyValueContainerData.java | 2 +-
.../TestSchemaTwoBackwardsCompatibility.java | 2 +-
.../common/helpers/TestContainerUtils.java | 20 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 6 +-
.../container/common/impl/TestContainerSet.java | 136 +-
.../common/statemachine/TestStateContext.java | 11 +-
.../TestDeleteBlocksCommandHandler.java | 95 +-
.../TestReconcileContainerCommandHandler.java | 2 +-
.../TestReconstructECContainersCommandHandler.java | 7 +-
.../TestReplicateContainerCommandHandler.java | 3 +-
.../states/endpoint/TestHeartbeatEndpointTask.java | 75 +-
.../server/ratis/TestContainerStateMachine.java | 43 +-
.../volume/TestCapacityVolumeChoosingPolicy.java | 2 +-
.../common/volume/TestPeriodicVolumeChecker.java | 2 +-
.../volume/TestRoundRobinVolumeChoosingPolicy.java | 2 +-
...stContainerReconciliationWithMockDatanodes.java | 12 +-
.../container/keyvalue/TestKeyValueContainer.java | 4 +
.../keyvalue/TestKeyValueContainerCheck.java | 4 +-
.../container/keyvalue/TestKeyValueHandler.java | 73 +
.../container/keyvalue/TestTarContainerPacker.java | 2 +-
.../container/ozoneimpl/TestContainerReader.java | 98 +-
.../container/ozoneimpl/TestOzoneContainer.java | 44 +-
.../replication/TestGrpcContainerUploader.java | 5 +-
.../replication/TestGrpcReplicationService.java | 4 +-
.../replication/TestReplicationSupervisor.java | 4 +-
.../replication/TestSendContainerOutputStream.java | 4 +-
.../TestSendContainerRequestHandler.java | 143 +-
.../stream/TestDirstreamClientHandler.java | 32 +
.../container/stream/TestStreamingServer.java | 38 +
.../TestDatanodeUpgradeToContainerIdsTable.java | 7 +-
.../upgrade/TestDatanodeUpgradeToHBaseSupport.java | 2 +-
.../upgrade/TestDatanodeUpgradeToSchemaV3.java | 2 +-
.../TestReconstructionECContainersCommands.java | 6 +-
hadoop-hdds/crypto-api/pom.xml | 4 +-
hadoop-hdds/crypto-default/pom.xml | 4 +-
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 5 +
.../dev-support/checkstyle/suppressions.xml | 1 +
hadoop-hdds/docs/content/concept/Datanodes.md | 2 +-
.../content/concept/HDDS-13515-Design-Document.md | 425 +++
hadoop-hdds/docs/content/concept/RocksDB.md | 161 +
hadoop-hdds/docs/content/concept/flowchart.png | Bin 0 -> 1212887 bytes
.../design/distributed-tracing-OpenTelemetry.md | 466 +++
.../content/design/distributed-tracing-flow.png | Bin 0 -> 589655 bytes
.../content/design/distributed-tracing-sample.png | Bin 0 -> 292240 bytes
hadoop-hdds/docs/content/feature/Decommission.md | 6 +-
hadoop-hdds/docs/content/feature/FairCallQueue.md | 4 +-
hadoop-hdds/docs/content/feature/Quota.md | 1 -
.../docs/content/feature/S3-Tenant-Commands.md | 46 +
.../feature/Snapshot-Configuration-Properties.md | 82 +
hadoop-hdds/docs/content/feature/Snapshot.md | 90 +-
hadoop-hdds/docs/content/interface/HttpFS.md | 98 +-
hadoop-hdds/docs/content/security/SecuringS3.md | 127 +-
hadoop-hdds/docs/content/start/FromSource.md | 66 +-
hadoop-hdds/docs/content/start/Kubernetes.md | 2 +
hadoop-hdds/docs/content/tools/Admin.md | 51 +-
hadoop-hdds/docs/content/tools/LogsInOzone.md | 1 +
hadoop-hdds/docs/content/tools/Repair.md | 42 +-
hadoop-hdds/docs/pom.xml | 4 +-
hadoop-hdds/erasurecode/pom.xml | 4 +-
.../rawcoder/ByteArrayEncodingState.java | 5 +-
.../rawcoder/ByteBufferEncodingState.java | 7 +-
.../ozone/erasurecode/rawcoder/EncodingState.java | 5 +-
.../ozone/erasurecode/rawcoder/RSRawDecoder.java | 3 +-
hadoop-hdds/framework/pom.xml | 16 +-
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 2 +-
.../hdds/conf/DatanodeRatisServerConfig.java | 20 +-
.../apache/hadoop/hdds/conf/HddsConfServlet.java | 187 +-
.../java/org/apache/hadoop/hdds/fs/DUFactory.java | 2 +-
.../hdds/fs/DedicatedDiskSpaceUsageFactory.java | 2 +-
.../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 2 +-
...DiskBalancerProtocolClientSideTranslatorPB.java | 16 +-
.../hdds/protocolPB/DiskBalancerProtocolPB.java | 2 +-
.../ReconfigureProtocolClientSideTranslatorPB.java | 12 +-
.../protocolPB/ReconfigureProtocolDatanodePB.java | 2 +-
.../hdds/protocolPB/ReconfigureProtocolOmPB.java | 2 +-
.../hdds/protocolPB/ReconfigureProtocolPB.java | 2 +-
.../SCMSecurityProtocolClientSideTranslatorPB.java | 6 +-
.../hdds/protocolPB/SCMSecurityProtocolPB.java | 2 +-
.../SecretKeyProtocolClientSideTranslatorPB.java | 6 +-
.../protocolPB/SecretKeyProtocolDatanodePB.java | 2 +-
.../hdds/protocolPB/SecretKeyProtocolOmPB.java | 2 +-
.../hdds/protocolPB/SecretKeyProtocolScmPB.java | 2 +-
.../apache/hadoop/hdds/scm/client/ScmClient.java | 7 +
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 2 +-
.../protocol/StorageContainerLocationProtocol.java | 10 +
.../hadoop/hdds/scm/protocol/package-info.java | 2 +-
...lockLocationProtocolClientSideTranslatorPB.java | 4 +-
.../scm/protocolPB/ScmBlockLocationProtocolPB.java | 2 +-
...inerLocationProtocolClientSideTranslatorPB.java | 31 +-
.../StorageContainerLocationProtocolPB.java | 2 +-
.../hadoop/hdds/scm/proxy/SCMClientConfig.java | 17 +-
.../scm/proxy/SCMFailoverProxyProviderBase.java | 4 +-
.../apache/hadoop/hdds/scm/proxy/SCMProxyInfo.java | 4 +-
.../hadoop/hdds/security/OzoneSecretKey.java | 4 +-
.../hdds/security/token/BlockTokenVerifier.java | 5 +-
.../security/token/ContainerTokenIdentifier.java | 0
.../certificate/authority/DefaultCAServer.java | 32 +-
.../authority/profile/DefaultProfile.java | 4 +-
.../client/DefaultCertificateClient.java | 2 +-
.../org/apache/hadoop/hdds/server/OzoneAdmins.java | 25 +
.../server/OzoneProtocolMessageDispatcher.java | 18 +-
.../org/apache/hadoop/hdds/server/ServerUtils.java | 4 +-
.../hadoop/hdds/server/events/EventQueue.java | 125 +-
.../hadoop/hdds/server/events/EventWatcher.java | 4 +-
.../apache/hadoop/hdds/server/http/HttpConfig.java | 3 +-
.../hadoop/hdds/server/http/HttpServer2.java | 8 +-
.../hadoop/hdds/utils/DBCheckpointServlet.java | 21 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 51 +-
.../apache/hadoop/hdds/utils/HttpServletUtils.java | 211 ++
.../org/apache/hadoop/hdds/utils/LogLevel.java | 41 +-
.../hadoop/hdds/utils/RDBSnapshotProvider.java | 26 +-
.../hadoop/hdds/utils/db/CodecBufferCodec.java | 87 +
.../apache/hadoop/hdds/utils/db/CodecRegistry.java | 3 +-
.../hdds/utils/db/DBColumnFamilyDefinition.java | 4 +-
.../hadoop/hdds/utils/db/DBConfigFromFile.java | 122 +-
.../apache/hadoop/hdds/utils/db/DBDefinition.java | 8 +-
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 6 +-
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 12 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 139 +-
.../utils/db/InodeMetadataRocksDBCheckpoint.java | 150 +
.../hadoop/hdds/utils/db/RDBSstFileWriter.java | 38 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 47 +-
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 18 +-
.../hadoop/hdds/utils/db/RocksDBCheckpoint.java | 2 +
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 14 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 58 +-
.../org/apache/hadoop/hdds/utils/db/Table.java | 2 +-
.../apache/hadoop/hdds/utils/db/TableConfig.java | 22 +-
.../org/apache/hadoop/ozone/common/BlockGroup.java | 49 +-
.../apache/hadoop/ozone/common/DeletedBlock.java | 58 +
.../apache/hadoop/ozone/common/StorageInfo.java | 12 +-
.../src/main/resources/webapps/static/ozone.js | 10 +-
.../resources/webapps/static/templates/config.html | 4 +-
.../hadoop/hdds/conf/TestHddsConfServlet.java | 78 +-
.../hdds/conf/TestReconfigurationHandler.java | 16 +-
.../certificate/authority/TestDefaultCAServer.java | 63 +-
.../client/CertificateClientTestImpl.java | 16 +-
.../client/TestRootCaRotationPoller.java | 18 +-
.../certificate/utils/TestCertificateCodec.java | 6 +-
.../certificate/utils/TestRootCertificate.java | 21 +-
.../hadoop/hdds/utils/MapBackedTableIterator.java | 8 +-
.../hdds/utils/TestDecayRpcSchedulerUtil.java | 2 +-
.../hadoop/hdds/utils/TestHttpServletUtils.java | 78 +
.../hadoop/hdds/utils/db/InMemoryTestTable.java | 29 +-
.../hdds/utils/db/StringInMemoryTestTable.java | 41 +-
.../hadoop/hdds/utils/db/TestDBConfigFromFile.java | 80 +-
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 2 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 4 +-
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 20 +-
.../src/test/resources/auditlog.properties | 25 +-
hadoop-hdds/hadoop-dependency-client/pom.xml | 9 +-
hadoop-hdds/interface-admin/pom.xml | 12 +-
.../src/main/proto/ScmAdminProtocol.proto | 10 +
.../interface-admin/src/main/resources/proto.lock | 70 +-
hadoop-hdds/interface-client/pom.xml | 64 +-
.../src/main/proto/DatanodeClientProtocol.proto | 21 +-
.../src/main/proto/IpcConnectionContext.proto | 50 +
.../src/main/proto/ProtobufRpcEngine.proto | 70 +
.../src/main/proto/ProtocolInfo.proto | 89 +
.../src/main/proto/RpcHeader.proto | 184 +
.../interface-client/src/main/proto/hdds.proto | 11 +
.../interface-client/src/main/resources/proto.lock | 187 +
hadoop-hdds/interface-server/pom.xml | 29 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 3 +
.../src/main/proto/ScmServerProtocol.proto | 2 +
.../interface-server/src/main/resources/proto.lock | 61 +-
hadoop-hdds/managed-rocksdb/pom.xml | 4 +-
.../db/managed/ManagedBlockBasedTableConfig.java | 2 +-
.../db/managed/ManagedColumnFamilyOptions.java | 5 +-
.../hdds/utils/db/managed/ManagedDirectSlice.java} | 38 +-
hadoop-hdds/pom.xml | 4 +-
hadoop-hdds/rocks-native/pom.xml | 2 +-
.../src/test/resources/auditlog.properties | 25 +-
hadoop-hdds/rocksdb-checkpoint-differ/pom.xml | 4 +-
.../hadoop/hdds/utils/db/TablePrefixInfo.java | 44 +-
.../apache/hadoop/hdds/utils/db}/package-info.java | 6 +-
.../ozone/compaction/log/CompactionFileInfo.java | 5 +-
.../ozone/compaction/log/CompactionLogEntry.java | 9 +-
.../org/apache/ozone/rocksdb/util/RdbUtil.java | 28 +-
.../log => rocksdb/util}/SstFileInfo.java | 12 +-
.../ozone/rocksdb/util/SstFileSetReader.java | 36 +-
.../org/apache/ozone/rocksdiff/CompactionNode.java | 44 +-
.../apache/ozone/rocksdiff/DifferSnapshotInfo.java | 72 +-
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 371 +-
.../org/apache/ozone/rocksdiff/RocksDiffUtils.java | 102 +-
.../ozone/compaction/log/TestSstFileInfo.java | 47 +
.../ozone/rocksdb/util/TestSstFileSetReader.java | 58 +-
.../apache/ozone/rocksdiff/TestCompactionDag.java | 21 +-
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 542 ++-
.../apache/ozone/rocksdiff/TestRocksDiffUtils.java | 132 +-
hadoop-hdds/server-scm/pom.xml | 4 +-
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 23 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 20 +-
.../hadoop/hdds/scm/block/DeletedBlockLog.java | 13 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 83 +-
.../scm/block/DeletedBlockLogStateManager.java | 14 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 79 +-
.../SCMDeletedBlockTransactionStatusManager.java | 187 +-
.../scm/block/ScmBlockDeletingServiceMetrics.java | 38 +-
.../scm/command/CommandStatusReportHandler.java | 6 +-
.../scm/container/balancer/ContainerBalancer.java | 38 +-
.../container/placement/metrics/SCMNodeMetric.java | 9 +-
.../placement/metrics/SCMPerformanceMetrics.java | 8 +
.../container/replication/ContainerReplicaOp.java | 28 +-
.../replication/ContainerReplicaPendingOps.java | 15 +-
.../replication/ECUnderReplicationHandler.java | 6 +-
.../QuasiClosedStuckUnderReplicationHandler.java | 7 +-
.../container/replication/ReplicationManager.java | 50 +-
.../replication/ReplicationManagerUtil.java | 25 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 5 +
.../hadoop/hdds/scm/ha/InterSCMGrpcClient.java | 6 +-
.../hdds/scm/ha/InterSCMGrpcProtocolService.java | 6 +-
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 5 -
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 10 +-
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 6 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 5 +-
.../hadoop/hdds/scm/ha/SCMServiceManager.java | 4 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 10 +-
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 13 +-
.../scm/ha/StatefulServiceStateManagerImpl.java | 6 +-
.../hadoop/hdds/scm/ha/io/BigIntegerCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/EnumCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/IntegerCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/LongCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/StringCodec.java | 4 +-
.../hdds/scm/ha/io/X509CertificateCodec.java | 4 +-
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 3 +-
.../hadoop/hdds/scm/node/DatanodeUsageInfo.java | 10 +
.../hdds/scm/node/HealthyReadOnlyNodeHandler.java | 6 +-
.../hadoop/hdds/scm/node/NodeReportHandler.java | 8 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 65 +-
.../hadoop/hdds/scm/node/SCMNodeMetrics.java | 9 +
.../hdds/scm/node/SCMNodeStorageStatMap.java | 17 +-
.../hadoop/hdds/scm/node/states/ReportResult.java | 6 +-
.../pipeline/InsufficientDatanodesException.java | 2 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 5 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 8 +-
.../scm/pipeline/PipelineStateManagerImpl.java | 4 +-
.../hadoop/hdds/scm/pipeline/PipelineStateMap.java | 51 +-
.../scm/pipeline/WritableECContainerProvider.java | 4 +-
...inerLocationProtocolServerSideTranslatorPB.java | 22 +
.../safemode/AbstractContainerSafeModeRule.java | 204 +
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 4 +-
.../hdds/scm/safemode/ECContainerSafeModeRule.java | 220 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 7 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 6 +-
.../scm/safemode/RatisContainerSafeModeRule.java | 176 +-
.../hadoop/hdds/scm/safemode/SafeModeExitRule.java | 4 +-
.../hadoop/hdds/scm/safemode/SafeModeMetrics.java | 18 +-
.../hdds/scm/safemode/SafeModeRuleFactory.java | 13 +
.../hdds/scm/safemode/StateMachineReadyRule.java | 68 +
.../scm/server/OzoneStorageContainerManager.java | 9 +
.../hdds/scm/server/SCMBlockProtocolServer.java | 18 +-
.../hadoop/hdds/scm/server/SCMCertStore.java | 4 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 50 +-
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 6 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 12 +-
.../hdds/scm/server/SCMHTTPServerConfig.java | 4 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 6 +-
.../hdds/scm/server/StorageContainerManager.java | 35 +-
.../server/upgrade/FinalizationManagerImpl.java | 16 +-
.../upgrade/FinalizationStateManagerImpl.java | 8 +-
.../upgrade/SCMUpgradeFinalizationContext.java | 16 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 12 +-
.../hdds/scm/TestSCMCommonPlacementPolicy.java | 52 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 2 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 184 +-
.../scm/block/TestSCMBlockDeletingService.java | 2 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 4 +-
.../scm/container/TestContainerReportHandler.java | 2 +-
.../scm/container/TestContainerStateManager.java | 2 +-
.../TestIncrementalContainerReportHandler.java | 2 +-
.../container/balancer/TestContainerBalancer.java | 9 +-
.../scm/container/balancer/TestMoveManager.java | 32 +-
.../replication/TestECContainerReplicaCount.java | 32 +-
.../replication/TestECMisReplicationHandler.java | 10 +-
.../replication/TestECOverReplicationHandler.java | 4 +-
.../replication/TestECUnderReplicationHandler.java | 2 +-
...TestQuasiClosedStuckOverReplicationHandler.java | 9 +-
...estQuasiClosedStuckUnderReplicationHandler.java | 4 +-
.../TestRatisContainerReplicaCount.java | 8 +-
.../TestRatisMisReplicationHandler.java | 10 +-
.../TestRatisOverReplicationHandler.java | 12 +-
.../TestRatisUnderReplicationHandler.java | 24 +-
.../replication/TestReplicationManager.java | 96 +-
.../replication/TestReplicationManagerMetrics.java | 10 +-
.../TestReplicationManagerScenarios.java | 6 +-
.../replication/TestReplicationManagerUtil.java | 36 +-
.../TestClosedWithUnhealthyReplicasHandler.java | 4 +-
.../health/TestClosingContainerHandler.java | 20 +-
.../health/TestDeletingContainerHandler.java | 27 +-
.../health/TestECMisReplicationCheckHandler.java | 13 +-
.../health/TestECReplicationCheckHandler.java | 33 +-
.../health/TestEmptyContainerHandler.java | 18 +-
.../health/TestMismatchedReplicasHandler.java | 30 +-
.../health/TestOpenContainerHandler.java | 26 +-
.../health/TestQuasiClosedContainerHandler.java | 30 +-
.../TestQuasiClosedStuckReplicationCheck.java | 17 +-
.../health/TestRatisReplicationCheckHandler.java | 48 +-
.../TestRatisUnhealthyReplicationCheckHandler.java | 22 +-
.../TestVulnerableUnhealthyReplicasHandler.java | 4 +-
.../hdds/scm/ha/TestSequenceIDGenerator.java | 4 +-
.../hdds/scm/ha/io/TestX509CertificateCodec.java | 4 +-
.../scm/node/DatanodeAdminMonitorTestUtil.java | 14 +
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hadoop/hdds/scm/node/TestSCMNodeMetrics.java | 3 +
...java => AbstractContainerSafeModeRuleTest.java} | 149 +-
.../scm/safemode/TestECContainerSafeModeRule.java | 185 +-
.../safemode/TestRatisContainerSafeModeRule.java | 178 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 3 +-
.../scm/security/TestRootCARotationManager.java | 14 +-
.../ozone/container/common/TestEndPoint.java | 2 +-
hadoop-hdds/test-utils/pom.xml | 9 +-
.../org/apache/ozone/test/GenericTestUtils.java | 23 +-
.../org/apache/ozone/test/LambdaTestUtils.java | 3 +-
hadoop-ozone/cli-admin/pom.xml | 4 +-
.../hdds/scm/cli/ContainerOperationClient.java | 16 +-
.../hadoop/hdds/scm/cli/cert/InfoSubcommand.java | 4 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 4 +-
.../hdds/scm/cli/container/ReportSubcommand.java | 2 +-
.../hdds/scm/cli/datanode/BasicDatanodeInfo.java | 72 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 32 +-
.../hdds/scm/cli/datanode/UsageInfoSubcommand.java | 16 +-
.../ozone/admin/om/CancelPrepareSubCommand.java | 19 +-
.../ozone/admin/om/DecommissionOMSubcommand.java | 16 +-
.../hadoop/ozone/admin/om/FetchKeySubCommand.java | 12 +-
.../admin/om/FinalizationStatusSubCommand.java | 28 +-
.../ozone/admin/om/FinalizeUpgradeSubCommand.java | 25 +-
.../ozone/admin/om/GetServiceRolesSubcommand.java | 18 +-
.../ozone/admin/om/ListOpenFilesSubCommand.java | 80 +-
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 41 +-
.../hadoop/ozone/admin/om/OmAddressOptions.java | 189 +
.../hadoop/ozone/admin/om/PrepareSubCommand.java | 24 +-
.../ozone/admin/om/TransferOmLeaderSubCommand.java | 20 +-
.../ozone/admin/om/UpdateRangerSubcommand.java | 31 +-
.../ozone/admin/om/snapshot/DefragSubCommand.java | 109 +
.../admin/om/snapshot/SnapshotSubCommand.java | 18 +-
.../ozone/admin/om/snapshot}/package-info.java | 6 +-
.../ozone/admin/scm/DeletedBlocksTxnCommands.java} | 20 +-
.../scm/GetDeletedBlockSummarySubcommand.java | 54 +
.../apache/hadoop/ozone/admin/scm/ScmAdmin.java | 3 +-
.../scm/cli/container/TestReportSubCommand.java | 6 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 45 +
.../admin/om/snapshot/TestDefragSubCommand.java | 162 +
hadoop-ozone/cli-shell/pom.xml | 4 +-
.../hadoop/ozone/shell/ListLimitOptions.java | 2 +-
hadoop-ozone/client/pom.xml | 4 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 4 +-
.../hadoop/ozone/client/OzoneClientFactory.java | 18 +-
.../ozone/client/OzoneMultipartUploadList.java | 4 +-
.../apache/hadoop/ozone/client/OzoneSnapshot.java | 20 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 4 +-
.../org/apache/hadoop/ozone/client/TenantArgs.java | 4 +-
.../client/checksum/ECBlockChecksumComputer.java | 5 +-
.../client/io/BlockDataStreamOutputEntryPool.java | 5 +-
.../client/io/BlockOutputStreamEntryPool.java | 5 +-
.../hadoop/ozone/client/io/ECKeyOutputStream.java | 3 +-
.../ozone/client/io/KeyDataStreamOutput.java | 5 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 5 +-
.../hadoop/ozone/client/io/OzoneOutputStream.java | 42 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 77 +-
.../hadoop/ozone/client/TestOzoneSnapshot.java | 3 +-
.../client/checksum/TestFileChecksumHelper.java | 6 +-
.../ozone/client/io/TestOzoneOutputStream.java | 152 +
hadoop-ozone/common/pom.xml | 8 +-
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 6 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 22 +-
.../java/org/apache/hadoop/ozone/OzoneAcl.java | 4 +-
.../apache/hadoop/ozone/conf/OMClientConfig.java | 8 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 34 +-
.../java/org/apache/hadoop/ozone/om/OmConfig.java | 129 +-
.../om/ha/HadoopRpcOMFailoverProxyProvider.java | 2 +-
.../ozone/om/ha/OMFailoverProxyProviderBase.java | 18 +-
.../hadoop/ozone/om/helpers/AclListBuilder.java | 94 +
.../hadoop/ozone/om/helpers/BasicOmKeyInfo.java | 15 +-
.../hadoop/ozone/om/helpers/OmBucketArgs.java | 22 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 157 +-
.../hadoop/ozone/om/helpers/OmDirectoryInfo.java | 22 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 183 +-
.../ozone/om/helpers/OmMultipartKeyInfo.java | 40 +-
.../hadoop/ozone/om/helpers/OmRangerSyncArgs.java | 4 +-
.../hadoop/ozone/om/helpers/OmTenantArgs.java | 6 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 226 +-
.../hadoop/ozone/om/helpers/OzoneAclUtil.java | 9 +
.../ozone/om/helpers/OzoneIdentityProvider.java | 8 +-
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 46 +-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 6 +-
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 88 +-
.../hadoop/ozone/om/helpers/WithMetadata.java | 24 +-
.../hadoop/ozone/om/helpers/WithObjectID.java | 108 +-
.../ozone/om/helpers/WithParentObjectId.java | 6 +-
.../hadoop/ozone/om/lock/DAGLeveledResource.java | 84 +
.../ozone/om/lock/DAGResourceLockTracker.java | 169 +
.../om/lock/HierarchicalResourceLockManager.java | 78 +
.../hadoop/ozone/om/lock/IOzoneManagerLock.java | 49 +
.../ozone/om/lock/LeveledResourceLockTracker.java | 94 +
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 179 +-
.../PoolBasedHierarchicalResourceLockManager.java | 286 ++
.../ReadOnlyHierarchicalResourceLockManager.java | 75 +
.../hadoop/ozone/om/lock/ResourceLockTracker.java | 55 +
.../hadoop/ozone/om/protocol/OMAdminProtocol.java | 8 +
.../ozone/om/protocolPB/GrpcOmTransport.java | 4 +-
.../ozone/om/protocolPB/Hadoop3OmTransport.java | 6 +-
.../protocolPB/OMAdminProtocolClientSideImpl.java | 34 +-
.../ozone/om/protocolPB/OMAdminProtocolPB.java | 2 +-
.../OMInterServiceProtocolClientSideImpl.java | 6 +-
.../om/protocolPB/OMInterServiceProtocolPB.java | 2 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 11 +-
.../om/protocolPB/OzoneManagerProtocolPB.java | 2 +-
.../apache/hadoop/ozone/protocolPB/OMPBHelper.java | 11 +-
.../hadoop/ozone/security/GDPRSymmetricKey.java | 5 +-
.../ozone/security/acl/AssumeRoleRequest.java | 127 +
.../ozone/security/acl/IAccessAuthorizer.java | 29 +-
.../apache/hadoop/ozone/security/acl/OzoneObj.java | 22 +-
.../hadoop/ozone/security/acl/OzoneObjInfo.java | 19 +
.../hadoop/ozone/security/acl/RequestContext.java | 46 +-
.../apache/hadoop/ozone/util/OzoneVersionInfo.java | 56 +-
.../org/apache/hadoop/ozone/util/PayloadUtils.java | 4 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 4 +-
.../main/resources/ozone-version-info.properties | 4 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 70 +-
.../ozone/om/helpers/TestAclListBuilder.java | 143 +
.../hadoop/ozone/om/helpers/TestOMNodeDetails.java | 429 +++
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 6 +-
.../ozone/om/helpers/TestOmSnapshotInfo.java | 8 -
.../hadoop/ozone/om/helpers/TestOmVolumeArgs.java | 98 +-
.../om/helpers/TestOzoneIdentityProvider.java | 4 +-
.../ozone/om/lock/TestDAGLeveledResource.java | 43 +
.../hadoop/ozone/om/lock/TestOzoneManagerLock.java | 37 +-
...stPoolBasedHierarchicalResourceLockManager.java | 724 ++++
.../ozone/security/acl/TestAssumeRoleRequest.java | 71 +
hadoop-ozone/csi/pom.xml | 27 +-
.../org/apache/hadoop/ozone/csi/CsiServer.java | 10 +-
hadoop-ozone/datanode/pom.xml | 4 +-
hadoop-ozone/dev-support/checks/_lib.sh | 8 +-
hadoop-ozone/dev-support/checks/acceptance.sh | 14 +-
hadoop-ozone/dev-support/checks/bats.sh | 6 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 2 +-
hadoop-ozone/dev-support/checks/docs.sh | 6 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 8 +-
hadoop-ozone/dev-support/checks/install/hugo.sh | 2 +-
hadoop-ozone/dev-support/checks/install/k3s.sh | 2 +-
.../dev-support/checks/{pmd.sh => javadoc.sh} | 17 +-
hadoop-ozone/dev-support/checks/junit.sh | 11 +-
hadoop-ozone/dev-support/checks/kubernetes.sh | 21 +-
hadoop-ozone/dev-support/checks/license.sh | 2 +-
hadoop-ozone/dev-support/checks/pmd.sh | 2 +-
hadoop-ozone/dev-support/checks/sonar.sh | 2 +-
hadoop-ozone/dist/pom.xml | 22 +-
.../dist/src/main/compose/common/hadoop-test.sh | 7 +-
.../src/main/compose/compatibility/docker-config | 1 +
.../dist/src/main/compose/ozone-csi/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/ozone-ha/.env | 2 +-
.../dist/src/main/compose/ozone-ha/docker-config | 2 +
.../src/main/compose/ozone-om-ha/docker-config | 8 +-
.../main/compose/ozone-om-prepare/docker-config | 1 +
.../src/main/compose/ozone-topology/docker-config | 2 +
hadoop-ozone/dist/src/main/compose/ozone/.env | 2 +-
.../dist/src/main/compose/ozone/docker-config | 2 +
.../dist/src/main/compose/ozone/monitoring.conf | 2 +-
.../dist/src/main/compose/ozone/monitoring.yaml | 4 +-
.../src/main/compose/ozoneblockade/docker-config | 1 +
.../src/main/compose/ozonescripts/docker-config | 1 +
.../dist/src/main/compose/ozonesecure-ha/.env | 4 +-
.../src/main/compose/ozonesecure-ha/docker-config | 2 +
.../compose/ozonesecure-ha/test-repair-tools.sh | 3 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 4 +-
.../src/main/compose/ozonesecure-mr/docker-config | 2 +
.../dist/src/main/compose/ozonesecure/.env | 4 +-
.../src/main/compose/ozonesecure/docker-config | 3 +
.../dist/src/main/compose/ozonesecure/fcq.yaml | 2 +-
.../dist/src/main/compose/ozonesecure/test.sh | 1 +
.../dist/src/main/compose/restart/docker-config | 1 +
.../dist/src/main/compose/upgrade/compose/ha/.env | 4 +-
.../main/compose/upgrade/compose/ha/docker-config | 2 +
.../compose/upgrade/compose/non-ha/docker-config | 2 +
hadoop-ozone/dist/src/main/compose/xcompat/.env | 4 +-
.../dist/src/main/compose/xcompat/docker-config | 1 +
.../dist/src/main/compose/xcompat/new-cluster.yaml | 1 +
.../dist/src/main/compose/xcompat/test-new.sh | 3 +-
.../dist/src/main/compose/xcompat/test-old.sh | 3 +-
.../dist/src/main/dockerlibexec/envtoconf.py | 2 +-
.../dist/src/main/dockerlibexec/transformation.py | 2 +-
hadoop-ozone/dist/src/main/k8s/README.md | 2 +
.../src/main/k8s/definitions/jaeger/jaeger.yaml | 5 +-
.../main/k8s/definitions/ozone-csi/csi-crd.yaml | 2 +-
.../src/main/k8s/definitions/prometheus/role.yaml | 2 +-
.../k8s/definitions/prometheus/rolebinding.yaml | 2 +-
.../csi/org.apache.hadoop.ozone-csidriver.yaml | 2 +-
.../k8s/examples/ozone-dev/jaeger-statefulset.yaml | 5 +-
.../examples/ozone-dev/prometheus-clusterrole.yaml | 2 +-
.../prometheus-operator-clusterrolebinding.yaml | 2 +-
.../csi/org.apache.hadoop.ozone-csidriver.yaml | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 1 -
.../src/main/smoketest/admincli/datanode.robot | 1 +
.../dist/src/main/smoketest/cli/envvars.robot | 19 +
.../src/main/smoketest/cli/ozone-insight.robot | 59 +
.../src/main/smoketest/compatibility/read.robot | 13 +
.../smoketest/debug/ozone-debug-keywords.robot | 20 +
.../src/main/smoketest/ec/awss3ecstorage.robot | 16 +-
.../main/smoketest/recon/recon-taskstatus.robot | 6 +-
.../dist/src/main/smoketest/s3/mpu_lib.robot | 1 +
.../dist/src/shell/conf/dn-audit-log4j2.properties | 30 +-
.../src/shell/conf/dn-container-log4j2.properties | 26 +-
.../dist/src/shell/conf/om-audit-log4j2.properties | 36 +-
.../src/shell/conf/s3g-audit-log4j2.properties | 30 +-
.../src/shell/conf/scm-audit-log4j2.properties | 30 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 14 +-
.../fault-injection-test/mini-chaos-tests/pom.xml | 4 +-
.../fault-injection-test/network-tests/pom.xml | 2 +-
.../network-tests/src/test/compose/docker-config | 7 +-
hadoop-ozone/fault-injection-test/pom.xml | 4 +-
hadoop-ozone/freon/pom.xml | 8 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 9 +-
.../hadoop/ozone/freon/DatanodeSimulator.java | 4 +-
.../apache/hadoop/ozone/freon/FollowerReader.java | 96 +
.../hadoop/ozone/freon/SCMThroughputBenchmark.java | 4 +-
.../freon/containergenerator/GeneratorOm.java | 4 +-
hadoop-ozone/httpfsgateway/pom.xml | 4 +-
.../httpfsgateway/src/site/configuration.xsl | 49 -
.../httpfsgateway/src/site/resources/css/site.css | 30 -
hadoop-ozone/httpfsgateway/src/site/site.xml | 29 -
hadoop-ozone/insight/pom.xml | 15 +-
.../ozone/insight/BaseInsightSubCommand.java | 105 +-
.../hadoop/ozone/insight/InsightHttpUtils.java | 118 +
.../apache/hadoop/ozone/insight/LogSubcommand.java | 41 +-
.../hadoop/ozone/insight/MetricsSubCommand.java | 30 +-
.../ozone/insight/TestBaseInsightSubCommand.java | 102 +
hadoop-ozone/integration-test-recon/pom.xml | 9 +-
.../TestReconInsightsForDeletedDirectories.java | 110 +-
.../hadoop/ozone/recon/TestReconScmSnapshot.java | 9 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 4 +-
.../recon/TestStorageDistributionEndpoint.java | 289 ++
hadoop-ozone/integration-test-s3/pom.xml | 4 +-
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 55 +-
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 10 +-
hadoop-ozone/integration-test/pom.xml | 4 +-
.../AbstractContractMultipartUploaderTest.java | 38 +-
.../fs/ozone/AbstractOzoneFileSystemTest.java | 2 +-
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 28 +-
.../hadoop/fs/ozone/TestOzoneFsSnapshot.java | 2 +-
.../java/org/apache/hadoop/hdds/TestRemoteEx.java | 2 +-
.../apache/hadoop/hdds/scm/TestCommitInRatis.java | 11 -
.../hadoop/hdds/scm/TestFailoverWithSCMHA.java | 8 -
.../hdds/scm/TestSCMDbCheckpointServlet.java | 12 +-
.../hdds/scm/TestSCMInstallSnapshotWithHA.java | 8 -
.../apache/hadoop/hdds/scm/TestSecretKeysApi.java | 2 +-
.../hdds/scm/TestStorageContainerManager.java | 36 +-
.../apache/hadoop/hdds/scm/TestWatchForCommit.java | 12 -
.../TestReplicationManagerIntegration.java | 111 +-
.../hdds/scm/pipeline/TestMultiRaftSetup.java | 3 -
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 9 -
.../hdds/scm/pipeline/TestPipelineClose.java | 8 -
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 8 -
.../hadoop/hdds/scm/storage/TestCommitWatcher.java | 10 -
.../TestDNDataDistributionFinalization.java | 326 ++
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 8 -
.../TestScmDataDistributionFinalization.java | 449 +++
.../ozone/TestContainerBalancerOperations.java | 63 +-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 37 +-
.../apache/hadoop/ozone/TestDelegationToken.java | 2 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 26 +-
.../ozone/client/rpc/OzoneRpcClientTests.java | 187 +-
.../apache/hadoop/ozone/client/rpc/TestBCSID.java | 9 -
.../ozone/client/rpc/TestBlockOutputStream.java | 640 ++--
.../rpc/TestBlockOutputStreamWithFailures.java | 850 +++--
.../rpc/TestContainerReplicationEndToEnd.java | 9 -
.../client/rpc/TestContainerStateMachine.java | 8 -
.../rpc/TestContainerStateMachineFailures.java | 72 +-
.../rpc/TestContainerStateMachineFlushDelay.java | 9 -
.../client/rpc/TestDeleteWithInAdequateDN.java | 9 -
.../ozone/client/rpc/TestECKeyOutputStream.java | 6 -
.../client/rpc/TestFailureHandlingByClient.java | 16 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 11 -
.../client/rpc/TestHybridPipelineOnDatanode.java | 10 -
.../rpc/TestMultiBlockWritesWithDnFailures.java | 11 -
...estOzoneClientRetriesOnExceptionFlushDelay.java | 11 -
.../rpc/TestOzoneClientRetriesOnExceptions.java | 10 -
.../rpc/TestOzoneRpcClientForAclAuditLog.java | 12 -
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 11 +-
.../client/rpc/TestValidateBCSIDOnRestart.java | 9 -
.../rpc/read/TestStreamBlockInputStream.java | 283 ++
.../ozone/client/rpc/read/TestStreamRead.java | 225 ++
.../ozone/container/TestECContainerRecovery.java | 6 -
.../TestCloseContainerByPipeline.java | 10 -
.../replication/TestContainerReplication.java | 147 +
.../TestContainerCommandReconciliation.java | 3 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 2 +-
.../TestDataValidateWithSafeByteOperations.java | 9 -
.../TestDataValidateWithUnsafeByteOperations.java | 9 -
.../freon/TestFreonWithDatanodeFastRestart.java | 9 -
.../ozone/freon/TestFreonWithPipelineDestroy.java | 9 -
.../hadoop/ozone/freon/TestOMSnapshotDAG.java | 117 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 319 +-
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 2 +-
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 12 +-
.../TestOMDbCheckpointServletInodeBasedXfer.java | 449 ++-
.../om/TestOMHALeaderSpecificACLEnforcement.java | 427 +++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 129 +-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 20 +-
.../org/apache/hadoop/ozone/om/TestOmInit.java | 10 -
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 6 -
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 12 +-
.../om/TestOzoneManagerHAWithStoppedNodes.java | 4 +-
.../ozone/om/TestOzoneManagerListVolumes.java | 8 +-
.../om/TestOzoneManagerListVolumesSecure.java | 7 +-
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 11 -
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 3 -
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 11 -
.../om/multitenant/TestMultiTenantVolume.java | 4 +-
.../ozone/om/service/TestBlockDeletionService.java | 236 ++
.../TestDirectoryDeletingServiceWithFSO.java | 24 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 29 +-
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 7 +-
.../om/snapshot/TestOmSnapshotFileSystem.java | 4 +-
.../snapshot/TestOmSnapshotFsoWithNativeLib.java | 2 +
.../om/snapshot/TestOmSnapshotObjectStore.java | 3 +
...TestOmSnapshotObjectStoreWithLinkedBuckets.java | 3 +
.../TestOmSnapshotWithoutBucketLinkingLegacy.java | 3 +
.../om/snapshot/TestOzoneManagerHASnapshot.java | 2 +-
.../om/snapshot/TestOzoneManagerSnapshotAcl.java | 2 +-
.../snapshot/TestOzoneManagerSnapshotProvider.java | 23 +-
.../om/snapshot/TestOzoneSnapshotRestore.java | 2 +-
.../snapshot/TestSnapshotBackgroundServices.java | 16 +-
.../ozone/om/snapshot/TestSnapshotDefragAdmin.java | 222 ++
.../ozone/parser/TestOzoneHARatisLogParser.java | 4 +-
.../reconfig/TestDatanodeReconfiguration.java | 8 +-
.../ozone/reconfig/TestOmReconfiguration.java | 28 +-
.../ozone/reconfig/TestScmReconfiguration.java | 19 +-
.../ozone/shell/TestDeletedBlocksTxnShell.java | 200 +
.../shell/TestOzoneContainerUpgradeShell.java | 4 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 27 +-
.../ozone/shell/TestOzoneShellHAWithFSO.java | 4 -
.../shell/TestOzoneShellHAWithFollowerRead.java | 99 +
.../hadoop/ozone/shell/TestOzoneTenantShell.java | 6 -
.../hadoop/ozone/shell/TestReconfigShell.java | 2 +-
.../ozone/shell/TestTransferLeadershipShell.java | 9 -
.../src/test/resources/auditlog.properties | 25 +-
hadoop-ozone/interface-client/pom.xml | 55 +-
.../src/main/proto/OMAdminProtocol.proto | 14 +
.../src/main/proto/OmClientProtocol.proto | 24 +-
.../interface-client/src/main/proto/Security.proto | 1 -
.../interface-client/src/main/resources/proto.lock | 357 +-
hadoop-ozone/interface-storage/pom.xml | 11 +-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 80 +-
.../hadoop/ozone/om/helpers/OmPrefixInfo.java | 34 +-
.../src/main/proto/OmStorageProtocol.proto | 10 +
.../hadoop/ozone/om/helpers/TestOmPrefixInfo.java | 4 +-
.../om/helpers/TestRepeatedOmKeyInfoCodec.java | 12 +-
hadoop-ozone/mini-cluster/pom.xml | 8 +-
hadoop-ozone/multitenancy-ranger/pom.xml | 4 +-
.../dev-support/findbugsExcludeFile.xml | 4 +
hadoop-ozone/ozone-manager/pom.xml | 4 +-
.../apache/hadoop/ozone/audit/OMSystemAction.java | 8 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 7 +-
.../hadoop/ozone/om/BucketUtilizationMetrics.java | 8 +-
.../apache/hadoop/ozone/om/DeleteKeysResult.java | 10 +-
.../hadoop/ozone/om/DeletingServiceMetrics.java | 24 +
.../org/apache/hadoop/ozone/om/KeyManager.java | 20 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 232 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 67 +-
.../om/OMDBCheckpointServletInodeBasedXfer.java | 310 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 47 +
.../hadoop/ozone/om/OMMultiTenantManagerImpl.java | 11 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 218 +-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 10 +-
.../hadoop/ozone/om/OmSnapshotLocalData.java | 128 +-
.../hadoop/ozone/om/OmSnapshotLocalDataYaml.java | 171 +-
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 321 +-
.../hadoop/ozone/om/OzoneListStatusHelper.java | 6 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 356 +-
.../hadoop/ozone/om/OzoneManagerServiceGrpc.java | 6 +-
.../hadoop/ozone/om/PendingKeysDeletion.java | 89 +-
.../hadoop/ozone/om/SstFilteringService.java | 36 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 2 +-
.../hadoop/ozone/om/execution/OMExecutionFlow.java | 36 +-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 6 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 29 +
.../ozone/om/ratis/OzoneManagerRatisServer.java | 34 +-
.../om/ratis/OzoneManagerRatisServerConfig.java | 41 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 21 +
.../om/ratis_snapshot/OmRatisSnapshotProvider.java | 8 +
.../hadoop/ozone/om/request/OMClientRequest.java | 14 +-
.../om/request/bucket/OMBucketCreateRequest.java | 38 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 4 +-
.../om/request/bucket/OMBucketSetOwnerRequest.java | 20 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 8 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 5 +-
.../om/request/file/OMDirectoryCreateRequest.java | 4 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 4 +-
.../request/file/OMFileCreateRequestWithFSO.java | 3 +-
.../ozone/om/request/file/OMFileRequest.java | 8 +-
.../om/request/file/OMRecoverLeaseRequest.java | 12 +-
.../om/request/key/OMAllocateBlockRequest.java | 10 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 11 +-
.../key/OMDirectoriesPurgeRequestWithFSO.java | 214 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 79 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 75 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 4 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 3 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 17 +-
.../om/request/key/OMKeyDeleteRequestWithFSO.java | 14 +-
.../ozone/om/request/key/OMKeyPurgeRequest.java | 108 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 8 +-
.../om/request/key/OMKeyRenameRequestWithFSO.java | 41 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 86 +-
.../ozone/om/request/key/OMKeySetTimesRequest.java | 29 +-
.../request/key/OMKeySetTimesRequestWithFSO.java | 11 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 28 +-
.../ozone/om/request/key/OMKeysRenameRequest.java | 4 +-
.../om/request/key/OMOpenKeysDeleteRequest.java | 44 +-
.../om/request/key/OmKeysDeleteRequestWithFSO.java | 81 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 4 +-
.../om/request/key/acl/OMKeyAclRequestWithFSO.java | 5 +-
.../request/key/acl/prefix/OMPrefixAclRequest.java | 4 +-
.../S3ExpiredMultipartUploadsAbortRequest.java | 12 +-
.../S3InitiateMultipartUploadRequest.java | 6 +-
.../S3InitiateMultipartUploadRequestWithFSO.java | 6 +-
.../multipart/S3MultipartUploadAbortRequest.java | 4 +-
.../S3MultipartUploadCommitPartRequest.java | 31 +-
.../S3MultipartUploadCommitPartRequestWithFSO.java | 4 +-
.../S3MultipartUploadCompleteRequest.java | 21 +-
.../S3MultipartUploadCompleteRequestWithFSO.java | 5 +-
.../request/s3/security/S3SecretRequestHelper.java | 2 +-
.../s3/tagging/S3DeleteObjectTaggingRequest.java | 8 +-
.../S3DeleteObjectTaggingRequestWithFSO.java | 4 +-
.../s3/tagging/S3PutObjectTaggingRequest.java | 8 +-
.../tagging/S3PutObjectTaggingRequestWithFSO.java | 4 +-
.../s3/tenant/OMTenantAssignAdminRequest.java | 4 +-
.../tenant/OMTenantAssignUserAccessIdRequest.java | 3 +-
.../request/s3/tenant/OMTenantCreateRequest.java | 36 +-
.../request/s3/tenant/OMTenantDeleteRequest.java | 13 +-
.../s3/tenant/OMTenantRevokeAdminRequest.java | 4 +-
.../tenant/OMTenantRevokeUserAccessIdRequest.java | 10 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 7 -
.../snapshot/OMSnapshotMoveDeletedKeysRequest.java | 72 +-
.../snapshot/OMSnapshotMoveTableKeysRequest.java | 140 +-
.../request/snapshot/OMSnapshotPurgeRequest.java | 29 +-
.../snapshot/OMSnapshotSetPropertyRequest.java | 56 +-
.../hadoop/ozone/om/request/util/ObjectParser.java | 4 +-
.../om/request/volume/OMQuotaRepairRequest.java | 14 +-
.../om/request/volume/OMVolumeCreateRequest.java | 58 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 4 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 15 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 21 +-
.../om/request/volume/acl/OMVolumeAclRequest.java | 37 +-
.../request/volume/acl/OMVolumeAddAclRequest.java | 14 +-
.../volume/acl/OMVolumeRemoveAclRequest.java | 14 +-
.../request/volume/acl/OMVolumeSetAclRequest.java | 16 +-
.../hadoop/ozone/om/response/OMClientResponse.java | 3 +-
.../response/key/AbstractOMKeyDeleteResponse.java | 39 +-
.../key/OMDirectoriesPurgeResponseWithFSO.java | 28 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 2 +-
.../response/key/OMKeyDeleteResponseWithFSO.java | 2 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 31 +-
.../om/response/key/OMKeysDeleteResponse.java | 2 +-
.../response/key/OMKeysDeleteResponseWithFSO.java | 2 +-
.../om/response/key/OMOpenKeysDeleteResponse.java | 9 +-
.../AbstractS3MultipartAbortResponse.java | 5 +-
.../S3MultipartUploadCommitPartResponse.java | 8 +-
...S3MultipartUploadCommitPartResponseWithFSO.java | 4 +-
.../S3MultipartUploadCompleteResponse.java | 8 +-
.../S3MultipartUploadCompleteResponseWithFSO.java | 2 +-
.../response/s3/tenant/OMTenantDeleteResponse.java | 3 +-
.../OMSnapshotMoveDeletedKeysResponse.java | 94 +-
.../snapshot/OMSnapshotMoveTableKeysResponse.java | 22 +-
.../response/snapshot/OMSnapshotPurgeResponse.java | 58 +-
.../om/service/AbstractKeyDeletingService.java | 63 +-
.../ozone/om/service/DirectoryDeletingService.java | 177 +-
.../ozone/om/service/KeyDeletingService.java | 217 +-
.../ozone/om/service/OMRangerBGSyncService.java | 3 +-
.../hadoop/ozone/om/service/QuotaRepairTask.java | 16 +-
.../ozone/om/service/SnapshotDeletingService.java | 22 +-
.../ozone/om/snapshot/MultiSnapshotLocks.java | 14 +-
.../ozone/om/snapshot/OMDBCheckpointUtils.java | 4 +-
.../om/snapshot/OmSnapshotLocalDataManager.java | 1083 ++++++
.../hadoop/ozone/om/snapshot/OmSnapshotUtils.java | 79 +-
.../hadoop/ozone/om/snapshot/SnapshotCache.java | 222 +-
.../ozone/om/snapshot/SnapshotDiffManager.java | 286 +-
.../hadoop/ozone/om/snapshot/SnapshotUtils.java | 42 +-
.../om/snapshot/defrag/SnapshotDefragService.java | 740 ++++
.../ozone/om/snapshot/defrag}/package-info.java | 6 +-
.../diff/delta/CompositeDeltaDiffComputer.java | 130 +
.../om/snapshot/diff/delta/DeltaFileComputer.java | 50 +
.../diff/delta/FileLinkDeltaFileComputer.java | 156 +
.../om/snapshot/diff/delta/FullDiffComputer.java | 119 +
.../om/snapshot/diff/delta/RDBDifferComputer.java | 117 +
.../om/snapshot/diff/delta}/package-info.java | 6 +-
.../om/snapshot/filter/ReclaimableFilter.java | 5 +-
.../ozone/om/snapshot/util/TableMergeIterator.java | 109 +
.../ozone/om/snapshot/util}/package-info.java | 6 +-
.../hadoop/ozone/om/upgrade/OMLayoutFeature.java | 3 +-
.../protocolPB/OMAdminProtocolServerSideImpl.java | 20 +
...OzoneManagerProtocolServerSideTranslatorPB.java | 111 +-
.../ozone/security/acl/OzoneNativeAuthorizer.java | 2 +-
.../ozone/om/ScmBlockLocationTestingClient.java | 51 +-
.../hadoop/ozone/om/TestBucketManagerImpl.java | 6 +
.../ozone/om/TestBucketUtilizationMetrics.java | 16 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 22 +-
.../apache/hadoop/ozone/om/TestOMDBDefinition.java | 3 +-
.../hadoop/ozone/om/TestOMMetadataReader.java | 2 +-
.../hadoop/ozone/om/TestOMTenantCreateRequest.java | 6 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 5 -
.../ozone/om/TestOmMetadataManagerMetrics.java | 260 ++
.../ozone/om/TestOmSnapshotLocalDataYaml.java | 166 +-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 155 +-
.../hadoop/ozone/om/failover/TestOMFailovers.java | 2 +-
.../ozone/om/request/OMRequestTestUtils.java | 37 +-
.../request/TestOMClientRequestWithUserInfo.java | 9 +-
.../request/bucket/TestOMBucketCreateRequest.java | 83 +-
.../request/file/TestOMDirectoryCreateRequest.java | 61 +-
.../file/TestOMDirectoryCreateRequestWithFSO.java | 68 +-
.../om/request/file/TestOMFileCreateRequest.java | 54 +
.../om/request/file/TestOMRecoverLeaseRequest.java | 24 +-
.../TestOMDirectoriesPurgeRequestAndResponse.java | 263 +-
.../om/request/key/TestOMKeyCreateRequest.java | 32 +-
.../key/TestOMKeyPurgeRequestAndResponse.java | 81 +-
.../request/key/TestOMKeyRenameRequestWithFSO.java | 4 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 6 +-
.../request/key/TestOMOpenKeysDeleteRequest.java | 79 +-
.../s3/security/TestS3GetSecretRequest.java | 6 +-
.../s3/security/TestS3SecretRequestHelper.java | 4 +-
.../snapshot/TestOMSnapshotCreateRequest.java | 11 +-
.../TestOMSnapshotPurgeRequestAndResponse.java | 24 +-
.../request/volume/TestOMVolumeCreateRequest.java | 83 +-
.../om/request/volume/TestOMVolumeRequest.java | 15 +-
.../hadoop/ozone/om/response/TestOMResponse.java | 150 +
.../om/response/key/TestOMKeyCommitResponse.java | 4 +-
.../key/TestOMKeyCommitResponseWithFSO.java | 2 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 4 +-
.../key/TestOMKeyRenameResponseWithFSO.java | 5 -
.../response/key/TestOMOpenKeysDeleteResponse.java | 34 +-
...TestS3ExpiredMultipartUploadsAbortResponse.java | 2 -
.../s3/multipart/TestS3MultipartResponse.java | 6 +-
...stS3MultipartUploadCompleteResponseWithFSO.java | 4 +-
.../snapshot/TestOMSnapshotCreateResponse.java | 22 +-
.../snapshot/TestOMSnapshotDeleteResponse.java | 22 +-
.../TestOMSnapshotMoveTableKeysResponse.java | 71 +-
.../om/service/TestDirectoryDeletingService.java | 85 +-
.../ozone/om/service/TestKeyDeletingService.java | 389 +-
.../ozone/om/service/TestQuotaRepairTask.java | 2 +-
.../ozone/om/snapshot/TestMultiSnapshotLocks.java | 21 +-
.../snapshot/TestOmSnapshotLocalDataManager.java | 1109 ++++++
.../ozone/om/snapshot/TestOmSnapshotUtils.java | 2 +-
.../ozone/om/snapshot/TestSnapshotCache.java | 101 +-
.../ozone/om/snapshot/TestSnapshotChain.java | 1 -
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 405 +-
.../hadoop/ozone/om/snapshot/TestSnapshotInfo.java | 1 -
.../snapshot/TestSnapshotRequestAndResponse.java | 8 +-
.../ozone/om/snapshot/TestSnapshotUtils.java | 9 +-
.../ozone/om/snapshot/TestSstFilteringService.java | 11 +-
.../defrag/TestInodeMetadataRocksDBCheckpoint.java | 76 +
.../snapshot/defrag/TestSnapshotDefragService.java | 939 +++++
.../diff/delta/TestCompositeDeltaDiffComputer.java | 725 ++++
.../diff/delta/TestFileLinkDeltaFileComputer.java | 449 +++
.../snapshot/diff/delta/TestFullDiffComputer.java | 338 ++
.../snapshot/diff/delta/TestRDBDifferComputer.java | 531 +++
.../filter/AbstractReclaimableFilterTest.java | 17 +-
.../om/snapshot/util/TestTableMergeIterator.java | 481 +++
.../ozone/security/acl/OzoneNativeAclTestUtil.java | 14 +-
.../ozone/security/acl/TestRequestContext.java | 37 +-
hadoop-ozone/ozonefs-common/pom.xml | 4 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 18 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 6 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 18 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 8 +
hadoop-ozone/ozonefs-hadoop2/pom.xml | 72 +-
.../hadoop/fs/ozone/Hadoop27RpcTransport.java | 6 +-
hadoop-ozone/ozonefs-hadoop3-client/pom.xml | 147 -
hadoop-ozone/ozonefs-hadoop3/pom.xml | 16 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 38 +-
hadoop-ozone/ozonefs/pom.xml | 4 +-
hadoop-ozone/pom.xml | 5 +-
hadoop-ozone/recon-codegen/pom.xml | 2 +-
hadoop-ozone/recon/pom.xml | 2 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 8 +
.../org/apache/hadoop/ozone/recon/ReconServer.java | 26 +-
.../hadoop/ozone/recon/ReconSqlDbConfig.java | 24 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 41 +-
.../ozone/recon/api/ClusterStateEndpoint.java | 99 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 2 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 18 +-
.../ozone/recon/api/OMDBInsightEndpoint.java | 249 +-
.../ozone/recon/api/ReconGlobalMetricsService.java | 248 ++
.../recon/api/StorageDistributionEndpoint.java | 255 ++
.../ozone/recon/api/UtilizationEndpoint.java | 91 +-
.../ozone/recon/api/filters/ReconAuthFilter.java | 5 +-
.../recon/api/handlers/BucketEntityHandler.java | 24 +-
.../recon/api/handlers/DirectoryEntityHandler.java | 24 +-
.../ozone/recon/api/handlers/FSOBucketHandler.java | 44 +-
.../recon/api/handlers/LegacyBucketHandler.java | 60 +-
.../ozone/recon/api/handlers/OBSBucketHandler.java | 36 +-
.../hadoop/ozone/recon/api/types/AclMetadata.java | 8 +-
.../ozone/recon/api/types/BucketObjectDBInfo.java | 8 +
.../recon/api/types/ClusterStateResponse.java | 4 +-
.../ozone/recon/api/types/DatanodeMetadata.java | 6 +-
.../recon/api/types/DatanodeStorageReport.java | 122 +-
...t.java => DeletionPendingBytesByComponent.java} | 44 +-
...orageReport.java => GlobalNamespaceReport.java} | 47 +-
.../ozone/recon/api/types/GlobalStorageReport.java | 61 +
.../hadoop/ozone/recon/api/types/NSSummary.java | 14 +-
.../recon/api/types/NamespaceSummaryResponse.java | 8 +-
.../ozone/recon/api/types/PipelineMetadata.java | 10 +-
.../ozone/recon/api/types/ReconBasicOmKeyInfo.java | 32 +
.../types/StorageCapacityDistributionResponse.java | 133 +
.../ozone/recon/api/types/UsedSpaceBreakDown.java | 76 +
.../hadoop/ozone/recon/codec/NSSummaryCodec.java | 29 +-
.../recon/metrics/OzoneManagerSyncMetrics.java | 12 -
.../ozone/recon/metrics/ReconSyncMetrics.java | 182 +
.../recon/metrics/ReconTaskControllerMetrics.java | 161 +
.../ozone/recon/metrics/ReconTaskMetrics.java | 182 +
.../recon/scm/ReconPipelineReportHandler.java | 2 +-
.../scm/ReconStorageContainerManagerFacade.java | 16 +
.../recon/spi/ReconContainerMetadataManager.java | 20 -
.../ozone/recon/spi/ReconFileMetadataManager.java | 97 +
.../ozone/recon/spi/ReconGlobalStatsManager.java | 83 +
.../recon/spi/impl/ContainerKeyPrefixCodec.java | 4 +-
.../recon/spi/impl/KeyPrefixContainerCodec.java | 110 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 127 +-
.../impl/ReconContainerMetadataManagerImpl.java | 31 -
.../ozone/recon/spi/impl/ReconDBDefinition.java | 22 +-
.../spi/impl/ReconFileMetadataManagerImpl.java | 114 +
.../spi/impl/ReconGlobalStatsManagerImpl.java | 101 +
.../hadoop/ozone/recon/tasks/FileSizeCountKey.java | 113 +
.../ozone/recon/tasks/FileSizeCountTaskFSO.java | 33 +-
.../ozone/recon/tasks/FileSizeCountTaskHelper.java | 309 +-
.../ozone/recon/tasks/FileSizeCountTaskOBS.java | 33 +-
.../hadoop/ozone/recon/tasks/GlobalStatsValue.java | 70 +
.../recon/tasks/MultipartInfoInsightHandler.java | 173 +
.../recon/tasks/NSSummaryTaskDbEventHandler.java | 117 +-
.../ozone/recon/tasks/OMDBUpdatesHandler.java | 28 +
.../ozone/recon/tasks/OMUpdateEventBuffer.java | 35 +-
.../ozone/recon/tasks/OmTableInsightTask.java | 74 +-
.../hadoop/ozone/recon/tasks/ReconTaskConfig.java | 8 +-
.../ozone/recon/tasks/ReconTaskControllerImpl.java | 202 +-
.../updater/ReconTaskStatusUpdaterManager.java | 98 +-
.../ozone/recon/upgrade/ReconLayoutFeature.java | 3 +-
.../ReplicatedSizeOfFilesUpgradeAction.java | 66 +
.../webapps/recon/ozone-recon-web/package.json | 2 +-
.../webapps/recon/ozone-recon-web/pnpm-lock.yaml | 82 +-
.../src/components/navBar/navBar.tsx | 6 +-
.../webapps/recon/ozone-recon-web/src/routes.tsx | 6 +-
.../recon/ozone-recon-web/src/utils/common.tsx | 48 +-
.../decommissioningSummary.tsx | 78 +-
.../components/duBreadcrumbNav/duBreadcrumbNav.tsx | 6 +-
.../src/v2/components/navBar/navBar.tsx | 38 +-
.../src/v2/components/nuMetadata/nuMetadata.tsx | 194 +-
.../overviewCard/overviewStorageCard.tsx | 204 +-
.../v2/components/plots/insightsContainerPlot.tsx | 2 +-
.../src/v2/components/plots/nuPieChart.tsx | 8 +-
.../src/v2/components/tables/containersTable.tsx | 30 +-
.../tables/insights/containerMismatchTable.tsx | 77 +-
.../tables/insights/deletePendingDirsTable.tsx | 67 +-
.../tables/insights/deletePendingKeysTable.tsx | 109 +-
.../tables/insights/deletedContainerKeysTable.tsx | 70 +-
.../components/tables/insights/openKeysTable.tsx | 113 +-
.../src/v2/constants/description.constants.tsx | 64 +
.../src/v2/hooks/useAPIData.hook.ts | 204 +-
.../src/v2/pages/buckets/buckets.tsx | 159 +-
.../src/v2/pages/containers/containers.tsx | 132 +-
.../src/v2/pages/datanodes/datanodes.tsx | 143 +-
.../src/v2/pages/heatmap/heatmap.tsx | 175 +-
.../src/v2/pages/insights/insights.tsx | 146 +-
.../src/v2/pages/insights/omInsights.tsx | 3 +-
.../v2/pages/namespaceUsage/namespaceUsage.less | 6 +
.../src/v2/pages/namespaceUsage/namespaceUsage.tsx | 82 +-
.../src/v2/pages/overview/overview.tsx | 21 +-
.../src/v2/pages/pipelines/pipelines.tsx | 104 +-
.../src/v2/pages/volumes/volumes.tsx | 113 +-
.../ozone-recon-web/src/v2/types/insights.types.ts | 28 +-
.../ozone-recon-web/src/views/buckets/buckets.tsx | 2 +-
.../src/views/datanodes/datanodes.tsx | 6 +-
.../src/views/datanodes/decommissionSummary.tsx | 2 +-
.../src/views/diskUsage/diskUsage.tsx | 8 +-
.../src/views/insights/insights.tsx | 6 +-
.../ozone-recon-web/src/views/insights/om/om.tsx | 44 +-
.../views/missingContainers/missingContainers.tsx | 4 +-
.../src/views/overview/overview.tsx | 4 +-
.../src/views/pipelines/pipelines.tsx | 2 +-
.../ozone-recon-web/src/views/volumes/volumes.tsx | 2 +-
.../ozone/recon/OMMetadataManagerTestUtils.java | 2 +-
.../hadoop/ozone/recon/ReconTestInjector.java | 10 +
.../CommonUtils.java => api/NSSummaryTests.java} | 68 +-
.../ozone/recon/api/TestClusterStateEndpoint.java | 7 +-
.../recon/api/TestDeletedKeysSearchEndpoint.java | 2 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 68 +-
.../recon/api/TestNSSummaryEndpointWithFSO.java | 287 +-
.../recon/api/TestNSSummaryEndpointWithLegacy.java | 217 +-
.../api/TestNSSummaryEndpointWithOBSAndLegacy.java | 17 +-
.../ozone/recon/api/TestOmDBInsightEndPoint.java | 184 +-
.../ozone/recon/api/TestOpenContainerCount.java | 6 +-
.../ozone/recon/api/TestTriggerDBSyncEndpoint.java | 5 +-
.../hadoop/ozone/recon/common/ReconTestUtils.java | 28 +-
.../scm/AbstractReconContainerManagerTest.java | 2 +-
.../spi/impl/TestKeyPrefixContainerCodec.java | 92 +
.../impl/TestOzoneManagerServiceProviderImpl.java | 14 +-
.../impl/TestReconNamespaceSummaryManagerImpl.java | 6 +-
.../recon/tasks/AbstractNSSummaryTaskTest.java | 3 -
.../ozone/recon/tasks/TestEventBufferOverflow.java | 26 +-
.../ozone/recon/tasks/TestFileSizeCountTask.java | 333 +-
.../TestNSSummaryTaskControllerIntegration.java | 7 +-
.../recon/tasks/TestNSSummaryTaskWithFSO.java | 15 +-
.../recon/tasks/TestNSSummaryTaskWithLegacy.java | 3 +-
.../recon/tasks/TestNSSummaryUnifiedControl.java | 2 +
.../ozone/recon/tasks/TestOMDBUpdatesHandler.java | 2 +-
.../ozone/recon/tasks/TestOMUpdateEventBuffer.java | 2 +-
.../ozone/recon/tasks/TestOmTableInsightTask.java | 194 +-
.../recon/tasks/TestReconTaskControllerImpl.java | 14 +-
.../TestReplicatedSizeOfFilesUpgradeAction.java | 96 +
hadoop-ozone/s3-secret-store/pom.xml | 4 +-
hadoop-ozone/s3gateway/pom.xml | 4 +-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 6 +
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 9 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 45 +-
.../ozone/s3/endpoint/ObjectEndpointStreaming.java | 5 +-
.../apache/hadoop/ozone/s3/util/ContinueToken.java | 3 +-
.../org/apache/hadoop/ozone/s3/util/S3Utils.java | 20 +
.../ozone/protocolPB/TestGrpcOmTransport.java | 4 +-
.../hadoop/ozone/s3/TestS3GatewayAuditLog.java | 8 +-
.../src/test/resources/auditlog.properties | 25 +-
hadoop-ozone/tools/pom.xml | 10 +-
.../org/apache/hadoop/fs/ozone/OzoneFsDelete.java | 0
.../org/apache/hadoop/fs/ozone/OzoneFsShell.java | 0
.../org/apache/hadoop/fs/ozone}/package-info.java | 6 +-
.../datanode/container/ContainerCommands.java | 4 +-
.../apache/hadoop/ozone/debug/om/PrefixParser.java | 19 +-
.../debug/replicas/ContainerStateVerifier.java | 105 +-
.../apache/hadoop/ozone/repair/OzoneRepair.java | 4 +-
.../org/apache/hadoop/ozone/repair/RepairTool.java | 36 +-
.../hadoop/ozone/repair/TransactionInfoRepair.java | 3 +-
.../schemaupgrade/UpgradeContainerSchema.java | 9 +-
.../datanode/schemaupgrade/UpgradeUtils.java | 3 +-
.../ozone/repair/ldb/RocksDBManualCompaction.java | 26 +-
.../apache/hadoop/ozone/repair/om/CompactOMDB.java | 3 +-
.../hadoop/ozone/repair/om/FSORepairTool.java | 18 +-
.../hadoop/ozone/repair/om/OMRatisLogRepair.java | 10 +-
.../ozone/repair/om/SnapshotChainRepair.java | 1 +
.../hadoop/ozone/repair/om/quota/QuotaRepair.java | 6 +-
.../repair/scm/cert/RecoverSCMCertificate.java | 3 +-
.../apache/hadoop/fs/ozone/TestOzoneFsShell.java | 0
.../hadoop/ozone/repair/ldb/TestLdbRepair.java | 5 +-
pom.xml | 191 +-
1313 files changed, 59186 insertions(+), 14824 deletions(-)
delete mode 100755 .github/close-pending.sh
delete mode 100644 .github/closing-message.txt
delete mode 100755 .github/comment-commands/close.sh
delete mode 100755 .github/comment-commands/pending.sh
delete mode 100755 .github/comment-commands/ready.sh
rename .github/workflows/{close-pending.yaml => close-stale-prs.yaml} (50%)
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamBlockInputStream.java
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/BoundedElasticByteBufferPool.java
create mode 100644
hadoop-hdds/client/src/test/java/org/apache/hadoop/ozone/client/io/TestBoundedElasticByteBufferPool.java
rename hadoop-hdds/common/src/main/java/com/google/protobuf/{Proto2Utils.java
=> ProtoUtils.java} (85%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/StreamingReadResponse.java
copy
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/{cli/GenericParentCommand.java
=> scm/StreamingReaderSpi.java} (65%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/GlobPattern.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/io/RandomAccessFileChannel.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/AlignmentContext.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/AsyncCallLimitExceededException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/CallQueueManager.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/CallerContext.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/Client.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ClientCache.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ClientId.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/CostProvider.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/DecayRpcScheduler.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/DecayRpcSchedulerMXBean.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/DefaultCostProvider.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/DefaultRpcScheduler.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ExternalCall.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/FairCallQueue.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/FairCallQueueMXBean.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/GenericRefreshProtocol.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/IdentityProvider.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/IpcException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ObserverRetryOnActiveException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProcessingDetails.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtoUtil.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtobufHelper.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtobufRpcEngine.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtobufRpcEngineCallback.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolInfo.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolMetaInfoPB.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolMetaInfoServerSideTranslatorPB.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolMetaInterface.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolProxy.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolSignature.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProtocolTranslator.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ProxyCombiner.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RPC.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RefreshCallQueueProtocol.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RefreshHandler.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RefreshRegistry.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RefreshResponse.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RemoteException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/ResponseBuffer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RetriableException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RetryCache.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcClientException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcClientUtil.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcConstants.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcEngine.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcInvocationHandler.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcMultiplexer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcNoSuchMethodException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcNoSuchProtocolException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcScheduler.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcServerException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/RpcWritable.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/Schedulable.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/Server.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/StandbyException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/UnexpectedServerException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/UserIdentityProvider.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/VersionedProtocol.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/WeightedRoundRobinMultiplexer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/WeightedTimeCostProvider.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/WritableRpcEngine.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/metrics/RetryCacheMetrics.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/metrics/RpcDetailedMetrics.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/metrics/RpcMetrics.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/metrics/package-info.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ipc_/package-info.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java
copy
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/package-info.java
=> common/src/main/java/org/apache/hadoop/ozone/util/WithChecksum.java} (72%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/security_/SaslRpcClient.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/security_/SaslRpcServer.java
delete mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/OpenContainerBlockMap.java
rename
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/{StorageVolumeScannerMetrics.java
=> BackgroundVolumeScannerMetrics.java} (87%)
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/BigIntegerCodec.java
=>
container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/PendingDelete.java}
(54%)
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerDiff.java
create mode 100644
hadoop-hdds/docs/content/concept/HDDS-13515-Design-Document.md
create mode 100644 hadoop-hdds/docs/content/concept/RocksDB.md
create mode 100644 hadoop-hdds/docs/content/concept/flowchart.png
create mode 100644
hadoop-hdds/docs/content/design/distributed-tracing-OpenTelemetry.md
create mode 100644 hadoop-hdds/docs/content/design/distributed-tracing-flow.png
create mode 100644
hadoop-hdds/docs/content/design/distributed-tracing-sample.png
create mode 100644
hadoop-hdds/docs/content/feature/Snapshot-Configuration-Properties.md
rename hadoop-hdds/{common =>
framework}/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java (98%)
rename hadoop-hdds/{common =>
framework}/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
(98%)
rename hadoop-hdds/{common =>
framework}/src/main/java/org/apache/hadoop/hdds/security/token/ContainerTokenIdentifier.java
(100%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HttpServletUtils.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/CodecBufferCodec.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/InodeMetadataRocksDBCheckpoint.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/ozone/common/DeletedBlock.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestHttpServletUtils.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeStorageReport.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/StringInMemoryTestTable.java
(54%)
create mode 100644
hadoop-hdds/interface-client/src/main/proto/IpcConnectionContext.proto
create mode 100644
hadoop-hdds/interface-client/src/main/proto/ProtobufRpcEngine.proto
create mode 100644
hadoop-hdds/interface-client/src/main/proto/ProtocolInfo.proto
create mode 100644 hadoop-hdds/interface-client/src/main/proto/RpcHeader.proto
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/LongCodec.java
=>
managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedDirectSlice.java}
(51%)
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeStorageReport.java
=>
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/hadoop/hdds/utils/db/TablePrefixInfo.java
(52%)
copy hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/protocol =>
rocksdb-checkpoint-differ/src/main/java/org/apache/hadoop/hdds/utils/db}/package-info.java
(90%)
rename
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/{compaction/log
=> rocksdb/util}/SstFileInfo.java (86%)
create mode 100644
hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/compaction/log/TestSstFileInfo.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/AbstractContainerSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/StateMachineReadyRule.java
copy
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/{TestECContainerSafeModeRule.java
=> AbstractContainerSafeModeRuleTest.java} (62%)
create mode 100644
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/om/OmAddressOptions.java
create mode 100644
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/om/snapshot/DefragSubCommand.java
copy
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/IncrementalReportSender.java
=>
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/om/snapshot/SnapshotSubCommand.java
(71%)
copy {hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol
=>
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/om/snapshot}/package-info.java
(87%)
copy
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotObjectStore.java
=>
cli-admin/src/main/java/org/apache/hadoop/ozone/admin/scm/DeletedBlocksTxnCommands.java}
(61%)
create mode 100644
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/scm/GetDeletedBlockSummarySubcommand.java
create mode 100644
hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/om/snapshot/TestDefragSubCommand.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/io/TestOzoneOutputStream.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/AclListBuilder.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/DAGLeveledResource.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/DAGResourceLockTracker.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/HierarchicalResourceLockManager.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/LeveledResourceLockTracker.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/PoolBasedHierarchicalResourceLockManager.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/ReadOnlyHierarchicalResourceLockManager.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/ResourceLockTracker.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/AssumeRoleRequest.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestAclListBuilder.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOMNodeDetails.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/lock/TestDAGLeveledResource.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/lock/TestPoolBasedHierarchicalResourceLockManager.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/acl/TestAssumeRoleRequest.java
copy hadoop-ozone/dev-support/checks/{pmd.sh => javadoc.sh} (73%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/cli/ozone-insight.robot
create mode 100644
hadoop-ozone/freon/src/main/java/org/apache/hadoop/ozone/freon/FollowerReader.java
delete mode 100644 hadoop-ozone/httpfsgateway/src/site/configuration.xsl
delete mode 100644 hadoop-ozone/httpfsgateway/src/site/resources/css/site.css
delete mode 100644 hadoop-ozone/httpfsgateway/src/site/site.xml
create mode 100644
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/InsightHttpUtils.java
create mode 100644
hadoop-ozone/insight/src/test/java/org/apache/hadoop/ozone/insight/TestBaseInsightSubCommand.java
create mode 100644
hadoop-ozone/integration-test-recon/src/test/java/org/apache/hadoop/ozone/recon/TestStorageDistributionEndpoint.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestDNDataDistributionFinalization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestScmDataDistributionFinalization.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestStreamBlockInputStream.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestStreamRead.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMHALeaderSpecificACLEnforcement.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/service/TestBlockDeletionService.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDefragAdmin.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestDeletedBlocksTxnShell.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHAWithFollowerRead.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/defrag/SnapshotDefragService.java
copy {hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol
=>
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/defrag}/package-info.java
(85%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/CompositeDeltaDiffComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/DeltaFileComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/RDBDifferComputer.java
copy {hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol
=>
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta}/package-info.java
(84%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/util/TableMergeIterator.java
copy {hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol
=>
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/util}/package-info.java
(86%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManagerMetrics.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/TestOMResponse.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotLocalDataManager.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/defrag/TestInodeMetadataRocksDBCheckpoint.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/defrag/TestSnapshotDefragService.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestCompositeDeltaDiffComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestFileLinkDeltaFileComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestFullDiffComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestRDBDifferComputer.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/util/TestTableMergeIterator.java
delete mode 100644 hadoop-ozone/ozonefs-hadoop3-client/pom.xml
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ReconGlobalMetricsService.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/StorageDistributionEndpoint.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/{DatanodeStorageReport.java
=> DeletionPendingBytesByComponent.java} (50%)
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/{DatanodeStorageReport.java
=> GlobalNamespaceReport.java} (51%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/GlobalStorageReport.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/StorageCapacityDistributionResponse.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/UsedSpaceBreakDown.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconSyncMetrics.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconTaskControllerMetrics.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconTaskMetrics.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/ReconFileMetadataManager.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/ReconGlobalStatsManager.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconFileMetadataManagerImpl.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconGlobalStatsManagerImpl.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountKey.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/GlobalStatsValue.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/MultipartInfoInsightHandler.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReplicatedSizeOfFilesUpgradeAction.java
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/v2/constants/description.constants.tsx
rename
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/{common/CommonUtils.java
=> api/NSSummaryTests.java} (81%)
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/StringCodec.java
=>
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/common/ReconTestUtils.java
(58%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/spi/impl/TestKeyPrefixContainerCodec.java
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/upgrade/TestReplicatedSizeOfFilesUpgradeAction.java
rename hadoop-ozone/{ozonefs-common =>
tools}/src/main/java/org/apache/hadoop/fs/ozone/OzoneFsDelete.java (100%)
rename hadoop-ozone/{ozonefs-common =>
tools}/src/main/java/org/apache/hadoop/fs/ozone/OzoneFsShell.java (100%)
rename {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol
=>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/fs/ozone}/package-info.java
(90%)
rename hadoop-ozone/{ozonefs-common/src/test =>
tools/src/test/java}/java/org/apache/hadoop/fs/ozone/TestOzoneFsShell.java
(100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
