This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git.
omit 3cb1725 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
omit 5d00641 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
omit 16089ba HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
omit 8a090f0 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
omit 3e5fdd4 HDDS-5747. Reset default values in
ContainerBalancerConfiguration (#2657)
omit d3599ac HDDS-5757. balancer should stop when the cluster can not be
balanced any more (#2676)
omit 29c711d HDDS-5523. Fix multipart upload failure in s3 compatibility
tests (#2509)
omit 28cd9b0 HDDS-5800. Bucket usedByte was incorrect when deleting
duplicate keys.
omit 27acd21 HDDS-5789. Enabling container tokens on a upgraded cluster
fails SCM to start up. (#2691)
omit 10521ad HDDS-5782. DatanodeRatisGrpcConfig use wrong type for config
keys. (#2678)
omit 448c633 HDDS-5792. Speed up TestNSSummaryAdmin by having zero
Datanodes (#2688)
omit 9541464 HDDS-5777. Provide an option to dump table scan data to file.
(#2675)
omit 12cb74b HDDS-5370. [FSO] Handle OMClientRequest based on the bucket
layout. (#2533)
omit f1e1e98 HDDS-5697. SCMContainerPlacementRackAware should both
affinity nodes before fallback (#2597)
omit d741015 HDDS-5770. Silent failures of k3s install are difficult to
debug (#2667)
omit 1f38332 HDDS-5759. Bump aspectj version (#2658)
omit b70315e HDDS-5773. Avoid code duplication for mini cluster without
datanodes (#2669)
omit dff93db HDDS-5691. Restrict Recon NSSummaryEndpoint and
ContainerEndpoint to admins. (#2638)
omit f0e05c2 HDDS-5771. Speed up TestDatanodeHddsVolumeFailureToleration
by reducing dead interval (#2668)
omit 4e5836b HDDS-5767. Unit check may timeout (#2664)
omit 9cb7614 HDDS-5765. Test cluster provider possibly returns null (#2663)
omit c54b91f HDDS-5768. Skip safemode check in
TestOzoneManagerRocksDBLogging (#2665)
omit ea63d0c HDDS-5766. Speed up some OM tests by skipping SCM safemode
check (#2662)
omit 3e640e8 HDDS-5761. should not shutdown om when setting a bigger
bucket quota than volume quota (#2659)
omit 467661f HDDS-5758. Speed up TestKeyInputStream and
TestChunkInputStream by combining some tests (#2656)
omit b9908b0 HDDS-5607. remove container manager v1 code (#2525)
omit 95e5ecc HDDS-5753. Split parts of misc suite (#2654)
omit 10a78f0 HDDS-5751. Use Mini Cluster Provider to speed up
TestHDDSUpgrade (#2650)
omit c1ec7ad HDDS-5728. ContainerBalancer should use remaining space to
calculate utilization. (#2625)
omit c7172f5 HDDS-5402 Support list node based on NodeOperationalState and
NodeState options in printTopology CLI (#2645)
omit 6d2c343 HDDS-5749. Reuse mini-clusters in TestOzoneFsHAURLs (#2647)
omit babf11d HDDS-5717. Refactor TestOzoneManagerListVolumes to reuse
mini-ozone clusters (#2615)
omit b398761 HDDS-5748. Reuse mini-clusters in
TestOzoneFSWithObjectStoreCreate (#2646)
omit 2ed458f HDDS-5744. Remove unnecessary duplicate semicolon in
PipelineStateMap (#2641)
omit 8d5d3a8 HDDS-5335. Method not found: allocateBlock - when tracing is
enabled (#2330)
omit d104ce3 HDDS-5729. support setting maxSizeEnteringTarget and
maxSizeLeavingSource in command line (#2626)
omit 412f907 HDDS-5696. Close grpcReplicationClient on exception. (#2595)
omit daac760 HDDS-5708. Skip sending container close command to unhealthy
replica. (#2610)
omit e89542d HDDS-5679. Use more defensive sizeRequired for replication
manager for container replication. (#2585)
omit 302a291 HDDS-5720. Reuse mini-clusters in TestOzoneFileInterfaces
(#2619)
omit 06c4bd2 HDDS-5496. Missing directory name in `start_k8s_env` log
(#2545)
omit 2edee75 HDDS-5735. Prometheus HTTP API Reference cannot be displayed
normally (#2630)
omit 620de30 HDDS-5502. [OFS] URI parser throws URISyntaxException when
path contains space (#2500)
omit 83b1673 HDDS-5715. Make XceiverServerRatis#raftGids a thread-safe
set. (#2613)
omit 25cf48b HDDS-5699. Added Log to show why a container was marked
UNHEALTHY. (#2627)
omit 5fde27b HDDS-5723. Increase time limit of Ozone acceptance tests.
(#2620)
omit 63bc2ba HDDS-5718. Refactor TestXceiverClientManager to reuse
mini-clusters (#2616)
omit acf6f71 HDDS-5724. Add RaftpeerId when getting scm roles (#2622)
omit 11b186e HDDS-5711. support -1 for running balancer infinitely (#2621)
omit da3c933 HDDS-5670. ContainerBalancer should get OzoneConfiguration
from ContainerBalancerConfiguration. (#2577)
omit ad1ed89 HDDS-5638. Fix docker-compose to make Recon come up. (#2563)
omit 41d9b07 HDDS-5726. Skip remove for already removed pipeline. (#2624)
omit 6783f69 HDDS-5719. Reduce number of mini-clusters needed for
decommission tests (#2617)
omit 70e1511 HDDS-5716. Fix create key failure error log print (#2614)
omit 93491d2 HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
(#2596)
omit 200308e HDDS-5432. Enable downgrade testing after 1.1.0 release.
(#2484)
omit dc89c89 HDDS-5709. do not call removeTransactionsFromDB if nothing to
remove (#2608)
omit 2e7eb29 HDDS-5700. Improve LOG message of decommission progress.
(#2598)
omit 6cac1caf HDDS-5690. Speed up TestContainerReplication by removing
testSkipDemmissionAndMaintenanceNode (#2591)
omit 9fc27ff HDDS-5706. Fix ReplicationManager zero metrics for inflight
actions. (#2605)
omit ddfa0f8 HDDS-5667. documentation page layout (#2604)
omit 6fd7a38 HDDS-5644. Speed up decommission tests using a background
Mini Cluster provider (#2554)
omit 9fa3e8e HDDS-5680. Fix bug for incomplete SCM ratis log (#2586)
omit f2e1207 HDDS-5689. chooseRandom in NetworkTopology should exclude the
affinity node too (#2590)
omit 2fc6d34 HDDS-5692. Combine some tests in TestOzoneManagerHAWithData
to reuse mini-Clusters (#2593)
omit 95dc1f8 HDDS-5368. Add CLI command: ozone admin namespace summary
(#2565)
omit bc2a74c HDDS-5669. Fix NSSummaryEndpoint's NPE bug during OM
initialization (#2592)
omit 476b537 HDDS-5535. SCM-HA documents configuration error (#2492)
omit b0ab8ec HDDS-5662. Fix blank traceId for FsShell command (#2568)
omit 7750b69 HDDS-5688. Rpc should not retry if exception is
ContainerNotFoundException. (#2589)
omit ad25987 HDDS-5681. Speed up TestOzoneManagerHAWithACL (#2587)
omit 446aaec HDDS-5687. support Optional<T> as parameters of commandLine
(#2588)
omit 6e60a16 HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
omit 3c719a0 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
omit bbfc4c0 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
omit 2972386 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
omit 76990b4 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
omit 06e193b HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add 30859fe HDDS-5687. support Optional<T> as parameters of commandLine
(#2588)
add 7dcaad1 HDDS-5681. Speed up TestOzoneManagerHAWithACL (#2587)
add b593fa8 HDDS-5688. Rpc should not retry if exception is
ContainerNotFoundException. (#2589)
add db75046 HDDS-5662. Fix blank traceId for FsShell command (#2568)
add 2958ab3 HDDS-5535. SCM-HA documents configuration error (#2492)
add 33b3f9e HDDS-5669. Fix NSSummaryEndpoint's NPE bug during OM
initialization (#2592)
add aab19a6 HDDS-5368. Add CLI command: ozone admin namespace summary
(#2565)
add 9baca7c HDDS-5692. Combine some tests in TestOzoneManagerHAWithData
to reuse mini-Clusters (#2593)
add 758b8ca HDDS-5689. chooseRandom in NetworkTopology should exclude the
affinity node too (#2590)
add 194de5b HDDS-5680. Fix bug for incomplete SCM ratis log (#2586)
add cdac8bc HDDS-5644. Speed up decommission tests using a background
Mini Cluster provider (#2554)
add fe42640 HDDS-5667. documentation page layout (#2604)
add 730cb66 HDDS-5706. Fix ReplicationManager zero metrics for inflight
actions. (#2605)
add ea1ff1e HDDS-5690. Speed up TestContainerReplication by removing
testSkipDemmissionAndMaintenanceNode (#2591)
add f71474a HDDS-5700. Improve LOG message of decommission progress.
(#2598)
add aa8cfcd HDDS-5709. do not call removeTransactionsFromDB if nothing to
remove (#2608)
add 7e2cdf0 HDDS-5432. Enable downgrade testing after 1.1.0 release.
(#2484)
add bbd0907 HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
(#2596)
add 0decca8 HDDS-5716. Fix create key failure error log print (#2614)
add 737297e HDDS-5719. Reduce number of mini-clusters needed for
decommission tests (#2617)
add 2d188cc HDDS-5726. Skip remove for already removed pipeline. (#2624)
add b64080b HDDS-5638. Fix docker-compose to make Recon come up. (#2563)
add e9cce6e HDDS-5670. ContainerBalancer should get OzoneConfiguration
from ContainerBalancerConfiguration. (#2577)
add ebbcf0a HDDS-5711. support -1 for running balancer infinitely (#2621)
add 6239270 HDDS-5724. Add RaftpeerId when getting scm roles (#2622)
add 2b6e1ac HDDS-5718. Refactor TestXceiverClientManager to reuse
mini-clusters (#2616)
add ee993d0 HDDS-5723. Increase time limit of Ozone acceptance tests.
(#2620)
add e392c1a HDDS-5699. Added Log to show why a container was marked
UNHEALTHY. (#2627)
add 753a5b7 HDDS-5715. Make XceiverServerRatis#raftGids a thread-safe
set. (#2613)
add 6569040 HDDS-5502. [OFS] URI parser throws URISyntaxException when
path contains space (#2500)
add bb37a55 HDDS-5735. Prometheus HTTP API Reference cannot be displayed
normally (#2630)
add 67be640 HDDS-5496. Missing directory name in `start_k8s_env` log
(#2545)
add 9aebf6e HDDS-5720. Reuse mini-clusters in TestOzoneFileInterfaces
(#2619)
add e97ea36 HDDS-5679. Use more defensive sizeRequired for replication
manager for container replication. (#2585)
add 2d57e9f HDDS-5708. Skip sending container close command to unhealthy
replica. (#2610)
add 873c0d2 HDDS-5696. Close grpcReplicationClient on exception. (#2595)
add 6543c67 HDDS-5729. support setting maxSizeEnteringTarget and
maxSizeLeavingSource in command line (#2626)
add ec5c51f HDDS-5335. Method not found: allocateBlock - when tracing is
enabled (#2330)
add c5a012b HDDS-5744. Remove unnecessary duplicate semicolon in
PipelineStateMap (#2641)
add ee19b95 HDDS-5748. Reuse mini-clusters in
TestOzoneFSWithObjectStoreCreate (#2646)
add da18617 HDDS-5717. Refactor TestOzoneManagerListVolumes to reuse
mini-ozone clusters (#2615)
add ed30752 HDDS-5749. Reuse mini-clusters in TestOzoneFsHAURLs (#2647)
add d716bf4 HDDS-5402 Support list node based on NodeOperationalState and
NodeState options in printTopology CLI (#2645)
add fab8a12 HDDS-5728. ContainerBalancer should use remaining space to
calculate utilization. (#2625)
add f2c20ce HDDS-5751. Use Mini Cluster Provider to speed up
TestHDDSUpgrade (#2650)
add 5c972c4 HDDS-5753. Split parts of misc suite (#2654)
add 0b9e23e HDDS-5607. remove container manager v1 code (#2525)
add 2d58933 HDDS-5758. Speed up TestKeyInputStream and
TestChunkInputStream by combining some tests (#2656)
add c68468c HDDS-5761. should not shutdown om when setting a bigger
bucket quota than volume quota (#2659)
add 79da6b7 HDDS-5766. Speed up some OM tests by skipping SCM safemode
check (#2662)
add b8afbbc HDDS-5768. Skip safemode check in
TestOzoneManagerRocksDBLogging (#2665)
add 79716f0 HDDS-5765. Test cluster provider possibly returns null (#2663)
add 2c2d2f3 HDDS-5767. Unit check may timeout (#2664)
add b508b25 HDDS-5771. Speed up TestDatanodeHddsVolumeFailureToleration
by reducing dead interval (#2668)
add fc61be5 HDDS-5691. Restrict Recon NSSummaryEndpoint and
ContainerEndpoint to admins. (#2638)
add 3116fa2 HDDS-5773. Avoid code duplication for mini cluster without
datanodes (#2669)
add c25b6d7 HDDS-5759. Bump aspectj version (#2658)
add 87b4971 HDDS-5770. Silent failures of k3s install are difficult to
debug (#2667)
add 605a734 HDDS-5697. SCMContainerPlacementRackAware should both
affinity nodes before fallback (#2597)
add e5c647e HDDS-5370. [FSO] Handle OMClientRequest based on the bucket
layout. (#2533)
add 181266b HDDS-5777. Provide an option to dump table scan data to file.
(#2675)
add 4b3502f HDDS-5792. Speed up TestNSSummaryAdmin by having zero
Datanodes (#2688)
add 2e3c46c HDDS-5782. DatanodeRatisGrpcConfig use wrong type for config
keys. (#2678)
add 041684f HDDS-5789. Enabling container tokens on a upgraded cluster
fails SCM to start up. (#2691)
add af7dc7e HDDS-5800. Bucket usedByte was incorrect when deleting
duplicate keys.
add 8cc6719 HDDS-5523. Fix multipart upload failure in s3 compatibility
tests (#2509)
add aa9eade HDDS-5757. balancer should stop when the cluster can not be
balanced any more (#2676)
add e2c3ac3 HDDS-5747. Reset default values in
ContainerBalancerConfiguration (#2657)
add 5c2ac71 HDDS-5788. Reduce run time for TestOzoneManagerHA tests
(#2689)
add af5b48e HDDS-5636. Use bucket layout specific DB table in
OmMetadataManager:getOpenKeyTable(). (#2561)
add 093f972 HDDS-5787. Provide capability in Ozone to enable ratis
pre-vote (#2680)
add 05eb172 HDDS-5762. remove piplelineStateManager V1 code (#2661)
add c4cbcad HDDS-5710. initialize sequenceIdToLastIdMap when
SequenceIdGenerator#StateManager reinitializes (#2611)
add 53a0009 HDDS-5793. Speed up TestBlockOutputStreamWithFailures by
combining test cases (#2690)
add 1da0bd8 HDDS-5609. Document CI workflow (#2607)
add c886a2f HDDS-5797. Support setting Datanode Reserved Space in
MiniOzoneCluster. (#2695)
add 7bdc1bb HDDS-5799. Speed up TestOzoneFileSystem by using shorter
trash interval (#2699)
add 1ea1d6c HDDS-5808. Update commons-io to 2.11.0 (#2705)
add 186cd6e HDDS-5817. Extract properties for some dependency versions
(#2710)
add 20e8429 HDDS-5815 FileEncryptionInfo is not necessary to be
referenced inside KeyOutputStream. (#2708)
add a59c501 HDDS-5830. Increase idea.max.intellisense.filesize (#2715)
add 8d885b4 HDDS-5557. Cleanup
OzoneBlockTokenSecretManager#validateToken. (#2711)
add ea81ef8 HDDS-5816 Rearrange code and refactor some logic into new
methods in prep for EC addition. (#2709)
add 8eaa9c5 HDDS-4440. Diagrams for old and new approach are mixed up
(#2555)
add c5a92a2 HDDS-5824. `ozone sh volume/bucket/key list` should print
valid JSON array (#2713)
add db97a32 HDDS-5733. Incorrect calculation of iteration related metrics
in ContainerBalancer (#2631)
add efb5263 HDDS-5806. remove BackgroundPipelineCreator v1 code (#2700)
add 57729b4 HDDS-5536. Add metrics for ReplicationSupervisor (#2493)
add 1ffcfe9 HDDS-5349. Avoid usage of locks in listStatus. (#2369)
add 3e05e71 HDDS-5373. [FSO] Define default bucket layout. (#2707)
add f902b13 HDDS-5785. containerCreationLock is a severe bottleneck when
there are many containers to create. (#2679)
add a62c61f HDDS-5823. Change the key permission to 600. (#2722)
add 2f49993 HDDS-5845. Add metrics for stateMachineDataCache cache hit
and miss. (#2726)
add 730a2e8 HDDS-5842. Parse and dump SCM ratis segment file to printable
text (#2731)
add e19fa38 HDDS-5814. Handle InterruptedException in Ozone (#2706)
add ae812c06 HDDS-5831. Remove empty TaskQueue in ContainerStateMachine.
(#2716)
add ed069bd HDDS-5854. Bump json-smart to 2.3.1 (#2736)
add 7376cfe HDDS-5855. Bump commons-compress to 1.21 (#2737)
add 0e5ad31 HDDS-5856. Bump Spring to 5.2.17 (#2738)
add d341c05 HDDS-5857. Bump aws-java-sdk to latest 1.11 (#2740)
add 24ddf0f HDDS-5659. Use bucket layout specific DB table in
OmMetadataManager.getKeyTable() - part1 (#2724)
add bf6da11 HDDS-5858. Fix resource leak due to Files.list (#2741)
add f2f6aa5 HDDS-5671. ContainerBalancer#stop should prevent the current
balancing thread from interrupting itself. (#2578)
new 8ea027b HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new 11860d3 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
new b98d551 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new 501dc12 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
new a710076 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new d9cbd7d HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
new 14ba242 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new d22551d HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new c9ca832 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new edbcf28 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3cb1725)
\
N -- N -- N refs/heads/HDDS-4454 (edbcf28)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/ci.md | 102 +++
CONTRIBUTING.md | 4 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 4 +
.../common/src/main/resources/ozone-default.xml | 33 +-
.../common/statemachine/DatanodeStateMachine.java | 7 +
.../common/transport/server/ratis/CSMMetrics.java | 10 +
.../server/ratis/ContainerStateMachine.java | 9 +-
.../transport/server/ratis/XceiverServerRatis.java | 12 +-
.../ozone/container/common/volume/VolumeInfo.java | 5 +
.../ozone/container/keyvalue/KeyValueHandler.java | 29 +-
.../replication/DownloadAndImportReplicator.java | 5 +-
.../replication/ReplicationSupervisor.java | 8 +
.../replication/ReplicationSupervisorMetrics.java | 72 +++
.../replication/SimpleContainerDownloader.java | 7 +-
.../replication/TestReplicationSupervisor.java | 59 +-
hadoop-hdds/docs/content/feature/PrefixFSO.md | 16 +
hadoop-hdds/docs/content/interface/CSI.md | 4 +-
hadoop-hdds/docs/content/interface/CSI.zh.md | 4 +-
hadoop-hdds/docs/content/interface/Cli.md | 7 +-
.../docs/static/design/s3-performance-new.png | Bin 33688 -> 34927 bytes
.../docs/static/design/s3-performance-old.png | Bin 34927 -> 33688 bytes
.../hdds/conf/DatanodeRatisServerConfig.java | 16 +
.../hadoop/hdds/security/x509/keys/KeyCodec.java | 31 +-
.../org/apache/hadoop/hdds/server/JsonUtils.java | 16 +-
.../hdds/security/x509/keys/TestKeyCodec.java | 5 +-
.../hdds/scm/container/ReplicationManager.java | 3 +
.../scm/container/balancer/ContainerBalancer.java | 219 ++++---
.../balancer/ContainerBalancerMetrics.java | 115 ++--
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 11 +-
.../apache/hadoop/hdds/scm/ha/SCMRatisRequest.java | 24 +
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 26 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 264 ++++++--
.../scm/pipeline/BackgroundPipelineCreatorV2.java | 332 ----------
.../hadoop/hdds/scm/pipeline/PipelineFactory.java | 2 +-
.../hdds/scm/pipeline/PipelineManagerImpl.java | 22 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 6 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 6 +-
.../hdds/scm/pipeline/PipelineStateManager.java | 266 +++-----
...erV2Impl.java => PipelineStateManagerImpl.java} | 58 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 2 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 4 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 2 +-
.../hadoop/hdds/scm/pipeline/StateManager.java | 127 ----
.../algorithms/DefaultLeaderChoosePolicy.java | 4 +-
.../choose/algorithms/LeaderChoosePolicy.java | 8 +-
.../algorithms/LeaderChoosePolicyFactory.java | 8 +-
.../algorithms/MinLeaderCountChoosePolicy.java | 6 +-
.../scm/container/TestContainerManagerImpl.java | 6 +-
.../container/balancer/TestContainerBalancer.java | 27 +
.../hdds/scm/container/balancer}/package-info.java | 7 +-
.../hdds/scm/node/TestContainerPlacement.java | 29 +-
.../hdds/scm/pipeline/MockPipelineManager.java | 27 +-
.../scm/pipeline/MockRatisPipelineProvider.java | 6 +-
.../TestPipelineDatanodesIntersection.java | 52 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 73 ++-
...ager.java => TestPipelineStateManagerImpl.java} | 237 ++++---
.../scm/pipeline/TestRatisPipelineProvider.java | 66 +-
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 209 -------
.../scm/pipeline/TestSimplePipelineProvider.java | 48 +-
.../choose/algorithms/TestLeaderChoosePolicy.java | 6 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 268 +++++---
.../client/io/BlockOutputStreamEntryPool.java | 109 +++-
.../hadoop/ozone/client/io/KeyOutputStream.java | 16 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 5 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 16 +
.../hadoop/ozone/om/helpers/BucketLayout.java | 8 +
.../org/apache/hadoop/ozone/csi/NodeService.java | 10 +-
.../dist/src/main/smoketest/basic/links.robot | 8 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 10 +-
.../dist/src/main/smoketest/omha/testOMHA.robot | 2 +-
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 18 +-
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 14 +-
.../hadoop/fs/ozone/TestOzoneFileInterfaces.java | 37 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 15 +-
.../fs/ozone/TestOzoneFileSystemPrefixParser.java | 9 +-
.../fs/ozone/TestOzoneFileSystemWithFSO.java | 9 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 9 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 19 +
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 7 +
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 16 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 20 +-
...estBlockOutputStreamWithFailuresFlushDelay.java | 20 +-
.../rpc/TestOzoneClientMultipartUploadWithFSO.java | 30 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 44 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 9 +-
.../apache/hadoop/ozone/om/TestObjectStore.java | 103 +++
.../hadoop/ozone/om/TestObjectStoreWithFSO.java | 19 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 113 +++-
.../ozone/om/TestOzoneManagerHAMetadataOnly.java | 22 +-
.../ozone/om/TestOzoneManagerHAWithData.java | 98 +--
.../ozone/om/TestOzoneManagerHAWithFailover.java | 65 ++
.../hadoop/ozone/om/TestOzoneManagerPrepare.java | 28 +-
.../TestOzoneHARatisLogParser.java} | 52 +-
.../ozone/recon/TestReconWithOzoneManagerFSO.java | 6 +-
.../hadoop/ozone/shell/TestNSSummaryAdmin.java | 6 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 128 +++-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 6 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 691 +++++++++++++--------
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 38 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 26 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 1 +
.../ozone/om/ratis/OzoneManagerRatisServer.java | 12 +-
.../om/request/bucket/OMBucketCreateRequest.java | 11 +-
.../om/request/file/OMDirectoryCreateRequest.java | 9 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 4 +-
.../request/file/OMFileCreateRequestWithFSO.java | 6 +
.../ozone/om/request/file/OMFileRequest.java | 85 ++-
.../om/request/key/OMAllocateBlockRequest.java | 9 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 6 +
.../ozone/om/request/key/OMKeyCommitRequest.java | 7 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 6 +
.../ozone/om/request/key/OMKeyCreateRequest.java | 6 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 6 +
.../ozone/om/request/key/OMKeyDeleteRequest.java | 6 +-
.../om/request/key/OMKeyDeleteRequestWithFSO.java | 8 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 9 +-
.../om/request/key/OMKeyRenameRequestWithFSO.java | 9 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 27 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 5 +-
.../om/request/key/OMPathsPurgeRequestWithFSO.java | 6 +
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 9 +-
.../om/request/key/acl/OMKeyAclRequestWithFSO.java | 5 +-
.../S3InitiateMultipartUploadRequest.java | 2 +-
.../S3InitiateMultipartUploadRequestWithFSO.java | 6 +
.../multipart/S3MultipartUploadAbortRequest.java | 16 +-
.../S3MultipartUploadAbortRequestWithFSO.java | 6 +
.../S3MultipartUploadCommitPartRequest.java | 4 +-
.../S3MultipartUploadCommitPartRequestWithFSO.java | 5 +
.../S3MultipartUploadCompleteRequest.java | 12 +-
.../S3MultipartUploadCompleteRequestWithFSO.java | 6 +
.../ozone/om/request/upgrade/OMPrepareRequest.java | 9 +-
.../hadoop/ozone/om/response/OMClientResponse.java | 26 +
.../response/file/OMDirectoryCreateResponse.java | 6 +-
.../response/file/OMFileCreateResponseWithFSO.java | 5 +
.../om/response/key/OMAllocateBlockResponse.java | 4 +-
.../key/OMAllocateBlockResponseWithFSO.java | 6 +
.../ozone/om/response/key/OMKeyCommitResponse.java | 8 +-
.../response/key/OMKeyCommitResponseWithFSO.java | 10 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 6 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 3 +-
.../response/key/OMKeyDeleteResponseWithFSO.java | 9 +-
.../ozone/om/response/key/OMKeyRenameResponse.java | 12 +-
.../response/key/OMKeyRenameResponseWithFSO.java | 14 +-
.../om/response/key/OMKeysDeleteResponse.java | 3 +-
.../om/response/key/OMOpenKeysDeleteRequest.java | 5 +-
.../om/response/key/OMOpenKeysDeleteResponse.java | 3 +-
.../response/key/OMPathsPurgeResponseWithFSO.java | 5 +
.../om/response/key/acl/OMKeyAclResponse.java | 4 +-
.../response/key/acl/OMKeyAclResponseWithFSO.java | 4 +-
.../S3InitiateMultipartUploadResponse.java | 4 +-
.../multipart/S3MultipartUploadAbortResponse.java | 4 +-
.../S3MultipartUploadAbortResponseWithFSO.java | 6 +
.../S3MultipartUploadCommitPartResponse.java | 4 +-
...S3MultipartUploadCommitPartResponseWithFSO.java | 6 +
.../S3MultipartUploadCompleteResponse.java | 34 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 1 +
.../security/OzoneBlockTokenSecretManager.java | 5 -
.../hadoop/ozone/om/TestOmMetadataManager.java | 7 +-
.../ozone/om/request/TestOMRequestUtils.java | 64 +-
.../bucket/TestOMBucketCreateRequestWithFSO.java | 5 +
.../request/file/TestOMDirectoryCreateRequest.java | 53 +-
.../file/TestOMDirectoryCreateRequestWithFSO.java | 28 +-
.../om/request/file/TestOMFileCreateRequest.java | 8 +-
.../file/TestOMFileCreateRequestWithFSO.java | 13 +-
.../key/TestOMAllocateBlockRequestWithFSO.java | 18 +-
.../ozone/om/request/key/TestOMKeyAclRequest.java | 30 +-
.../om/request/key/TestOMKeyAclRequestWithFSO.java | 6 +
.../om/request/key/TestOMKeyCommitRequest.java | 31 +-
.../om/request/key/TestOMKeyCreateRequest.java | 50 +-
.../request/key/TestOMKeyCreateRequestWithFSO.java | 9 +-
.../om/request/key/TestOMKeyDeleteRequest.java | 20 +-
.../request/key/TestOMKeyDeleteRequestWithFSO.java | 17 +-
.../om/request/key/TestOMKeyRenameRequest.java | 8 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 8 +-
.../om/request/key/TestOMKeysDeleteRequest.java | 4 +-
.../om/request/key/TestOMKeysRenameRequest.java | 20 +-
.../request/key/TestOMOpenKeysDeleteRequest.java | 10 +-
.../TestS3InitiateMultipartUploadRequest.java | 25 +-
...estS3InitiateMultipartUploadRequestWithFSO.java | 10 +-
.../s3/multipart/TestS3MultipartRequest.java | 22 +
.../TestS3MultipartUploadAbortRequest.java | 4 +-
.../TestS3MultipartUploadCommitPartRequest.java | 5 +-
...tS3MultipartUploadCommitPartRequestWithFSO.java | 6 +
.../TestS3MultipartUploadCompleteRequest.java | 17 +-
.../file/TestOMDirectoryCreateResponse.java | 7 +-
.../file/TestOMFileCreateResponseWithFSO.java | 6 +
.../response/key/TestOMAllocateBlockResponse.java | 12 +-
.../key/TestOMAllocateBlockResponseWithFSO.java | 5 +
.../om/response/key/TestOMKeyCommitResponse.java | 18 +-
.../key/TestOMKeyCommitResponseWithFSO.java | 6 +
.../om/response/key/TestOMKeyCreateResponse.java | 12 +-
.../key/TestOMKeyCreateResponseWithFSO.java | 6 +
.../om/response/key/TestOMKeyDeleteResponse.java | 18 +-
.../key/TestOMKeyDeleteResponseWithFSO.java | 6 +
.../om/response/key/TestOMKeyRenameResponse.java | 30 +-
.../ozone/om/response/key/TestOMKeyResponse.java | 5 +
.../om/response/key/TestOMKeysDeleteResponse.java | 9 +-
.../om/response/key/TestOMKeysRenameResponse.java | 16 +-
.../response/key/TestOMOpenKeysDeleteResponse.java | 18 +-
.../TestS3InitiateMultipartUploadResponse.java | 3 +-
...stS3InitiateMultipartUploadResponseWithFSO.java | 9 +-
.../s3/multipart/TestS3MultipartResponse.java | 15 +-
.../TestS3MultipartUploadAbortResponse.java | 11 +-
...S3MultipartUploadCommitPartResponseWithFSO.java | 15 +-
...stS3MultipartUploadCompleteResponseWithFSO.java | 27 +-
hadoop-ozone/pom.xml | 2 +-
.../ozone/recon/fsck/ContainerHealthTask.java | 3 +
.../hadoop/ozone/recon/scm/PipelineSyncTask.java | 3 +
.../ozone/recon/scm/ReconPipelineManager.java | 10 +-
.../ozone/recon/OMMetadataManagerTestUtils.java | 25 +-
.../ozone/admin/om/FinalizeUpgradeSubCommand.java | 5 +-
.../admin/scm/FinalizeScmUpgradeSubcommand.java | 5 +-
.../apache/hadoop/ozone/debug/PrefixParser.java | 6 +-
.../ozone/debug/container/ContainerCommands.java | 16 +-
.../ozone/freon/ClosedContainerReplicator.java | 10 +-
.../containergenerator/GeneratorDatanode.java | 23 +-
.../hadoop/ozone/segmentparser/RatisLogParser.java | 4 +-
...MRatisLogParser.java => SCMRatisLogParser.java} | 15 +-
.../org/apache/hadoop/ozone/shell/Handler.java | 17 +
.../ozone/shell/bucket/CreateBucketHandler.java | 6 +-
.../ozone/shell/bucket/ListBucketHandler.java | 10 +-
.../hadoop/ozone/shell/keys/ListKeyHandler.java | 8 +-
.../ozone/shell/volume/ListVolumeHandler.java | 9 +-
pom.xml | 55 +-
224 files changed, 3946 insertions(+), 2455 deletions(-)
create mode 100644 .github/ci.md
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisorMetrics.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreatorV2.java
rename
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/{PipelineStateManagerV2Impl.java
=> PipelineStateManagerImpl.java} (88%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/StateManager.java
copy
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate
=>
server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer}/package-info.java
(84%)
rename
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/{TestPipelineStateManager.java
=> TestPipelineStateManagerImpl.java} (64%)
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestObjectStore.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAWithFailover.java
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/{om/parser/TestOMRatisLogParser.java
=> parser/TestOzoneHARatisLogParser.java} (68%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/segmentparser/{OMRatisLogParser.java
=> SCMRatisLogParser.java} (75%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
