This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git.
discard c062de2 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
omit 6a2233d HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
omit e04c27d Revert "HDDS-5742. [Ozone-Streaming] Avoid unnecessary
Bytebuffer conversions (#2640)" (#2643)
omit fe15324 HDDS-5742. [Ozone-Streaming] Avoid unnecessary Bytebuffer
conversions (#2640)
omit 4156bf7 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
omit 9cf9c4d HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
omit 04b4c17 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
omit 2ef30bb HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
omit 1924810 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
omit ecff9ab HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
omit f89e1f5 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add bbd0907 HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
(#2596)
add 0decca8 HDDS-5716. Fix create key failure error log print (#2614)
add 737297e HDDS-5719. Reduce number of mini-clusters needed for
decommission tests (#2617)
add 2d188cc HDDS-5726. Skip remove for already removed pipeline. (#2624)
add b64080b HDDS-5638. Fix docker-compose to make Recon come up. (#2563)
add e9cce6e HDDS-5670. ContainerBalancer should get OzoneConfiguration
from ContainerBalancerConfiguration. (#2577)
add ebbcf0a HDDS-5711. support -1 for running balancer infinitely (#2621)
add 6239270 HDDS-5724. Add RaftpeerId when getting scm roles (#2622)
add 2b6e1ac HDDS-5718. Refactor TestXceiverClientManager to reuse
mini-clusters (#2616)
add ee993d0 HDDS-5723. Increase time limit of Ozone acceptance tests.
(#2620)
add e392c1a HDDS-5699. Added Log to show why a container was marked
UNHEALTHY. (#2627)
add 753a5b7 HDDS-5715. Make XceiverServerRatis#raftGids a thread-safe
set. (#2613)
add 6569040 HDDS-5502. [OFS] URI parser throws URISyntaxException when
path contains space (#2500)
add bb37a55 HDDS-5735. Prometheus HTTP API Reference cannot be displayed
normally (#2630)
add 67be640 HDDS-5496. Missing directory name in `start_k8s_env` log
(#2545)
add 9aebf6e HDDS-5720. Reuse mini-clusters in TestOzoneFileInterfaces
(#2619)
add e97ea36 HDDS-5679. Use more defensive sizeRequired for replication
manager for container replication. (#2585)
add 2d57e9f HDDS-5708. Skip sending container close command to unhealthy
replica. (#2610)
add 873c0d2 HDDS-5696. Close grpcReplicationClient on exception. (#2595)
add 6543c67 HDDS-5729. support setting maxSizeEnteringTarget and
maxSizeLeavingSource in command line (#2626)
add ec5c51f HDDS-5335. Method not found: allocateBlock - when tracing is
enabled (#2330)
add c5a012b HDDS-5744. Remove unnecessary duplicate semicolon in
PipelineStateMap (#2641)
add ee19b95 HDDS-5748. Reuse mini-clusters in
TestOzoneFSWithObjectStoreCreate (#2646)
add da18617 HDDS-5717. Refactor TestOzoneManagerListVolumes to reuse
mini-ozone clusters (#2615)
add ed30752 HDDS-5749. Reuse mini-clusters in TestOzoneFsHAURLs (#2647)
add d716bf4 HDDS-5402 Support list node based on NodeOperationalState and
NodeState options in printTopology CLI (#2645)
add fab8a12 HDDS-5728. ContainerBalancer should use remaining space to
calculate utilization. (#2625)
add f2c20ce HDDS-5751. Use Mini Cluster Provider to speed up
TestHDDSUpgrade (#2650)
add 5c972c4 HDDS-5753. Split parts of misc suite (#2654)
add 0b9e23e HDDS-5607. remove container manager v1 code (#2525)
add 2d58933 HDDS-5758. Speed up TestKeyInputStream and
TestChunkInputStream by combining some tests (#2656)
add c68468c HDDS-5761. should not shutdown om when setting a bigger
bucket quota than volume quota (#2659)
add 79da6b7 HDDS-5766. Speed up some OM tests by skipping SCM safemode
check (#2662)
add b8afbbc HDDS-5768. Skip safemode check in
TestOzoneManagerRocksDBLogging (#2665)
add 79716f0 HDDS-5765. Test cluster provider possibly returns null (#2663)
add 2c2d2f3 HDDS-5767. Unit check may timeout (#2664)
add b508b25 HDDS-5771. Speed up TestDatanodeHddsVolumeFailureToleration
by reducing dead interval (#2668)
add fc61be5 HDDS-5691. Restrict Recon NSSummaryEndpoint and
ContainerEndpoint to admins. (#2638)
add 3116fa2 HDDS-5773. Avoid code duplication for mini cluster without
datanodes (#2669)
add c25b6d7 HDDS-5759. Bump aspectj version (#2658)
add 87b4971 HDDS-5770. Silent failures of k3s install are difficult to
debug (#2667)
add 605a734 HDDS-5697. SCMContainerPlacementRackAware should both
affinity nodes before fallback (#2597)
add e5c647e HDDS-5370. [FSO] Handle OMClientRequest based on the bucket
layout. (#2533)
add 181266b HDDS-5777. Provide an option to dump table scan data to file.
(#2675)
add 4b3502f HDDS-5792. Speed up TestNSSummaryAdmin by having zero
Datanodes (#2688)
add 2e3c46c HDDS-5782. DatanodeRatisGrpcConfig use wrong type for config
keys. (#2678)
add 041684f HDDS-5789. Enabling container tokens on a upgraded cluster
fails SCM to start up. (#2691)
add af7dc7e HDDS-5800. Bucket usedByte was incorrect when deleting
duplicate keys.
add 8cc6719 HDDS-5523. Fix multipart upload failure in s3 compatibility
tests (#2509)
add aa9eade HDDS-5757. balancer should stop when the cluster can not be
balanced any more (#2676)
add e2c3ac3 HDDS-5747. Reset default values in
ContainerBalancerConfiguration (#2657)
new 225a802 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new fac149d HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
new 2cd909d HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new 8df196a HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
new b5bd217 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new 69464ce HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
new e064c4b HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new e29e77f HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new 420e472 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c062de2)
\
N -- N -- N refs/heads/HDDS-4454 (420e472)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 6 +-
.../apache/hadoop/hdds/recon/ReconConfigKeys.java | 10 +
.../apache/hadoop/hdds/scm/client/ScmClient.java | 4 +-
.../hadoop/hdds/scm/net/NetworkTopology.java | 3 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 5 +-
.../protocol/StorageContainerLocationProtocol.java | 4 +-
.../apache/hadoop/hdds/tracing/TraceAllMethod.java | 23 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 6 +-
.../common/src/main/resources/ozone-default.xml | 11 +
.../hadoop/hdds/tracing/TestTraceAllMethod.java | 68 +++
.../hadoop/hdds/tracing/TestTracingUtil.java} | 33 +-
.../ozone/container/common/impl/ContainerSet.java | 3 +
.../container/common/impl/HddsDispatcher.java | 1 +
.../CloseContainerCommandHandler.java | 2 +
.../ClosePipelineCommandHandler.java | 13 +-
.../transport/server/ratis/XceiverServerRatis.java | 3 +-
.../container/keyvalue/KeyValueContainerCheck.java | 2 +
.../replication/SimpleContainerDownloader.java | 20 +-
hadoop-hdds/docs/content/interface/ReconApi.md | 23 +-
hadoop-hdds/docs/content/interface/ReconApi.zh.md | 2 +-
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 4 +-
...inerLocationProtocolClientSideTranslatorPB.java | 19 +-
.../certificate/client/DNCertificateClient.java | 5 -
.../client/DefaultCertificateClient.java | 2 +-
.../certificate/client/OMCertificateClient.java | 5 -
.../certificate/client/SCMCertificateClient.java | 10 +-
.../src/main/proto/ScmAdminProtocol.proto | 2 +
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 6 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 12 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 9 +-
.../container/AbstractContainerReportHandler.java | 6 +-
.../scm/container/CloseContainerEventHandler.java | 4 +-
.../hdds/scm/container/ContainerManager.java | 144 ++---
.../hdds/scm/container/ContainerManagerImpl.java | 6 +-
.../hdds/scm/container/ContainerManagerV2.java | 190 ------
.../hdds/scm/container/ContainerReportHandler.java | 6 +-
.../IncrementalContainerReportHandler.java | 2 +-
.../hdds/scm/container/ReplicationManager.java | 31 +-
.../hdds/scm/container/SCMContainerManager.java | 672 ---------------------
.../scm/container/balancer/ContainerBalancer.java | 86 ++-
.../balancer/ContainerBalancerConfiguration.java | 56 +-
.../ContainerBalancerSelectionCriteria.java | 14 +-
.../scm/container/balancer/FindTargetGreedy.java | 6 +-
.../algorithms/SCMContainerPlacementRackAware.java | 72 ++-
.../container/placement/metrics/SCMNodeStat.java | 23 -
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 3 +-
.../hadoop/hdds/scm/node/DatanodeUsageInfo.java | 65 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 6 +-
.../hdds/scm/node/NodeDecommissionManager.java | 8 +-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 8 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 60 +-
.../hadoop/hdds/scm/pipeline/PipelineStateMap.java | 2 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 18 -
.../pipeline/WritableRatisContainerProvider.java | 6 +-
...inerLocationProtocolServerSideTranslatorPB.java | 15 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 10 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 10 +-
.../scm/server/OzoneStorageContainerManager.java | 4 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 23 +-
.../hadoop/hdds/scm/server/SCMConfigurator.java | 8 +-
.../hdds/scm/server/StorageContainerManager.java | 73 ++-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 12 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 6 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 6 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 16 +-
.../hdds/scm/container/SimpleMockNodeManager.java | 5 +
.../container/TestCloseContainerEventHandler.java | 2 +-
.../scm/container/TestContainerManagerImpl.java | 2 +-
.../scm/container/TestContainerReportHandler.java | 4 +-
.../TestIncrementalContainerReportHandler.java | 4 +-
.../hdds/scm/container/TestReplicationManager.java | 33 +-
.../scm/container/TestSCMContainerManager.java | 324 ----------
.../scm/container/TestUnknownContainerReport.java | 4 +-
.../container/balancer/TestContainerBalancer.java | 64 +-
.../TestSCMContainerPlacementRackAware.java | 15 +
.../hdds/scm/node/TestContainerPlacement.java | 95 +--
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 8 +-
.../testutils/ReplicationNodeManagerMock.java | 11 +
.../scm/cli/ContainerBalancerStartSubcommand.java | 13 +-
.../hdds/scm/cli/ContainerOperationClient.java | 7 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 46 +-
.../datanode/TestContainerBalancerSubCommand.java | 6 +-
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 45 +-
hadoop-ozone/dev-support/checks/_lib.sh | 8 +-
.../dist/src/main/compose/ozone-ha/test.sh | 2 +
.../src/main/compose/ozone-mr/hadoop27/test.sh | 2 +
.../src/main/compose/ozone-mr/hadoop31/test.sh | 2 +
.../src/main/compose/ozone-mr/hadoop32/test.sh | 2 +
.../src/main/compose/ozone-mr/hadoop33/test.sh | 2 +
.../dist/src/main/compose/ozone-mr/test.sh | 3 +
.../compose/ozonesecure-ha/docker-compose.yaml | 2 +
.../dist/src/main/compose/ozonesecure-ha/test.sh | 2 +
.../dist/src/main/compose/ozonesecure-mr/test.sh | 2 +
.../src/main/compose/ozonesecure/docker-config | 1 +
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh | 2 +-
.../dist/src/main/smoketest/recon/recon-api.robot | 53 +-
.../src/main/smoketest/s3/MultipartUpload.robot | 23 +-
.../dist/src/main/smoketest/topology/cli.robot | 6 +
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 30 +-
.../hadoop/fs/ozone/TestOzoneFileInterfaces.java | 69 ++-
.../fs/ozone/TestOzoneFileInterfacesWithFSO.java | 8 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 35 +-
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 4 +-
.../apache/hadoop/hdds/scm/TestSCMSnapshot.java | 4 +-
.../TestContainerStateManagerIntegration.java | 140 ++---
.../metrics/TestSCMContainerManagerMetrics.java | 4 +-
.../hdds/scm/pipeline/TestNode2PipelineMap.java | 4 +-
.../hdds/scm/pipeline/TestPipelineClose.java | 4 +-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 6 +-
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 41 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 12 +
.../hadoop/ozone/MiniOzoneClusterProvider.java | 3 +
.../ozone/TestContainerBalancerOperations.java | 8 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 26 +
.../client/rpc/read/TestChunkInputStream.java | 13 +-
.../ozone/client/rpc/read/TestInputStreamBase.java | 44 --
.../ozone/client/rpc/read/TestKeyInputStream.java | 58 +-
.../apache/hadoop/ozone/container/TestHelper.java | 4 +-
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 4 +-
.../TestDatanodeHddsVolumeFailureToleration.java | 40 +-
.../org/apache/hadoop/ozone/om/TestOmLDBCli.java | 64 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 28 +-
.../ozone/om/TestOzoneManagerConfiguration.java | 2 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 29 +-
.../ozone/om/TestOzoneManagerHAMetadataOnly.java | 17 +
.../ozone/om/TestOzoneManagerListVolumes.java | 117 ++--
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 2 +-
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 7 +-
.../om/ratis/TestOzoneManagerRatisRequest.java | 71 +++
.../hadoop/ozone/recon/TestReconAsPassiveScm.java | 10 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 4 +-
.../org/apache/hadoop/ozone/scm/TestSCMMXBean.java | 4 +-
.../hadoop/ozone/scm/TestXceiverClientManager.java | 18 +-
.../scm/node/TestDecommissionAndMaintenance.java | 94 +--
.../hadoop/ozone/shell/TestNSSummaryAdmin.java | 3 +-
hadoop-ozone/ozone-manager/pom.xml | 10 +-
.../hadoop/ozone/om/DirectoryDeletingService.java | 21 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 47 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 4 +
.../org/apache/hadoop/ozone/om/OzoneManager.java | 26 +-
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 2 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 130 +++-
.../request/bucket/OMBucketSetPropertyRequest.java | 9 +-
.../request/file/OMFileCreateRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 4 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 58 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 15 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 38 ++
.../om/request/volume/OMVolumeSetQuotaRequest.java | 5 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 8 +-
.../protocolPB/OzoneManagerRequestHandler.java | 12 +-
.../ozone/om/request/TestOMRequestUtils.java | 22 +-
.../bucket/TestOMBucketSetPropertyRequest.java | 30 +-
.../om/request/key/TestOMKeyAclRequestWithFSO.java | 2 +-
.../request/key/TestOMKeyCreateRequestWithFSO.java | 2 -
.../om/request/key/TestOMKeyDeleteRequest.java | 13 +-
.../request/key/TestOMKeyDeleteRequestWithFSO.java | 10 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 7 +-
.../volume/TestOMVolumeSetQuotaRequest.java | 28 +-
.../org/apache/hadoop/fs/ozone/TestOFSPath.java | 26 +
.../apache/hadoop/ozone/recon/ReconHttpServer.java | 1 +
.../hadoop/ozone/recon/ReconRestServletModule.java | 144 +++--
.../org/apache/hadoop/ozone/recon/ReconServer.java | 26 +-
.../apache/hadoop/ozone/recon/api/AdminOnly.java | 23 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 1 +
.../hadoop/ozone/recon/api/NSSummaryEndpoint.java | 5 +-
.../ozone/recon/api/filters/ReconAdminFilter.java | 113 ++++
.../ozone/recon/api/filters/ReconAuthFilter.java | 117 ++++
.../recon/api/{ => filters}/package-info.java | 5 +-
.../ozone/recon/fsck/ContainerHealthTask.java | 6 +-
.../recon/scm/ReconContainerReportHandler.java | 4 +-
.../ozone/recon/scm/ReconDeadNodeHandler.java | 4 +-
.../ReconIncrementalContainerReportHandler.java | 4 +-
.../scm/ReconStorageContainerManagerFacade.java | 4 +-
.../ozone/recon/api/TestNSSummaryEndpoint.java | 12 +-
.../ozone/recon/api/filters/TestAdminFilter.java | 161 +++++
.../ozone/recon/fsck/TestContainerHealthTask.java | 6 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 39 +-
.../s3/endpoint/TestMultipartUploadWithCopy.java | 219 +++++--
.../admin/scm/GetScmRatisRolesSubcommand.java | 7 +-
.../org/apache/hadoop/ozone/debug/DBScanner.java | 74 ++-
pom.xml | 3 +
182 files changed, 2926 insertions(+), 2407 deletions(-)
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/tracing/TestTraceAllMethod.java
copy
hadoop-hdds/{framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLogAppender.java
=> common/src/test/java/org/apache/hadoop/hdds/tracing/TestTracingUtil.java}
(50%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerV2.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisRequest.java
copy
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/AuditAction.java
=>
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/AdminOnly.java
(61%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/filters/ReconAdminFilter.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/filters/ReconAuthFilter.java
copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{ =>
filters}/package-info.java (87%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/filters/TestAdminFilter.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
