This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-1564
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
discard 22570c4 HDDS-1569 Support creating multiple pipelines with same
datanode. Contributed by Li Cheng.
omit 764fac5 HDDS-2089: Add createPipeline CLI. (#1418)
omit baa5626 HDDS-1571. Create an interface for pipeline placement policy
to support network topologies. (#1395)
omit f236926 HDDS-1577. Add default pipeline placement policy
implementation. (#1366)
add f59d359 HDDS-2298. Fix maven warning about duplicated metrics-core jar
add 920dde9 HDDS-2298. Fix maven warning about duplicated metrics-core
jar.
add e227ba4 HDDS-2220. HddsVolume needs a toString method.
add 83b5a67 HDDS-2220. HddsVolume needs a toString method.
add b3322b5 HDDS-2204. Avoid buffer coping in checksum verification.
Contributed by Tsz Wo Nicholas Sze.
add 44a6a3f BlockManager should allocate a block in excluded pipelines if
none other left (#19)
add 8868d4e HDDS-2194. Replication of Container fails with "Only closed
containers could be exported" (#25)
add 546e36b HDDS-2305. Update Ozone to latest ratis
snapshot(0.5.0-3f446aa-SNAPSHOT). Contributed by Mukul Kumar Singh. (#26)
add 616d7a4 HDDS-2295. Display log of freon on the standard output
add b4a1afd HDDS 2181. Ozone Manager should send correct ACL type in ACL
requests to Authorizer Contributed by Vivek Ratnavel Subramanian.
add 59d66bc HDDS-1998.
TestSecureContainerServer#testClientServerRatisGrpc is failing
add f1835dd Revert "HDDS-1998.
TestSecureContainerServer#testClientServerRatisGrpc is failing"
add 5adf6a1 HDDS-1988. Fix listParts API
add fecdba7 HDDS-2267. Container metadata scanner interval mismatch
add 34df0bf HDDS-2312. Fix typo in ozone command
add 17081c2 Revert "HDDS 2181. Ozone Manager should send correct ACL type
in ACL requests to Authorizer"
add 96f1dcc HDDS-1737. Add Volume check in KeyManager and File Operations
add 01e24e8 HDDS-2316. Support to skip recon and/or ozonefs during the
build (#39)
add 0f5f7ba HDDS-2289. Put testing information and a problem description
to the g… (#5)
add 9f88f42 HDDS-2032. Ozone client should retry writes in case of any
ratis/stateMachine exceptions
add 61c161c Revert "HDDS-2032. Ozone client should retry writes in case
of any ratis/stateMachine exceptions"
add 85ba643 HDDS-2302. Manage common pom versions in one common place.
add 38ec79e HDDS-2271. Avoid buffer copying in KeyValueHandler.
add 47f0bf5 Merge pull request #44 from szetszwo/HDDS-2271
add 6c575f2 HDDS-2275. In BatchOperation.SingleOperation, do not clone
byte[].
add 61f4aa3 Merge pull request #45 from szetszwo/HDDS-2275
add b041050 HDDS-1985. Fix listVolumes API
add 37f48c4 HDDS-2221. Monitor datanodes in ozoneperf compose cluster
add bcd04c2 HDDS-2181. Ozone Manager should send correct ACL type in ACL
requests to Authorizer (#43)
add 2529cee HDDS-2254 : Fix flaky unit test
TestContainerStateMachine#testRatisSnapshotRetention.
add b961883 HDDS-2323. Mem allocation: Optimise AuditMessage::build().
Contributed by Siddharth Wagle.
add 25cfd0c HDDS-2318. Avoid proto::tostring in preconditions to save CPU
cycles. (#48)
add 7651aec HDDS-2286. Add a log info in ozone client and scm to print
the exclusion list during allocate block. Contributed by Siddharth Wagle. (#46)
add 18c4cab HDDS-2283. Container creation on datanodes take time because
of Rocksdb option creation. Contributed by Siddharth Wagle.(#41)
add bfaa640 HDDS-2281. ContainerStateMachine#handleWriteChunk should
ignore close container exception. Contributed by Shashikant Banerjee. (#54)
add 29cd37d HDDS-2280. HddsUtils#CheckForException should not return null
in case the ratis exception cause is not set. Contributed by Shashikant
Banerjee. (#57)
add aaa333c HDDS-2336. Fix
TestKeyValueContainer#testRocksDBCreateUsesCachedOptions. Contributed by
Attila Doroszlai. (#63)
add 65a1d00 HDDS-2337. Fix checkstyle errors. Contributed by Attila
Doroszlai. (#64)
add 40f57a5 HDDS-2335. Params not included in AuditMessage. Contributed
by Attila Doroszlai. (#62)
add 636df43 HDDS-2326. Http server of Freon is not started for new Freon
tests (#52)
add 445fe62 HDDS-2206. Separate handling for OMException and IOException
in the Ozone Manager. Contributed by Supratim Deka (#12)
add 02d2eb7 HDDS-2340. Updated ratis.version to get latest snapshot. (#67)
add e61e71d HDDS-2320. Negative value seen for OM NumKeys Metric in JMX.
(#68)
add d47fe4e HDDS-2310. Add support to add ozone ranger plugin to Ozone
Manager cl… (#49)
add c6c9794 HDDS-2333. Enable sync option for OM non-HA. (#61)
add 599b0d9 HDDS-2343. Add immutable entries in to the DoubleBuffer for
Bucket requests. (#69)
add eb1d77e HDDS-2330. Random key generator can get stuck (#53)
add d92223a HDDS-2346. Replication manager config has wrong description
(#73)
add fdd1b15 HDDS-2214. TestSCMContainerPlacementRackAware has an
intermittent failure. (#6)
add 4bfad90 HDDS-2131. Optimize replication type and creation time
calculation in S3 MPU list call. (#50)
add 1baa5a1 HDDS-2278. Run S3 test suite on OM HA cluste. (#27)
add 59d0786 HDDS-2334. Dummy chunk manager fails with length mismatch
error. Contributed by Attila Doroszlai. (#65)
add 9475d26 HDDS-2352. Client gets internal error instead of volume not
found in secure cluster (#76)
add 1c6b02a HDDS-2296. ozoneperf compose cluster shouln't start freon by
default (#15)
add e16275d HDDS-2297. Enable Opentracing for new Freon tests (#16)
add c0cbdd2 HDDS-2354. SCM log is full of AllocateBlock logs. (#77)
add 4d1e811 HDDS-1228. Chunk Scanner Checkpoints (#7)
add 28170e5 HDDS-2358. Change to replication factor THREE in acceptance
tests (#78)
add da01364 HDDS-2357. Add replication factor option to new Freon tests
(#79)
add faf10fe HDDS-2071. Support filters in ozone insight point (#4)
add 682599e HDDS-2360. Update Ratis snapshot to d6d58d0 (#83)
add d7e2fb1 HDDS-2344. Add immutable entries in to the DoubleBuffer for
Volume requests. (#71)
add e1641fb Revert "HDDS-2206. Separate handling for OMException and
IOException in the Ozone Manager. Contributed by Supratim Deka (#12)"
add d1f57f8 HDDS-2341. Validate tar entry path during extraction (#72)
add 82833a0 HDDS-2361. Ozone Manager init & start command prints out
unnecessary line in the beginning. (#91)
add 6a65580 HDDS-2365. Fix
TestRatisPipelineProvider#testCreatePipelinesDnExclude (#84)
add 8541c56 HDDS-2368. TestOzoneManagerDoubleBufferWithDummyResponse
failing intermittently (#93)
add 91e0c02 HDDS-2285. GetBlock and ReadChunk command from the client
should be sent to the same datanode to re-use the same connection.
add 758bd09 concurrent hash map for dn cache
add 41b51b9 Merge pull request #40 from hanishakoneru/HDDS-2285
add 9dd2f05 HDDS-2345. Add a UT for newly added clone() in OmBucketInfo
(#92)
add 19f0044 HDDS-2374. Make Ozone Readme.txt point to the Ozone websites
instead of Hadoop. (#96)
add 7b043e9 HDDS-2240. Ozone Admin OM command line tool. (#9)
add aea6c37 HDDS-2292. Create Ozone specific README.md to the new
hadoop-ozone re… (#55)
add e3e6dc5 HDDS-2322. DoubleBuffer flush termination and OM shutdown's
after that. Make entry returned from cache a new copy. (#89)
add 86925de HDDS-2373 Move isUseRatis getFactor and getType from
XCeiverClientManager (#95)
add 112d1fb Revert "HDDS-2292. Create Ozone specific README.md to the new
hadoop-ozone re… (#55)"
add 27b6042 HDDS-2379. OM terminates with RocksDB error while
continuously writing keys. (#100)
add d3021fb HDDS-2347 XCeiverClientGrpc's parallel use leads to NPE (#81)
add 9f75e5a HDDS-2378 - Change OZONE as string used in the code where
OzoneConsts.OZONE is suitable (#103)
add 69aa475 HDDS-2366. Remove ozone.enabled as a flag and config item.
(#90)
add e857643 HDDS-2292. Create Ozone specific README.md to the new
hadoop-ozone repository (#106)
add 11ff08f HDDS-2381. In ExcludeList, add if not exist only. (#104)
add e16c7ed HDDS-2348. remove log4j properties for org.apache.hadoop.ozone
add b6704fd HDDS-2208. Propagate System Exceptions from OM transaction
apply phase
add cbb8b66 HDDS-2219. Move all the ozone dist scripts/configs to one
location
add 5b082e7 HDDS-2293. Create a new CONTRIBUTION.md for the new
repository (#58)
add 629ab5d HDDS-2380. OMFileRequest should use the isExist API while
checking for pre-existing files in the directory path.
add 8527a9d HDDS-1847: Datanode Kerberos principal and keytab config key
looks inconsistent Contributed by christeoh.
add 5922420 HDDS-2363. Failed to create Ratis container. (#98)
add a04a5d8 HDDS-2311. Fix logic of RetryPolicy in
OzoneClientSideTranslatorPB. (#51)
add 6ba7115 HDDS-2397. Fix calling cleanup for few missing tables in OM.
(#112)
add feedc6a Revert "HDDS-1847: Datanode Kerberos principal and keytab
config key looks inconsistent"
add c47aaa1 HDDS-2398. Remove usage of LogUtils class from ratis-common.
(#113)
new ab1c6d6 HDDS-1577. Add default pipeline placement policy
implementation. (#1366)
new c82339c HDDS-1571. Create an interface for pipeline placement policy
to support network topologies. (#1395)
new 042062a HDDS-2089: Add createPipeline CLI. (#1418)
new 07e5348 HDDS-1569 Support creating multiple pipelines with same
datanode. Contributed by Li Cheng.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (22570c4)
\
N -- N -- N refs/heads/HDDS-1564 (07e5348)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/pull_request_template.md | 18 +-
BUILDING.txt | 511 ---------------------
CONTRIBUTION.md | 206 +++++++++
README.md | 98 ++++
README.txt | 7 -
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 79 +++-
.../hadoop/hdds/scm/XceiverClientManager.java | 38 +-
.../hdds/scm/client/ContainerOperationClient.java | 111 +++--
.../hadoop/hdds/scm/client/HddsClientUtils.java | 5 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 32 +-
.../scm/container/common/helpers/ExcludeList.java | 28 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 2 +-
.../apache/hadoop/hdds/utils/BatchOperation.java | 22 +-
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 59 ++-
.../org/apache/hadoop/hdds/utils/RocksDBStore.java | 19 +-
.../org/apache/hadoop/hdds/utils/db/Codec.java | 6 +
.../apache/hadoop/hdds/utils/db/CodecRegistry.java | 16 +
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 21 +-
.../apache/hadoop/hdds/utils/db/IntegerCodec.java | 5 +
.../org/apache/hadoop/hdds/utils/db/LongCodec.java | 5 +
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 17 +-
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 16 +
.../apache/hadoop/hdds/utils/db/StringCodec.java | 5 +
.../apache/hadoop/hdds/utils/db/TypedTable.java | 3 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 7 -
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 1 +
.../apache/hadoop/ozone/audit/AuditMessage.java | 7 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 218 +++++----
.../apache/hadoop/ozone/common/ChecksumData.java | 4 +-
.../ozone/container/common/helpers/BlockData.java | 5 +
.../common/src/main/resources/ozone-default.xml | 27 +-
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 21 +
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 29 +-
.../apache/hadoop/ozone/common/TestChecksum.java | 6 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 106 +++--
.../container/common/helpers/ContainerUtils.java | 2 +-
.../ozone/container/common/impl/ContainerData.java | 41 +-
.../common/impl/ContainerDataScanOrder.java | 57 +++
.../container/common/impl/ContainerDataYaml.java | 15 +-
.../ozone/container/common/impl/ContainerSet.java | 2 +
.../container/common/interfaces/Container.java | 4 +
.../server/ratis/ContainerStateMachine.java | 12 +-
.../ozone/container/common/volume/HddsVolume.java | 8 +
.../container/keyvalue/KeyValueContainer.java | 29 +-
.../container/keyvalue/KeyValueContainerCheck.java | 3 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 9 +-
.../container/keyvalue/TarContainerPacker.java | 31 +-
.../container/keyvalue/helpers/ChunkUtils.java | 23 +-
.../container/keyvalue/helpers/SmallFileUtils.java | 4 +-
.../keyvalue/impl/ChunkManagerDummyImpl.java | 17 +-
.../container/ozoneimpl/ContainerController.java | 7 +
.../container/ozoneimpl/ContainerDataScanner.java | 31 +-
.../ozoneimpl/ContainerMetadataScanner.java | 7 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 2 -
.../hadoop/ozone/TestHddsSecureDatanodeInit.java | 1 -
.../ozone/container/common/SCMTestUtils.java | 22 +
.../common/impl/TestContainerDataYaml.java | 33 +-
.../container/common/impl/TestContainerSet.java | 59 ++-
.../container/common/impl/TestHddsDispatcher.java | 2 +-
.../container/keyvalue/TestBlockManagerImpl.java | 11 +-
.../container/keyvalue/TestChunkManagerImpl.java | 113 +++--
.../container/keyvalue/TestKeyValueContainer.java | 65 ++-
.../container/keyvalue/TestTarContainerPacker.java | 204 ++++++--
.../ozoneimpl/TestContainerScrubberMetrics.java | 138 +++---
hadoop-hdds/docs/content/beyond/Containers.md | 1 -
hadoop-hdds/docs/content/start/OnPrem.md | 16 -
.../server/OzoneProtocolMessageDispatcher.java | 4 +-
.../hadoop/hdds/server/events/EventQueue.java | 4 +-
.../src/main/resources/webapps/datanode/dn.js | 2 -
hadoop-hdds/pom.xml | 8 -
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 12 +
.../hdds/scm/container/ReplicationManager.java | 19 +-
.../metrics/SCMContainerManagerMetrics.java | 3 +-
.../algorithms/SCMContainerPlacementMetrics.java | 3 +-
.../hadoop/hdds/scm/metadata/BigIntegerCodec.java | 5 +
.../metadata/DeletedBlocksTransactionCodec.java | 5 +
.../apache/hadoop/hdds/scm/metadata/LongCodec.java | 5 +
.../hdds/scm/metadata/X509CertificateCodec.java | 5 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 48 +-
.../hadoop/hdds/scm/node/SCMNodeMetrics.java | 3 +-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 3 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 5 +
.../hdds/scm/server/SCMContainerMetrics.java | 3 +-
.../hdds/scm/server/StorageContainerManager.java | 10 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 3 -
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 3 -
.../hadoop/hdds/scm/block/TestBlockManager.java | 84 +++-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 2 -
.../hadoop/hdds/scm/container/MockNodeManager.java | 5 +-
.../container/TestCloseContainerEventHandler.java | 5 +-
.../scm/container/TestSCMContainerManager.java | 30 +-
.../TestSCMContainerPlacementRackAware.java | 67 +--
.../hdds/scm/node/TestContainerPlacement.java | 10 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 180 ++++++++
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 89 +---
.../client/io/BlockOutputStreamEntryPool.java | 1 +
.../ozone/client/protocol/ClientProtocol.java | 8 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 19 +
.../hadoop/ozone/om/codec/OmBucketInfoCodec.java | 4 +
.../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 5 +
.../ozone/om/codec/OmMultipartKeyInfoCodec.java | 5 +
.../hadoop/ozone/om/codec/OmPrefixInfoCodec.java | 5 +
.../hadoop/ozone/om/codec/OmVolumeArgsCodec.java | 4 +
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 5 +
.../hadoop/ozone/om/codec/S3SecretValueCodec.java | 5 +
.../ozone/om/codec/TokenIdentifierCodec.java | 4 +
.../hadoop/ozone/om/codec/UserVolumeInfoCodec.java | 4 +
.../ozone/om/exceptions/NotLeaderException.java | 45 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 67 ++-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 39 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 53 ++-
.../ozone/om/helpers/OmMultipartKeyInfo.java | 56 ++-
.../hadoop/ozone/om/helpers/OmOzoneAclMap.java | 31 ++
.../hadoop/ozone/om/helpers/OmPrefixInfo.java | 18 +
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 29 +-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 32 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 78 ++--
.../ozone/security/acl/IAccessAuthorizer.java | 2 +-
.../apache/hadoop/ozone/util/OzoneVersionInfo.java | 3 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 8 +-
.../src/main/proto/OzoneManagerProtocol.proto | 11 +-
.../om/codec/TestOmMultipartKeyInfoCodec.java | 8 +-
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 56 ++-
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 83 ++++
.../ozone/om/helpers/TestOmMultipartKeyInfo.java | 64 +++
...TestOmBucketInfo.java => TestOmPrefixInfo.java} | 42 +-
.../hadoop/ozone/om/helpers/TestOmVolumeArgs.java | 87 ++++
hadoop-ozone/dev-support/checks/acceptance.sh | 2 +-
hadoop-ozone/dev-support/intellij/ozone-site.xml | 8 -
.../dist/dev-support/bin/dist-layout-stitching | 35 +-
hadoop-ozone/dist/pom.xml | 90 +++-
.../dist/src/main/compose/ozone-hdfs/docker-config | 1 -
.../dist/src/main/compose/ozone-mr/common-config | 3 +-
.../compose/{ozonescripts => ozone-om-ha-s3}/.env | 0
.../docker-compose.yaml | 10 +
.../{ozone-om-ha => ozone-om-ha-s3}/docker-config | 9 +-
.../compose/{ozones3 => ozone-om-ha-s3}/test.sh | 4 +-
.../src/main/compose/ozone-om-ha/docker-config | 1 -
.../src/main/compose/ozone-recon/docker-config | 3 +-
.../src/main/compose/ozone-topology/docker-config | 3 +-
.../dist/src/main/compose/ozone/docker-config | 3 +-
.../src/main/compose/ozoneblockade/docker-config | 1 -
.../dist/src/main/compose/ozoneperf/README.md | 20 +-
.../src/main/compose/ozoneperf/docker-compose.yaml | 11 +-
.../dist/src/main/compose/ozoneperf/docker-config | 3 +-
.../datasources.yml => ozoneperf/freon-ockg.yaml} | 18 +-
.../datasources.yml => ozoneperf/freon-rk.yaml} | 18 +-
.../prometheus => ozoneperf}/prometheus.yml | 10 +
.../src/main/compose/ozones3-haproxy/docker-config | 3 +-
.../dist/src/main/compose/ozones3/docker-config | 3 +-
hadoop-ozone/dist/src/main/compose/ozones3/test.sh | 2 +
.../src/main/compose/ozonescripts/docker-config | 1 -
.../src/main/compose/ozonesecure-mr/docker-config | 3 +-
.../src/main/compose/ozonesecure/docker-config | 3 +-
.../src/main/k8s/definitions/ozone/config.yaml | 1 -
.../examples/getting-started/config-configmap.yaml | 1 -
.../k8s/examples/minikube/config-configmap.yaml | 1 -
.../k8s/examples/ozone-dev/config-configmap.yaml | 1 -
.../main/k8s/examples/ozone/config-configmap.yaml | 1 -
.../dist/src/main/smoketest/basic/basic.robot | 2 +-
.../src/main/smoketest/basic/ozone-shell.robot | 2 +
.../dist/src/main/smoketest/s3/commonawslib.robot | 13 +-
.../main/smoketest/security/ozone-secure-s3.robot | 2 +-
.../conf/dn-audit-log4j2.properties | 0
.../dist/src/{main => shell}/conf/log4j.properties | 21 -
.../conf/om-audit-log4j2.properties | 0
.../conf/ozone-shell-log4j.properties | 0
.../dist/src/{main => shell}/conf/ozone-site.xml | 0
.../conf/scm-audit-log4j2.properties | 0
.../dist/src/shell/hdds}/hadoop-config.cmd | 0
.../dist/src/shell/hdds}/hadoop-config.sh | 0
.../dist/src/shell/hdds}/hadoop-daemons.sh | 0
.../dist/src/shell/hdds}/hadoop-functions.sh | 0
.../dist/src/shell/hdds}/workers.sh | 0
.../src/main/bin => dist/src/shell/ozone}/ozone | 9 +-
.../bin => dist/src/shell/ozone}/ozone-config.sh | 0
.../bin => dist/src/shell/ozone}/start-ozone.sh | 9 -
.../bin => dist/src/shell/ozone}/stop-ozone.sh | 9 -
.../shell/shellprofile.d/hadoop-ozone-manager.sh | 21 +-
.../src/shell}/shellprofile.d/hadoop-ozone.sh | 0
.../network-tests/src/test/compose/docker-config | 1 -
.../hadoop/ozone/insight/BaseInsightPoint.java | 56 +--
.../apache/hadoop/ozone/insight/InsightPoint.java | 6 +-
.../apache/hadoop/ozone/insight/LogSubcommand.java | 26 +-
.../ozone/insight/datanode/RatisInsight.java | 5 +
.../hadoop/ozone/insight/TestBaseInsightPoint.java | 73 +++
.../TestContainerStateManagerIntegration.java | 134 +++---
.../metrics/TestSCMContainerManagerMetrics.java | 6 +-
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 6 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 13 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 1 -
.../hadoop/ozone/TestContainerOperations.java | 19 +-
.../TestContainerStateMachineIdempotency.java | 3 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 3 -
.../hadoop/ozone/TestStorageContainerManager.java | 11 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 4 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 5 +-
.../client/rpc/TestContainerStateMachine.java | 26 +-
.../rpc/TestContainerStateMachineFailures.java | 118 +++++
.../client/rpc/TestOzoneRpcClientAbstract.java | 2 +-
.../ozone/client/rpc/TestWatchForCommit.java | 10 +-
.../ozone/container/ContainerTestHelper.java | 25 +-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 6 +-
.../ozone/om/TestOzoneManagerConfiguration.java | 2 -
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 155 ++++++-
.../ozone/ozShell/TestOzoneDatanodeShell.java | 104 +----
.../apache/hadoop/ozone/ozShell/TestS3Shell.java | 4 +-
.../hadoop/ozone/scm/TestAllocateContainer.java | 13 +-
.../hadoop/ozone/scm/TestContainerSmallFile.java | 17 +-
.../scm/TestGetCommittedBlockLengthAndPutKey.java | 13 +-
.../hadoop/ozone/scm/TestXceiverClientManager.java | 46 +-
.../hadoop/ozone/scm/TestXceiverClientMetrics.java | 9 +-
.../hadoop/ozone/scm/node/TestQueryNode.java | 5 +-
.../security/acl/TestOzoneNativeAuthorizer.java | 114 ++++-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 66 +--
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 22 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 49 ++
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 3 +
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 55 ++-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 3 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 14 +-
.../om/request/bucket/OMBucketCreateRequest.java | 2 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 2 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 4 +-
.../om/request/file/OMDirectoryCreateRequest.java | 22 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 16 +-
.../ozone/om/request/file/OMFileRequest.java | 4 +-
.../om/request/key/OMAllocateBlockRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 17 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 5 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 16 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 16 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 39 +-
.../S3InitiateMultipartUploadRequest.java | 3 +-
.../OzoneManagerHARequestHandlerImpl.java | 61 +--
...OzoneManagerProtocolServerSideTranslatorPB.java | 8 +-
.../ozone/security/acl/OzoneNativeAuthorizer.java | 52 ++-
.../ozone/web/ozShell/keys/GetKeyHandler.java | 2 +-
.../hadoop/ozone/om/TestS3BucketManager.java | 2 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 69 ++-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 3 +-
.../request/file/TestOMDirectoryCreateRequest.java | 73 ++-
.../om/request/file/TestOMFileCreateRequest.java | 21 +
.../om/request/key/TestOMAllocateBlockRequest.java | 4 +-
.../om/request/key/TestOMKeyCommitRequest.java | 4 +-
.../om/request/key/TestOMKeyCreateRequest.java | 4 +-
.../om/request/key/TestOMKeyDeleteRequest.java | 26 +-
.../om/request/key/TestOMKeyRenameRequest.java | 28 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 1 +
.../s3/bucket/TestS3BucketDeleteRequest.java | 3 +-
.../s3/bucket/TestS3BucketDeleteResponse.java | 3 +-
.../s3/multipart/TestS3MultipartResponse.java | 4 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 2 -
hadoop-ozone/pom.xml | 38 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 11 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 2 +-
.../recon/spi/impl/ContainerKeyPrefixCodec.java | 5 +
.../s3/endpoint/TestAbortMultipartUpload.java | 7 +-
.../hadoop/ozone/s3/endpoint/TestBucketDelete.java | 11 +-
.../hadoop/ozone/s3/endpoint/TestBucketHead.java | 5 +-
.../s3/endpoint/TestInitiateMultipartUpload.java | 9 +-
.../hadoop/ozone/s3/endpoint/TestListParts.java | 29 +-
.../s3/endpoint/TestMultipartUploadComplete.java | 33 +-
.../s3/endpoint/TestMultipartUploadWithCopy.java | 32 +-
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 3 +-
.../hadoop/ozone/s3/endpoint/TestPartUpload.java | 24 +-
.../hadoop/ozone/s3/endpoint/TestRootList.java | 3 +-
hadoop-ozone/tools/pom.xml | 5 -
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 68 +++
.../ozone/admin/om/GetServiceRolesSubcommand.java} | 47 +-
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 46 +-
.../hadoop/ozone/admin/om}/package-info.java | 5 +-
.../apache/hadoop/ozone/admin}/package-info.java | 5 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 23 +-
.../apache/hadoop/ozone/freon/OmKeyGenerator.java | 8 +-
.../ozone/freon/OzoneClientKeyGenerator.java | 8 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 12 +-
.../GenerateOzoneRequiredConfigurations.java | 4 +-
.../ozone/genesis/BenchMarkContainerStateMap.java | 11 +-
.../hadoop/ozone/genesis/BenchMarkOMClient.java | 30 +-
.../ozone/genesis/BenchMarkOzoneManager.java | 33 +-
.../apache/hadoop/ozone/genesis/BenchMarkSCM.java | 18 +-
pom.xml | 15 +-
283 files changed, 4499 insertions(+), 2420 deletions(-)
delete mode 100644 BUILDING.txt
create mode 100644 CONTRIBUTION.md
create mode 100644 README.md
delete mode 100644 README.txt
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerDataScanOrder.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmMultipartKeyInfo.java
copy
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/{TestOmBucketInfo.java
=> TestOmPrefixInfo.java} (50%)
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmVolumeArgs.java
copy hadoop-ozone/dist/src/main/compose/{ozonescripts => ozone-om-ha-s3}/.env
(100%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha =>
ozone-om-ha-s3}/docker-compose.yaml (92%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha =>
ozone-om-ha-s3}/docker-config (77%)
copy hadoop-ozone/dist/src/main/compose/{ozones3 => ozone-om-ha-s3}/test.sh
(96%)
mode change 100755 => 100644
copy
hadoop-ozone/dist/src/main/compose/{common/grafana/provisioning/datasources/datasources.yml
=> ozoneperf/freon-ockg.yaml} (77%)
mode change 100755 => 100644
copy
hadoop-ozone/dist/src/main/compose/{common/grafana/provisioning/datasources/datasources.yml
=> ozoneperf/freon-rk.yaml} (78%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/compose/{common/prometheus =>
ozoneperf}/prometheus.yml (72%)
rename hadoop-ozone/dist/src/{main => shell}/conf/dn-audit-log4j2.properties
(100%)
rename hadoop-ozone/dist/src/{main => shell}/conf/log4j.properties (84%)
rename hadoop-ozone/dist/src/{main => shell}/conf/om-audit-log4j2.properties
(100%)
rename hadoop-ozone/dist/src/{main => shell}/conf/ozone-shell-log4j.properties
(100%)
rename hadoop-ozone/dist/src/{main => shell}/conf/ozone-site.xml (100%)
rename hadoop-ozone/dist/src/{main => shell}/conf/scm-audit-log4j2.properties
(100%)
rename {hadoop-hdds/common/src/main/bin =>
hadoop-ozone/dist/src/shell/hdds}/hadoop-config.cmd (100%)
rename {hadoop-hdds/common/src/main/bin =>
hadoop-ozone/dist/src/shell/hdds}/hadoop-config.sh (100%)
rename {hadoop-hdds/common/src/main/bin =>
hadoop-ozone/dist/src/shell/hdds}/hadoop-daemons.sh (100%)
rename {hadoop-hdds/common/src/main/bin =>
hadoop-ozone/dist/src/shell/hdds}/hadoop-functions.sh (100%)
rename {hadoop-hdds/common/src/main/bin =>
hadoop-ozone/dist/src/shell/hdds}/workers.sh (100%)
rename hadoop-ozone/{common/src/main/bin => dist/src/shell/ozone}/ozone (97%)
rename hadoop-ozone/{common/src/main/bin =>
dist/src/shell/ozone}/ozone-config.sh (100%)
rename hadoop-ozone/{common/src/main/bin =>
dist/src/shell/ozone}/start-ozone.sh (91%)
rename hadoop-ozone/{common/src/main/bin =>
dist/src/shell/ozone}/stop-ozone.sh (89%)
copy hadoop-hdds/docs/dev-support/bin/generate-site.sh =>
hadoop-ozone/dist/src/shell/shellprofile.d/hadoop-ozone-manager.sh (67%)
mode change 100755 => 100644
rename hadoop-ozone/{common/src/main =>
dist/src/shell}/shellprofile.d/hadoop-ozone.sh (100%)
create mode 100644
hadoop-ozone/insight/src/test/java/org/apache/hadoop/ozone/insight/TestBaseInsightPoint.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/OzoneAdmin.java
copy
hadoop-ozone/{insight/src/main/java/org/apache/hadoop/ozone/insight/ListSubCommand.java
=>
tools/src/main/java/org/apache/hadoop/ozone/admin/om/GetServiceRolesSubcommand.java}
(54%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCommands.java
=>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java
(54%)
copy {hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli =>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om}/package-info.java
(92%)
copy {hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli =>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin}/package-info.java
(92%)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-commits-h...@hadoop.apache.org
