This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-3816-ec
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from ad790b6 HDDS-4883. Persist replicationIndex on datanode side (#2069)
add 6f9c3f2 HDDS-5060. [SCM HA Security] Make InterSCM grpc channel
secure. (#2187)
add bddc873 HDDS-4515. Datanodes should be able to persist and load CRL
(#2181)
add ce29843 HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT
(#2184)
add bfb6be8 HDDS-5152. Fix Suggested leader in Client. (#2189)
add bccc439 HDDS-5147. Intermittent test failure in
TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (#2188)
add a920f25 HDDS-5153. Decommissioning a dead node should complete
immediately (#2190)
add 66a411b HDDS-4585. Support bucket acl operation in S3g (#1701)
add 5dd0943 HDDS-5144. Create github check to alert when dependency tree
is changed (#2177)
add 10312fb HDDS-4983. Display key offset for each block in command key
info (#2051)
add 799c38e Revert "HDDS-5144. Create github check to alert when
dependency tree is changed (#2177)"
add dcac0f1 HDDS-5166. Remove duplicate assignment of OZONE_OPTS for
freon and sh (#2195)
add 48b84c5 HDDS-2212. Genconf tool should generate config files for
secure clust… (#1788)
add a3e7f04 HDDS-5182. Acceptance test may exit with 0 in case of error
(#2212)
add 2e07c9f HDDS-5178. Update project information of Contribution
guideline (#2210)
add 8bbbf3f HDDS-5177. Update link of weekly calls in README (#2209)
add 00eba3e HDDS-5144. Create github check to alert when dependency tree
is changed (#2199)
add 142a2dd HDDS-5185. Update commons-io to 2.8.0 (#2215)
add d430ff3 HDDS-5159. Make periodic disk check interval configurable.
(#2193)
add 74496a7 HDDS-5196. Fix typo "vaild" in HeartbeatEndpointTask.java and
RegisterEndpointTask (#2222).
add 2061d7d HDDS-5192. Intermittent failure in TestOzoneRpcClient due to
volume name conflict (#2220)
add 5e8ae99 HDDS-5186. If primordial SCM id is set, a non-HA cluster can
not be initialized. (#2218)
add a16e648 HDDS-5126.Recon should check new containers of a container
report with batch (#2172)
add 2c37a15 HDDS-5183. For AccessControlException do not perform
failover. (#2213)
add 75c5c10 HDDS-5099. Error with unit test for hdds.container-service
TestSchemaOneBackwardsCompatibility (#2183)
add 8e8a01d HDDS-5197. Pass option variables to OZONE_OPTS before adding
default GC opts (#2225)
add 7ba1499 HDDS-5198. Remove JBoss repo definition (#2226)
add 2c4a4de HDDS-5173. Divide snapshot related work into
notifyInstallSnapshotFromLeader and reinitialize for SCMStateMachine. (#2224)
add 49538aa HDDS-5207. Intermittent failure in
TestRatisPipelineProvider#testCreatePipelineWithFactorThree (#2233)
add 3e66778 HDDS-5200. Fix scm roles command if one of the host is
unresolvable. (#2227)
add abe136f HDDS-5203. Allow suppressing deprecation warning for HADOOP_
variables (#2231)
add 248d72d Bump logical release name of Ozone 1.2 (#2238)
add b5cbb65 HDDS-5164. Improve client and server logging. (#2234)
add 7262004 HDDS-5089. On-demand disk checker for hdds volume (#2158)
add 842e6f6 HDDS-4729. Add token support for container admin operations
(#2186)
add 03620da HDDS-5059. Increase number of client retries/ failovers to
OMs (#2171)
add f8a06e0 HDDS-5202. Use scm#checkLeader before processing client
requests . (#2229)
add 6075acb HDDS-5205. Make admin check work for SCM HA cluster (#2236)
add 0f6bb36 HDDS-5040. Append om1,om2,om3 to ozone.administrators config
for secure OM HA docker config (#2245)
add a7c8695 HDDS-5160. Remove some Freon integration tests (#2194)
add 6b24c18 HDDS-4976. Add container replica related commands to debug
(#2070)
add fd71f09 The modificationTime should modify when setAcl success (#2242)
add 4189088 HDDS-5165. OM DB checkpoint servlet not accessible in a
secure cluster (#2200)
add eff4914 HDDS-5208. bump rocksdb version to 6.20.3 (#2235)
add 36778af HDDS-5128. Return specific error messages for S3-SDK (#2169)
add 19f989f HDDS-5176. Increase default block cache capacity for
Datanodes (#2207)
add 9cab560 HDDS-5155. Add revokeCertificate to
SCMSecurityProtocolServer. (#2191)
add 727b258 HDDS-5189. fix markdown files to make them hugo83 compatible
(and valid) (#2232)
add 7a4a93e HDDS-5229. Update copyright year in NOTICE (#2248)
add ac7fc7e HDDS-5082. Create unit (!) test for OzoneClient (#2138)
add 2254abf HDDS-5216. Fix race condition causing SCM failOverProxy which
is causing failover wrongly. (#2247)
add fcc8e92 HDDS-5187. Avoid Maven connection errors in CI (#2237)
add 10a5b39 HDDS-5113. Disable flaky
TestBlockDeletingService#testContainerThrottle
add 91993c1 HDDS-5234.Change default grpc and ratis ports for scm ha
(#2251)
add ea1ed7c HDDS-5238. Disable animal-sniffer maven plugin (#2253)
add 9f6d7f2 HDDS-5167. Add clean option for Freon randomkeys to clean
generated objects (#2198)
add d63dd6b HDDS-5239. Create http Basedir if not exist (#2255)
add 5a50866 HDDS-5209. Datanode hasEnoughSpace check should apply on
volume instead of global DN (#2246)
add 30fe6ea HDDS-5245. Fix OzoneContainer TLS configuration. (#2264)
add 4fd8187 HDDS-5206. Support revoking S3 secret (#2239)
add 08375d7 HDDS-5246. Wait for ever to obtain CA list which is needed
during OM/DN startup (#2266)
add 70ef475 HDDS-5233. SCM subsequent init failed when previous scm init
failed. (#2249)
add 09c2278 HDDS-5257. Avoid SCM call to get CA certs in non-HA from OM.
(#2273)
add e1acbb0 HDDS-5256. Fix fall back of config in SCM HA Cluster (#2272)
add 61a237a HDDS-5261. Delete Useless ozone subcommand (#2275)
add e50fe52 HDDS-5041. Use getShortUserName in getTrashRoot(s) (#2244)
add fc7b0bd HDDS-5260. Bump node to v16.2.0 for Recon (#2276)
add 9080fc3 HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen
in SCM log. (#2267)
add e00fa80 HDDS-5206. Addendum: Support revoking S3 secret (#2270)
add 8d29be1 HDDS-5142. Make generic streaming client/service for
container re-replication, data read, scm/om snapshot download (#2256)
add ab8f07d HDDS-5249. Race Condition between Full and Incremental
Container Reports (#2268)
add 931a041 HDDS-5266 Misspelt words in
S3MultipartUploadCommitPartRequest.java line 202 (#2279)
add 355096b HDDS-5236. Require block token for more operations (#2254)
add 19f5bb2 HDDS-5250. Build integration tests with Maven cache (#2269)
add 0b4779c HDDS-5073. Use ReplicationConfig on client side (#2136)
new 9c4008d Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
new c960ec7 Merge remote-tracking branch 'origin/HDDS-3816-ec' into
HDDS-3816-ec
add 2dbf04e HDDS-5191. Increase default pvc storage size (#2219)
add 000e7c5 HDDS-5252. PipelinePlacementPolicy filter out datanodes with
not enough space. (#2271)
add 2671b48 HDDS-5219. Limit number of bad volumes by
dfs.datanode.failed.volumes.tolerated. (#2243)
add d9ae1b3 HDDS-5241. SCM UI should have leader/follower and Primordial
SCM information (#2260)
add 53bbabf HDDS-5275. Datanode Report Publisher publishes one extra
report after DN shutdown (#2283)
add 4301315 HDDS-5158. Add documentation for SCM HA Security. (#2205)
add d567360 HDDS-5273. Handle unsecure cluster convert to secure cluster
for SCM. (#2281)
add 00b675d HDDS-4927. Determine over and under utilized datanodes in
Container Balancer. (#2230)
add 8586815 HDDS-4043. allow deletion from Trash directory without
-skipTrash option (#2110)
add fa4ddbc HDDS-4936. Change ozone groupId from org.apache.hadoop to
org.apache.ozone (#2018)
add 3ffbdd0 HDDS-4993. Add guardrail for reserved buffer size when DN
reads a chunk (#2058)
add e5a774f HDDS-5123. Use the pre-created apache/ozone-testkrb5 image
during secure acceptance tests (#2165)
add 5fd9780 HDDS-5272. Make ozonefs.robot execution repeatable (#2280)
add 12e2918 HDDS-5280. Make XceiverClientManager creation when necessary
in ContainerOperationClient (#2289)
add 6b09219 HDDS-5264. SCM should send token for CloseContainer command
(#2277)
add e35dde9 HDDS-5151. Support ByteBuffer read in OzoneInputStream (#2203)
add 96d9ed7 HDDS-5240. Fix out of bound exception when loading
auditparser (#2258)
add 69b92a0 HDDS-5281. Add reinitialize() for SequenceIdGenerator. (#2292)
new f904539 Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/buildenv/Dockerfile | 20 --
.github/buildenv/entrypoint.sh | 18 --
.github/workflows/post-commit.yml | 103 ++++---
hadoop-hdds/client/pom.xml | 30 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 29 +-
.../hadoop/hdds/scm/storage/ByteArrayReader.java | 67 ++++
.../hadoop/hdds/scm/storage/ByteBufferReader.java | 71 +++++
.../hdds/scm/storage/ByteReaderStrategy.java | 41 +++
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 38 ++-
.../hdds/scm/storage/TestBlockInputStream.java | 35 +++
hadoop-hdds/common/pom.xml | 26 +-
.../apache/hadoop/hdds/DFSConfigKeysLegacy.java | 5 -
.../java/org/apache/hadoop/hdds/HddsUtils.java | 17 +
.../hadoop/hdds/client/RatisReplicationConfig.java | 15 +
.../hadoop/hdds/client/ReplicationConfig.java | 55 ++++
.../hdds/client/StandaloneReplicationConfig.java | 9 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 4 +-
.../hadoop/hdds/client/TestReplicationConfig.java | 147 +++++++++
hadoop-hdds/config/pom.xml | 10 +-
hadoop-hdds/container-service/pom.xml | 42 +--
.../container/common/report/ReportPublisher.java | 4 +-
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../CloseContainerCommandHandler.java | 9 +-
.../states/endpoint/HeartbeatEndpointTask.java | 4 +
.../container/common/volume/HddsVolumeChecker.java | 11 +-
.../container/common/volume/MutableVolumeSet.java | 42 ++-
.../container/keyvalue/helpers/ChunkUtils.java | 12 +
.../keyvalue/impl/ChunkManagerDummyImpl.java | 6 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 5 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 5 +-
.../hadoop/ozone/protocol/commands/SCMCommand.java | 10 +
.../container/common/TestBlockDeletingService.java | 2 +-
.../TestSchemaOneBackwardsCompatibility.java | 4 +-
.../common/impl/TestContainerPersistence.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 5 +-
.../common/report/TestReportPublisher.java | 9 +-
.../statemachine/TestDatanodeConfiguration.java | 12 +
.../container/common/volume/TestVolumeSet.java | 5 +-
.../common/volume/TestVolumeSetDiskChecks.java | 11 +-
.../keyvalue/TestKeyValueBlockIterator.java | 2 +-
.../keyvalue/TestKeyValueContainerCheck.java | 2 +-
.../container/keyvalue/TestKeyValueHandler.java | 3 +-
.../keyvalue/impl/CommonChunkManagerTestCases.java | 33 ++
.../container/ozoneimpl/TestContainerReader.java | 2 +-
.../container/ozoneimpl/TestOzoneContainer.java | 3 +-
hadoop-hdds/docs/content/feature/SCM-HA.md | 68 +++-
hadoop-hdds/docs/content/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
hadoop-hdds/docs/content/interface/O3fs.md | 4 +-
hadoop-hdds/docs/content/interface/O3fs.zh.md | 8 +-
hadoop-hdds/docs/content/interface/Ofs.md | 4 +-
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md | 6 +-
.../docs/content/recipe/SparkOzoneFSK8S.zh.md | 6 +-
hadoop-hdds/docs/pom.xml | 6 +-
hadoop-hdds/framework/pom.xml | 34 +-
...etManager.java => ContainerTokenGenerator.java} | 58 ++--
.../token/ContainerTokenSecretManager.java | 30 +-
.../hdds/scm/protocol/TestReplicationConfig.java | 86 ------
hadoop-hdds/hadoop-dependency-client/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 6 +-
hadoop-hdds/interface-admin/pom.xml | 10 +-
hadoop-hdds/interface-client/pom.xml | 6 +-
hadoop-hdds/interface-server/pom.xml | 10 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 1 +
hadoop-hdds/pom.xml | 78 ++---
hadoop-hdds/server-scm/pom.xml | 56 ++--
.../scm/container/CloseContainerEventHandler.java | 9 +
.../hdds/scm/container/ReplicationManager.java | 12 +-
.../scm/container/balancer/ContainerBalancer.java | 341 ++++++++++++++++++---
.../balancer/ContainerBalancerConfiguration.java | 52 ++--
.../balancer/ContainerBalancerMetrics.java | 127 ++++++++
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 13 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 1 +
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 29 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 29 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 17 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 4 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 10 +
.../hdds/scm/server/StorageContainerManager.java | 52 +++-
.../main/resources/webapps/scm/scm-overview.html | 8 +
.../hadoop/hdds/scm/container/MockNodeManager.java | 88 ++++--
.../container/balancer/TestContainerBalancer.java | 219 +++++++++++++
.../apache/hadoop/hdds/scm/ha/TestSCMContext.java | 6 +-
.../hadoop/hdds/scm/ha/TestSCMServiceManager.java | 4 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 4 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 25 ++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 2 +-
hadoop-hdds/test-utils/pom.xml | 6 +-
hadoop-hdds/tools/pom.xml | 26 +-
.../hdds/scm/cli/ContainerOperationClient.java | 42 ++-
.../org/apache/hadoop/hdds/scm/cli/ScmOption.java | 14 +-
hadoop-ozone/client/pom.xml | 14 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 106 ++++---
.../hadoop/ozone/client/io/KeyInputStream.java | 51 ++-
.../hadoop/ozone/client/io/OzoneInputStream.java | 15 +-
.../ozone/client/protocol/ClientProtocol.java | 58 ++++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 69 +++--
hadoop-ozone/common/pom.xml | 26 +-
.../hadoop/ozone/client/io/LengthInputStream.java | 17 +-
hadoop-ozone/csi/pom.xml | 14 +-
hadoop-ozone/datanode/pom.xml | 14 +-
hadoop-ozone/dev-support/checks/coverage.sh | 3 +-
hadoop-ozone/dev-support/checks/integration.sh | 2 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dev-support/docker/Dockerfile | 2 +-
.../intellij/runConfigurations/CsiServer.xml | 4 +-
.../intellij/runConfigurations/Datanode1.xml | 2 +-
.../intellij/runConfigurations/Datanode2.xml | 2 +-
.../intellij/runConfigurations/Datanode3.xml | 2 +-
.../intellij/runConfigurations/FreonStandalone.xml | 4 +-
.../intellij/runConfigurations/OzoneFsShell.xml | 2 +-
.../intellij/runConfigurations/OzoneManager.xml | 2 +-
.../runConfigurations/OzoneManagerInit.xml | 2 +-
.../intellij/runConfigurations/OzoneShell.xml | 2 +-
.../intellij/runConfigurations/Recon.xml | 4 +-
.../intellij/runConfigurations/S3Gateway.xml | 4 +-
.../runConfigurations/StorageContainerManager.xml | 2 +-
.../StorageContainerManagerInit.xml | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 3 +
.../dist/dev-support/bin/dist-tar-stitching | 6 +-
hadoop-ozone/dist/pom.xml | 73 ++---
.../docker-image/docker-krb5/Dockerfile-krb5 | 35 ---
.../common/docker-image/docker-krb5/README.md | 34 --
.../common/docker-image/docker-krb5/kadm5.acl | 20 --
.../common/docker-image/docker-krb5/launcher.sh | 25 --
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../main/compose/ozone-mr/hadoop27/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop31/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop32/docker-config | 4 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 44 ++-
.../src/main/compose/ozonesecure-ha/docker-config | 24 +-
.../compose/ozonesecure-ha/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-ha/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-ha/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-ha/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-ha/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-ha/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-ha/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-ha/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-ha}/krb5.conf | 4 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 30 +-
.../src/main/compose/ozonesecure-mr/docker-config | 25 +-
.../compose/ozonesecure-mr/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-mr/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-mr/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-mr/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-mr/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-mr/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-mr/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-mr/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-mr}/krb5.conf | 6 +-
.../dist/src/main/compose/ozonesecure/.env | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 23 +-
.../src/main/compose/ozonesecure/docker-config | 24 +-
.../main/compose/ozonesecure/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../src/main/compose/ozonesecure/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../src/main/compose/ozonesecure/keytabs/om.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../compose/ozonesecure/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure}/krb5.conf | 4 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
.../definitions/ozone/definitions/persistence.yaml | 4 +-
.../examples/ozone-ha/datanode-statefulset.yaml | 2 +-
.../k8s/examples/ozone/datanode-statefulset.yaml | 2 +-
hadoop-ozone/dist/src/main/keytabs/HTTP.keytab | Bin 0 -> 432 bytes
hadoop-ozone/dist/src/main/keytabs/dn.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/hadoop.keytab | Bin 0 -> 146 bytes
hadoop-ozone/dist/src/main/keytabs/jhs.keytab | Bin 0 -> 142 bytes
hadoop-ozone/dist/src/main/keytabs/nm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/om.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/recon.keytab | Bin 0 -> 296 bytes
hadoop-ozone/dist/src/main/keytabs/rm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/s3g.keytab | Bin 0 -> 434 bytes
hadoop-ozone/dist/src/main/keytabs/scm.keytab | Bin 0 -> 586 bytes
hadoop-ozone/dist/src/main/keytabs/testuser.keytab | Bin 0 -> 450 bytes
.../dist/src/main/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../dist/src/main/keytabs/update-keytabs.sh | 77 +++++
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 58 ++--
.../dist/src/main/license/update-jar-report.sh | 2 +-
.../src/main/smoketest/admincli/container.robot | 11 +-
.../dist/src/main/smoketest/basic/links.robot | 10 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 6 +-
.../dist/src/main/smoketest/cli/classpath.robot | 14 +-
.../dist/src/main/smoketest/commonlib.robot | 8 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 10 +-
.../main/smoketest/security/ozone-secure-fs.robot | 18 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 42 +--
.../fault-injection-test/mini-chaos-tests/pom.xml | 24 +-
.../fault-injection-test/network-tests/pom.xml | 6 +-
hadoop-ozone/fault-injection-test/pom.xml | 6 +-
hadoop-ozone/insight/pom.xml | 42 +--
.../hadoop/ozone/insight/BaseInsightPoint.java | 4 +-
hadoop-ozone/integration-test/pom.xml | 70 ++---
.../ozone/client/rpc/read/TestKeyInputStream.java | 63 +++-
.../container/metrics/TestContainerMetrics.java | 2 +-
.../server/TestSecureContainerServer.java | 2 +-
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 31 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 6 +
.../TestDatanodeHddsVolumeFailureToleration.java | 131 ++++++++
.../hadoop/ozone/shell/TestOzoneShellHA.java | 99 ++++++
hadoop-ozone/interface-client/pom.xml | 10 +-
hadoop-ozone/interface-storage/pom.xml | 30 +-
hadoop-ozone/native-client/README.md | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 60 ++--
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 24 ++
.../S3MultipartUploadCompleteRequest.java | 3 +-
hadoop-ozone/ozonefs-common/pom.xml | 42 +--
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 32 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 29 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 30 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 16 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 14 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 10 +-
hadoop-ozone/ozonefs/pom.xml | 14 +-
hadoop-ozone/pom.xml | 142 ++++-----
hadoop-ozone/recon-codegen/pom.xml | 10 +-
hadoop-ozone/recon/pom.xml | 38 +--
hadoop-ozone/s3gateway/pom.xml | 42 +--
.../hadoop/ozone/client/OzoneBucketStub.java | 5 +-
hadoop-ozone/tools/pom.xml | 50 +--
.../ozone/debug/container/ContainerCommands.java | 2 +-
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../ozone/freon/ClosedContainerReplicator.java | 3 +-
.../containergenerator/GeneratorDatanode.java | 2 +-
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 2 +-
.../hadoop/ozone/shell/keys/CopyKeyHandler.java | 28 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 27 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 31 +-
.../tools/src/test/resources/testaudit.log | 3 +
.../resources/{testaudit.log => testloadaudit.log} | 24 +-
pom.xml | 4 +-
240 files changed, 3485 insertions(+), 1481 deletions(-)
delete mode 100644 .github/buildenv/Dockerfile
delete mode 100755 .github/buildenv/entrypoint.sh
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteArrayReader.java
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteBufferReader.java
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteReaderStrategy.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/TestReplicationConfig.java
create mode 100644 hadoop-hdds/docs/content/feature/scm-secure-ha.png
copy
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/{ContainerTokenSecretManager.java
=> ContainerTokenGenerator.java} (50%)
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocol/TestReplicationConfig.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancer.java
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/Dockerfile-krb5
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/README.md
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/kadm5.acl
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/launcher.sh
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure-ha}/krb5.conf (96%)
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure-mr}/krb5.conf (91%)
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser2.keytab
rename hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure}/krb5.conf (96%)
create mode 100755 hadoop-ozone/dist/src/main/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/hadoop.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/jhs.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/nm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/rm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser2.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/update-keytabs.sh
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/volume/TestDatanodeHddsVolumeFailureToleration.java
copy hadoop-ozone/tools/src/test/resources/{testaudit.log =>
testloadaudit.log} (69%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
