This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch HDDS-2939
in repository https://gitbox.apache.org/repos/asf/ozone.git.
omit dced857 HDDS-4717. Fix TestOzoneFileSystemV1 and TestObjectStoreV1
cases (#1815)
omit 3825571 HDDS-4658. LookupKey: do lookup in dir and file tables (#1775)
omit af7d54c HDDS-2942. Putkey : create key table entries for intermediate
directories in the key path (#1764)
omit 62d01b8 HDDS-4514. AllocateBlock : lookup and update open file table
for the given path (#1679)
omit 31cb08f HDDS-4596. Directory table, fileTable and openFile Table is
missing from the OM DB Definition. (#1724)
omit 0be5c10 HDDS-4321. Fix compilation errors : merge HDDS-4308 and
HDDS-4473 changes into the branch (#1668)
omit eb84c94 HDDS-4358: Delete : make delete an atomic operation (#1607)
omit b416406 HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
omit a7d568f HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
omit 9aca1d5 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
omit eaf6859 HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
add 984cf66 HDDS-4667. BlockInputStream should give up read retry if
pipeline is not updated (#1774)
add a7e24fd HDDS-4621. Set, add and remove ACL have no audit logs (#1734)
add 3168991 HDDS-4671. Use mvn to get ozone.version instead of complex
shell commands (#1777)
add 1c5a34d HDDS-4679. Intermittent failure in TestSCMRestart. (#1792)
add ad207a3 HDDS-4403. Update the container replica history to the Recon
DB lazily instead of for every report (#1670)
add abb6be3 HDDS-4570. Reduce memory footprint of OMUpdateEventBatch for
Recon (#1680)
add bc49c45 HDDS-4682. Fix several typos in docs (#1787)
add 01a5785 HDDS-4671. Use mvn to get ozone.version instead of complex
shell commands (addendum) (#1794)
add 159b0c6 HDDS-4556. Add smoketest for ozonescripts environment (#1671)
add 2c8ba0f HDDS-4162. Use BeforeClass to init test cluster in
TestOzoneFileSystem (#1791)
add befba44 HDDS-4681. SCM webui display wrong Node counts. (#1789)
add e1601c5 HDDS-4674. Bucket usedBytes inaccurate because of
preallocated blocks (#1783)
add ee4c8f1 HDDS-4663. Missing "Expose any volume" content in S3.md (ZH
Docs) (#1771)
add 6fe3e8a HDDS-4680. Change default OM Node ID from UUID to a constant
(#1793)
add 60bf992 HDDS-4631. Fix client set quota with 0. (#1746)
add a02ea47 HDDS-4672. Add warning log when old volume and bucket set
quota (#1781)
add 5a35cbd change quota unit suffix from BYTES to B (#1813)
add c68a3c3 HDDS-4699. Remove unused OzoneClientUtils (#1808)
add 03943ac HDDS-4573. Inconsistent path when the item ozone.recon.db.dir
is not configured (#1809)
add 6019dda HDDS-4707. Failed to generate reports using "mvn site" (#1803)
add 026c625 HDDS-4593. Refine IAccessAuthorizer interface to do recursive
ACL check on a path (#1790)
add c06851e HDDS-4571. DN handle expired certificates when validate block
token. (#1754)
add 46fa85d HDDS-4641. OM handle expired certificate when verify token
signature. (#1755)
add 1807e67 HDDS-4724. Revert column family names of Datanode db to avoid
compatibility issue. (#1824)
add 896126b HDDS-4669.TestOzoneFileSystem#testTrash() fails when OM Ratis
is enabled (#1818)
add 39e90f9 HDDS-4721. Allow Default Space Quota Unit as Bytes (#1819)
add 2aa483f HDDS-4728.ozone debug chunkinfo is not working correctly.
(#1829)
add 3d70fab HDDS-4710. PipelinePlacementPolicy altered to randomly pick
(#1821)
add bc3e3e5 HDDS-4525. Replace Hadoop variables and functions in Ozone
shell scripts with Ozone-specific ones (#1667)
add b83d4b2 HDDS-4725. Change metrics unit from nanosecond to millisecond
(#1823)
add bd00a04 HDDS-4688. Update Hadoop version to 3.2.2 (#1795)
new d4c16ed HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
new 005c464 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
new ad3aca5 HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
new 0d9a65b HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
new 2433e01 HDDS-4358: Delete : make delete an atomic operation (#1607)
new f0fa1b6 HDDS-4321. Fix compilation errors : merge HDDS-4308 and
HDDS-4473 changes into the branch (#1668)
new a6984f5 HDDS-4596. Directory table, fileTable and openFile Table is
missing from the OM DB Definition. (#1724)
new 09023c4 HDDS-4514. AllocateBlock : lookup and update open file table
for the given path (#1679)
new 695c838 HDDS-2942. Putkey : create key table entries for intermediate
directories in the key path (#1764)
new d029dca HDDS-4658. LookupKey: do lookup in dir and file tables (#1775)
new cec383b HDDS-4717. Fix TestOzoneFileSystemV1 and TestObjectStoreV1
cases (#1815)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (dced857)
\
N -- N -- N refs/heads/HDDS-2939 (cec383b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 1 +
hadoop-hdds/client/pom.xml | 5 +
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 8 +-
.../hadoop/hdds/scm/XceiverClientMetrics.java | 4 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 4 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 32 +-
.../hdds/scm/storage/TestBlockInputStream.java | 77 ++
.../src/test/resources/log4j.properties | 0
hadoop-hdds/common/src/main/conf/hadoop-env.cmd | 90 --
hadoop-hdds/common/src/main/conf/hadoop-env.sh | 451 -------
hadoop-hdds/common/src/main/conf/ozone-env.sh | 279 ++++
.../org/apache/hadoop/hdds/client/OzoneQuota.java | 98 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +-
.../common/src/main/resources/ozone-default.xml | 2 +-
.../container/common/helpers/ContainerMetrics.java | 6 +-
.../container/common/impl/HddsDispatcher.java | 5 +-
.../common/transport/server/ratis/CSMMetrics.java | 7 +-
.../server/ratis/ContainerStateMachine.java | 2 +-
.../metadata/DatanodeSchemaTwoDBDefinition.java | 6 +-
hadoop-hdds/docs/content/design/decommissioning.md | 10 +-
hadoop-hdds/docs/content/interface/S3.zh.md | 19 +
hadoop-hdds/docs/content/tools/AuditParser.md | 2 +-
hadoop-hdds/docs/dev-support/bin/generate-site.sh | 2 +-
.../hdds/security/token/BlockTokenVerifier.java | 13 +
.../x509/certificate/client/CertificateClient.java | 12 +
.../client/DefaultCertificateClient.java | 30 +-
.../server/OzoneProtocolMessageDispatcher.java | 5 +-
.../hadoop/hdds/utils/db/DBConfigFromFile.java | 12 +-
.../apache/hadoop/hdds/server/TestJsonUtils.java | 14 +-
.../hdds/scm/container/SCMContainerManager.java | 6 +-
.../scm/container/states/ContainerStateMap.java | 2 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 12 +-
.../main/resources/webapps/scm/scm-overview.html | 14 +-
.../src/main/resources/webapps/scm/scm.js | 25 +-
.../scm/TestStorageContainerManagerHttpServer.java | 2 +
.../org/apache/hadoop/ozone/client/BucketArgs.java | 3 +
.../hadoop/ozone/client/OzoneClientUtils.java | 37 -
.../org/apache/hadoop/ozone/client/VolumeArgs.java | 8 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 40 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 4 +-
.../hadoop/ozone/om/helpers/OmBucketArgs.java | 8 +
.../hadoop/ozone/security/acl/RequestContext.java | 48 +-
hadoop-ozone/dev-support/checks/acceptance.sh | 2 +-
hadoop-ozone/dev-support/checks/blockade.sh | 2 +-
hadoop-ozone/dev-support/checks/kubernetes.sh | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 14 +-
.../compose/{ozonescripts => compatibility}/.env | 0
.../{ozone => compatibility}/docker-compose.yaml | 5 -
.../compose/{ozone => compatibility}/docker-config | 10 +-
.../compose/{failing1 => compatibility}/test.sh | 12 +-
hadoop-ozone/dist/src/main/compose/ozone-csi/.env | 2 +-
.../src/main/compose/ozone-csi/docker-compose.yaml | 8 +-
.../dist/src/main/compose/ozone-csi/docker-config | 3 +
hadoop-ozone/dist/src/main/compose/ozone-ha/.env | 2 +-
.../dist/src/main/compose/ozone-ha/docker-config | 3 +
.../dist/src/main/compose/ozone-mr/common-config | 3 +
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 5 +-
.../main/compose/ozone-mr/hadoop27/docker-config | 3 +
.../dist/src/main/compose/ozone-mr/hadoop31/.env | 2 +-
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 13 +-
.../main/compose/ozone-mr/hadoop31/docker-config | 3 +
.../dist/src/main/compose/ozone-mr/hadoop32/.env | 4 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 11 +-
.../main/compose/ozone-mr/hadoop32/docker-config | 3 +
.../dist/src/main/compose/ozone-om-ha-s3/.env | 2 +-
.../compose/ozone-om-ha-s3/docker-compose.yaml | 12 +-
.../src/main/compose/ozone-om-ha-s3/docker-config | 3 +
.../src/main/compose/ozone-om-ha/docker-config | 3 +
.../dist/src/main/compose/ozone-topology/.env | 2 +-
.../compose/ozone-topology/docker-compose.yaml | 16 +-
.../src/main/compose/ozone-topology/docker-config | 3 +
hadoop-ozone/dist/src/main/compose/ozone/.env | 2 +-
hadoop-ozone/dist/src/main/compose/ozone/README.md | 2 +-
.../src/main/compose/ozone/docker-compose.yaml | 10 +-
.../dist/src/main/compose/ozone/docker-config | 3 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
.../src/main/compose/ozoneblockade/docker-config | 3 +
.../src/main/compose/ozones3-haproxy/docker-config | 3 +
.../src/main/compose/ozonescripts/docker-config | 3 +
.../dist/src/main/compose/ozonescripts/start.sh | 8 +-
.../dist/src/main/compose/ozonescripts/stop.sh | 2 +-
.../{ozone-om-ha-s3 => ozonescripts}/test.sh | 20 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 4 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 16 +-
.../src/main/compose/ozonesecure-mr/docker-config | 5 +
.../dist/src/main/compose/ozonesecure-om-ha/.env | 2 +-
.../compose/ozonesecure-om-ha/docker-compose.yaml | 20 +-
.../main/compose/ozonesecure-om-ha/docker-config | 3 +
.../dist/src/main/compose/ozonesecure/.env | 2 +-
.../main/compose/ozonesecure/docker-compose.yaml | 28 +-
.../src/main/compose/ozonesecure/docker-config | 3 +
hadoop-ozone/dist/src/main/compose/test-all.sh | 2 +-
.../src/main/compose/upgrade/docker-compose.yaml | 7 +
.../dist/src/main/compose/upgrade/docker-config | 3 +
.../src/main/smoketest/basic/ozone-shell-lib.robot | 10 -
.../dist/src/main/smoketest/cli/classpath.robot | 46 +
.../dist/src/main/smoketest/cli/envvars.robot | 77 ++
.../validate.robot => compatibility/dn.robot} | 14 +-
.../validate.robot => compatibility/om.robot} | 14 +-
.../validate.robot => compatibility/recon.robot} | 13 +-
.../validate.robot => compatibility/scm.robot} | 14 +-
hadoop-ozone/dist/src/main/smoketest/lib/os.robot | 4 +
.../dist/src/main/smoketest/ozone-lib/shell.robot | 4 +
hadoop-ozone/dist/src/shell/hdds/hadoop-config.cmd | 317 -----
hadoop-ozone/dist/src/shell/hdds/hadoop-config.sh | 165 ---
hadoop-ozone/dist/src/shell/hdds/workers.sh | 47 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 245 ++--
hadoop-ozone/dist/src/shell/ozone/ozone-config.sh | 100 +-
.../ozone-functions.sh} | 1397 ++++++++++----------
hadoop-ozone/dist/src/shell/ozone/start-ozone.sh | 84 +-
hadoop-ozone/dist/src/shell/ozone/stop-ozone.sh | 60 +-
.../shell/shellprofile.d/hadoop-ozone-manager.sh | 8 +-
.../dist/src/shell/shellprofile.d/hadoop-ozone.sh | 6 +-
hadoop-ozone/dist/src/test/shell/gc_opts.bats | 40 +-
.../shell/ozone-functions_test_helper.bash} | 29 +-
.../shell/ozone_set_var_for_compatibility.bats | 86 ++
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 356 +++--
.../hadoop/fs/ozone/TestOzoneFileSystemV1.java | 202 +--
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 4 +-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 2 +
.../ozone/client/CertificateClientTestImpl.java | 11 +
.../ozone/client/rpc/TestKeyInputStream.java | 15 -
.../client/rpc/TestOzoneRpcClientAbstract.java | 191 +--
.../ozone/om/TestOzoneManagerConfiguration.java | 6 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 222 +++-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 9 -
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 124 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 2 +-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 11 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 4 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 5 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 26 +-
.../request/bucket/acl/OMBucketAddAclRequest.java | 20 +-
.../bucket/acl/OMBucketRemoveAclRequest.java | 20 +-
.../request/bucket/acl/OMBucketSetAclRequest.java | 20 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 4 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 16 +-
.../om/request/key/acl/OMKeyAddAclRequest.java | 24 +-
.../om/request/key/acl/OMKeyRemoveAclRequest.java | 24 +-
.../om/request/key/acl/OMKeySetAclRequest.java | 24 +-
.../request/key/acl/prefix/OMPrefixAclRequest.java | 9 +-
.../key/acl/prefix/OMPrefixAddAclRequest.java | 13 +-
.../key/acl/prefix/OMPrefixRemoveAclRequest.java | 13 +-
.../key/acl/prefix/OMPrefixSetAclRequest.java | 13 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 6 +-
.../om/request/volume/acl/OMVolumeAclRequest.java | 20 +-
.../request/volume/acl/OMVolumeAddAclRequest.java | 19 +-
.../volume/acl/OMVolumeRemoveAclRequest.java | 19 +-
.../request/volume/acl/OMVolumeSetAclRequest.java | 20 +-
.../OzoneDelegationTokenSecretManager.java | 24 +-
.../ozone/om/TestOzoneManagerHttpServer.java | 2 +
.../security/TestOzoneBlockTokenSecretManager.java | 159 ++-
.../ozone/security/acl/TestRequestContext.java | 94 ++
.../recon/schema/ContainerSchemaDefinition.java | 19 -
.../hadoop/ozone/recon/ReconControllerModule.java | 19 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 25 +-
.../recon/api/types/MissingContainerMetadata.java | 2 +-
.../api/types/UnhealthyContainerMetadata.java | 2 +-
.../codec/ContainerReplicaHistoryListCodec.java | 86 ++
.../ozone/recon/fsck/ContainerHealthTask.java | 14 +-
...ager.java => ContainerHealthSchemaManager.java} | 51 +-
.../ozone/recon/persistence/ContainerHistory.java | 79 ++
.../ozone/recon/scm/ContainerReplicaHistory.java | 62 +
...nager.java => ContainerReplicaHistoryList.java} | 29 +-
.../ozone/recon/scm/ReconContainerManager.java | 215 ++-
.../scm/ReconStorageContainerManagerFacade.java | 20 +-
.../recon/spi/ContainerDBServiceProvider.java | 33 +
.../spi/impl/ContainerDBServiceProviderImpl.java | 91 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 4 +-
.../ozone/recon/spi/impl/ReconDBDefinition.java | 14 +-
.../ozone/recon/tasks/ContainerKeyMapperTask.java | 7 +-
.../ozone/recon/tasks/FileSizeCountTask.java | 6 +-
.../ozone/recon/tasks/OMUpdateEventBatch.java | 21 +-
.../hadoop/ozone/recon/tasks/ReconOmTask.java | 9 -
.../ozone/recon/tasks/ReconTaskControllerImpl.java | 8 +-
.../hadoop/ozone/recon/tasks/TableCountTask.java | 7 +-
.../ozone/recon/api/TestContainerEndpoint.java | 194 ++-
.../hadoop/ozone/recon/api/TestEndpoints.java | 5 +-
.../ozone/recon/fsck/TestContainerHealthTask.java | 10 +-
.../scm/AbstractReconContainerManagerTest.java | 6 +-
.../ozone/recon/scm/TestReconContainerManager.java | 102 +-
.../hadoop/ozone/recon/tasks/DummyReconDBTask.java | 1 -
.../ozone/recon/tasks/TestFileSizeCountTask.java | 8 +
.../recon/tasks/TestReconTaskControllerImpl.java | 7 -
.../apache/hadoop/ozone/debug/ChunkKeyHandler.java | 4 -
.../hadoop/ozone/freon/RandomKeyGenerator.java | 2 +-
.../org/apache/hadoop/ozone/shell/Handler.java | 4 +
.../hadoop/ozone/shell/SetSpaceQuotaOptions.java | 4 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 13 +-
.../hadoop/ozone/shell/bucket/SetQuotaHandler.java | 23 +-
.../ozone/shell/volume/CreateVolumeHandler.java | 13 +-
.../hadoop/ozone/shell/volume/SetQuotaHandler.java | 22 +-
pom.xml | 8 +-
193 files changed, 4305 insertions(+), 3195 deletions(-)
copy hadoop-hdds/{container-service =>
client}/src/test/resources/log4j.properties (100%)
delete mode 100644 hadoop-hdds/common/src/main/conf/hadoop-env.cmd
delete mode 100644 hadoop-hdds/common/src/main/conf/hadoop-env.sh
create mode 100644 hadoop-hdds/common/src/main/conf/ozone-env.sh
delete mode 100644
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientUtils.java
copy hadoop-ozone/dist/src/main/compose/{ozonescripts => compatibility}/.env
(100%)
copy hadoop-ozone/dist/src/main/compose/{ozone =>
compatibility}/docker-compose.yaml (92%)
copy hadoop-ozone/dist/src/main/compose/{ozone => compatibility}/docker-config
(88%)
copy hadoop-ozone/dist/src/main/compose/{failing1 => compatibility}/test.sh
(79%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha-s3 =>
ozonescripts}/test.sh (78%)
mode change 100644 => 100755
create mode 100644 hadoop-ozone/dist/src/main/smoketest/cli/classpath.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/cli/envvars.robot
copy hadoop-ozone/dist/src/main/smoketest/{freon/validate.robot =>
compatibility/dn.robot} (71%)
copy hadoop-ozone/dist/src/main/smoketest/{freon/validate.robot =>
compatibility/om.robot} (71%)
copy hadoop-ozone/dist/src/main/smoketest/{freon/validate.robot =>
compatibility/recon.robot} (74%)
copy hadoop-ozone/dist/src/main/smoketest/{freon/validate.robot =>
compatibility/scm.robot} (69%)
delete mode 100644 hadoop-ozone/dist/src/shell/hdds/hadoop-config.cmd
delete mode 100755 hadoop-ozone/dist/src/shell/hdds/hadoop-config.sh
rename hadoop-ozone/dist/src/shell/{hdds/hadoop-functions.sh =>
ozone/ozone-functions.sh} (58%)
copy hadoop-ozone/dist/src/{shell/shellprofile.d/hadoop-ozone-manager.sh =>
test/shell/ozone-functions_test_helper.bash} (65%)
mode change 100644 => 100755
create mode 100644
hadoop-ozone/dist/src/test/shell/ozone_set_var_for_compatibility.bats
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/acl/TestRequestContext.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ContainerReplicaHistoryListCodec.java
rename
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/{ContainerSchemaManager.java
=> ContainerHealthSchemaManager.java} (66%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/ContainerHistory.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ContainerReplicaHistory.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/{ReconSafeModeManager.java
=> ContainerReplicaHistoryList.java} (59%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
