[
https://issues.apache.org/jira/browse/HDDS-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928393#comment-17928393
]
Sammi Chen edited comment on HDDS-6611 at 2/19/25 11:17 AM:
------------------------------------------------------------
The container yaml file has this "checksum" field, which is the SHA-256 HASH of
the whole yaml file content. For a schema V2/V3 container, the container file
checksum check will fail if chunkPath and metadataPath are not included in HASH
calculation.
My first code version didn't introduce schema V4, for I also thought it maybe
too much to introduce a new schema version for this. But later, found that
it's hard to distinguish schema v3 container created before feature is
finalized, which has the chunkPath and metadataPath, from those schema V3
container created after feature is finalized, which doesn't have chunkPath and
metadataPath, for container yaml file doesn't have information about whether
the feature is finalized or not. It has "layOutVersion" which is "file per
block", and "schemaVersion". To workaround this for V3 container, first try
with no chunkPath and metadataPath yaml file loading, verify the checksum, if
verification fails, if load yaml file with checksum and metadataPath, do the
checksum calculation and verification again, if verification still fails, then
it's a final fail. if the verification succeeds, then the container file pass
the verification.
The code. The code is kind ugly and tricky, not easy to understand, maintain.
That's why later schema V4 is introduced.
was (Author: sammi):
The container yaml file has this "checksum" field, which is the SHA-256 HASH of
the whole yaml file content. For a schema V2/V3 container, the container file
checksum check will fail if chunkPath and metadataPath are not included in HASH
calculation.
My first code version doesn't introduce schema V4, for I also thought it maybe
too much to introduce a new schema version for this. But later, found that
it's hard to distinguish schema v3 container created before feature is
finalized, which has the chunkPath and metadataPath, from those schema V3
container created after feature is finalized, which doesn't have chunkPath and
metadataPath, for container yaml file doesn't have information about whether
the feature is finalized or not. It has "layOutVersion" which is "file per
block", and "schemaVersion". To workaround this for V3 container, first try
with no chunkPath and metadataPath yaml file loading, verify the checksum, if
verification fails, if load yaml file with checksum and metadataPath, do the
checksum calculation and verification again, if verification still fails, then
it's a final fail. if the verification succeeds, then the container file pass
the verification.
The code. The code is kind ugly and tricky, not easy to understand, maintain.
That's why later schema V4 is introduced.
> Remove chunksPath and metadataPath from container yaml file
> -----------------------------------------------------------
>
> Key: HDDS-6611
> URL: https://issues.apache.org/jira/browse/HDDS-6611
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Ethan Rose
> Assignee: Sammi Chen
> Priority: Major
> Labels: pull-request-available
>
> Currently the .container files of KeyValueContainers contain absolute paths
> to the chunks directory and metadata directory, which is used to find the
> container's contents after reading the container file. This means that if the
> mount points of volumes specified in hdds.datanode.dir configs change, the
> datanode will find the container files and try to load the containers, but
> report that the container is missing pieces because the container file's
> chunks and metadata directories no longer exist.
> This is somewhat inconsistent with other directory configurations, because it
> is the only one (as far as I know) where changing its value to a new location
> that has the same data will cause a failure.
> This task will remove the chunksPath and metadataPath from container yaml
> file, when container is moved around(replication, or mount point changes),
> container directory will still be self constructed, and there is no need to
> update the yaml file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
