[
https://issues.apache.org/jira/browse/HDDS-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh resolved HDDS-16320.
---------------------------------------
Fix Version/s: 2.2.2
Resolution: Fixed
> Recon incorrectly marks healthy EC containers as REPLICA_MISMATCH by
> comparing checksums across different EC replica indexes
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16320
> URL: https://issues.apache.org/jira/browse/HDDS-16320
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Affects Versions: 2.2.1
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.2.2
>
>
> Recon incorrectly classifies healthy erasure-coded containers as
> REPLICA_MISMATCH.
> For an EC container, each replica index contains a different data or parity
> fragment. Consequently, checksums belonging to different EC replica indexes
> are not expected to be identical.
> Recon currently compares dataChecksum across all replicas of a container
> without considering the EC replica index. Therefore, a healthy EC container
> with all required indexes can be reported as REPLICA_MISMATCH.
> *### Steps to reproduce*
> 1. Start an Ozone docker cluster with at least five healthy datanodes and
> Recon.
> 2. Create a volume and an EC bucket:
>
> {code:java}
> ozone sh volume create /recon-ec-vol
> ozone sh bucket create /recon-ec-vol/recon-ec-bucket \
> --type EC \
> --replication rs-3-2-1024k
> {code}
> 3. Write a key:
>
> {code:java}
> ozone freon ockg \
> --volume recon-ec-vol \
> --bucket recon-ec-bucket \
> --number-of-tests 1 \
> --size 1MB \
> --threads 1 \
> --prefix reconec \
> --type EC \
> --replication rs-3-2-1024k
> {code}
> 4. Obtain the container ID:
>
> {code:java}
> KEY_PATH=/recon-ec-vol/recon-ec-bucket/reconec/0
> CID=$(ozone sh key info "$KEY_PATH" |
> jq -er '.ozoneKeyLocations[0].containerID')
> echo "$CID"
> {code}
> 5. Close the container so that its checksum information is available:
>
> {code:java}
> ozone admin container close "$CID"
> {code}
> 6. Wait until the container has five replicas, and inspect their indexes
> and checksums:
>
> {code:java}
> ozone admin container info "$CID" --json |
> jq -r '
> .replicas[] |
> [
> .replicaIndex,
> .datanodeDetails.hostName,
> .state,
> .dataChecksum
> ] | @tsv'
> {code}
> Verify that:
> - Replica indexes 1-5 are present.
> - The indexes are hosted on five distinct datanodes.
> - The replicas are healthy.
> - Different indexes have different dataChecksum values.
> 7. Verify that the key remains readable:
>
> {code:java}
> ozone sh key get --force "$KEY_PATH" /tmp/recon-ec-download
> test "$(stat -c %s /tmp/recon-ec-download)" -eq 1048576
> {code}
> 8. Wait for the Recon container-health scan and query REPLICA_MISMATCH:
> {code:java}
> curl -s \
> 'http://localhost:9888/api/v1/containers/unhealthy/REPLICA_MISMATCH' |
> jq --argjson cid "$CID" \
> '[.. | objects | select(.containerID? == $cid)]'
> {code}
> * ### Actual result*
> Recon reports the healthy EC container as:
> REPLICA_MISMATCH
> Data checksum mismatch across replicas
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
