[
https://issues.apache.org/jira/browse/HDDS-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754900#comment-17754900
]
Kohei Sugihara commented on HDDS-8371:
--------------------------------------
Hi [~Sammi] [~kkasawa] , we've fixed the key corruption issue in the write path
in HDDS-8292. But we also need to fix the read path for the corrupted keys. In
our environment, we continuously see metadata corruptions in listKeys after
introducing HDDS-8292. Some other long-living environments might feel this
problem too.
The proposed PR is a tentative workaround for fixing the read path against
corrupted keys, and I think we need some approach for reading the corrupted
keys in the correct way even now. What do you think?
Has anyone worked fixing the read path, or are some patches already available?
> A keyName field in the keyTable might contain a full path for the key instead
> of the file name
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-8371
> URL: https://issues.apache.org/jira/browse/HDDS-8371
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.3.0
> Reporter: Kohei Sugihara
> Assignee: Ashish Kumar
> Priority: Major
> Labels: pull-request-available
>
> The listStatus API serves a repeated path in the list when a path for the key
> is deep. We noticed the listStatus API serves a corrupt result against some
> specific keys in a bucket. The corruption is that repeats a requested key
> prefix in a final list of the listStatus result like the following:
> {code:java}
> # expected case
> % aws s3 ls s3://bucket/a/b/c/d/e/f/g/file.zip
> <timestamp> <bytes> a/b/c/d/e/f/g/file.zip
> ...
> # actual: "a/b/c/d/e/f/g" is duplicated
> % aws s3 ls s3://bucket/a/b/c/d/e/f/g/file.zip
> <timestamp> <bytes> a/b/c/d/e/f/g/a/b/c/d/e/f/g/file.zip
> ... {code}
> Environment:
> * Ozone 1.3
> [compatible|https://github.com/apache/ozone/commit/9c61a8aa497ab96c014ad3bb7b1ee4f731ebfaf8]
> version (same environment as HDDS-7701, HDDS-7925)
> * Several large files, all of them are uploaded by multipart using AWS-CLI,
> divided into 8 MB chunks
> * An FSO-enabled bucket
> * OM HA
> Problem Details:
> I've dug the OM DB and found metadata in the keyTable has the full path for
> the key, so it finally appears redundant prefix twice in the result of the
> listStatus API.
> {code:java}
> # dump keyTable entries
> while (keyIter.hasNext()) {
> Table.KeyValue<String, OmKeyInfo> kv = keyIter.next();
> OmKeyInfo v = kv.getValue();
> LOG.info("v/b={}/{} parent={} Key={} size={} time={} checksum={} id={}
> keyName={}",
> kv.getValue().getVolumeName(), kv.getValue().getBucketName(),
> nodeId, kv.getValue().getFileName(),
> kv.getValue().getDataSize(), kv.getValue().getCreationTime(),
> kv.getValue().getFileChecksum(), kv.getValue().getObjectID(),
> kv.getValue().getKeyName());
> }
> # keyName has a full path for the key
> - v/b=volume/bucket parent=-9223371931475161596 Key=0g0pustv.tar.gz
> size=2276021708 time=1672052153340 checksum=null id=-9223371931457566207
> keyName=a/b/c/d/e/0g0pustv.tar.gz
> - v/b=volume/bucket parent=-9223371931475161596 Key=0g0pustv.zip
> size=2333733892 time=1672052222395 checksum=null id=-9223371931408929023
> keyName=a/b/c/d/e/0g0pustv.zip
> - v/b=volume/bucket parent=-9223371931475161596 Key=0nh5ww00.tar.gz
> size=249321741 time=1672052233487 checksum=null id=-9223371931393057791
> keyName=a/b/c/d/e/0nh5ww00.tar.gz
> - v/b=volume/bucket parent=-9223371931475161596 Key=0nh5ww00.zip
> size=255764877 time=1672052242830 checksum=null id=-9223371931388326655
> keyName=a/b/c/d/e/0nh5ww00.zip
> - v/b=volume/bucket parent=-9223371931475161596 Key=5b2uha1h.tar.gz
> size=2276346612 time=1672052331175 checksum=null id=-9223371931348859135
> keyName=a/b/c/d/e/5b2uha1h.tar.gz
> ...
> # other keys which have the same parent do not have their prefix in the key
> - v/b=volume/bucket parent=-9223371931475161596 Key=kh7vbwlh.zip
> size=573797127 time=1672052273970 checksum=null id=-9223371931375503871
> keyName=kh7vbwlh.zip
> - v/b=volume/bucket parent=-9223371931475161596 Key=ngaxsd8c.tar.gz
> size=380094900 time=1672052284433 checksum=null id=-9223371931368669695
> keyName=ngaxsd8c.tar.gz
> - v/b=volume/bucket parent=-9223371931475161596 Key=ngaxsd8c.zip
> size=393085953 time=1672052099618 checksum=null id=-9223371931473057023
> keyName=ngaxsd8c.zip
> - v/b=volume/bucket parent=-9223371931475161596 Key=nrou31c3.tar.gz
> size=568466718 time=1672052124043 checksum=null id=-9223371931461502975
> keyName=nrou31c3.tar.gz
> - v/b=volume/bucket parent=-9223371931475161596 Key=nrou31c3.zip
> size=574807485 time=1672052149947 checksum=null id=-9223371931446918911
> keyName=nrou31c3.zip
> - v/b=volume/bucket parent=-9223371931475161596 Key=ol8dhbqo.tar.gz
> size=555722830 time=1672052168904 checksum=null id=-9223371931435349759
> keyName=ol8dhbqo.tar.gz {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
