[
https://issues.apache.org/jira/browse/HADOOP-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14036:
-----------------------------------
Attachment: HADOOP-14036-HADOOP-13345.002.patch
Discussed with [[email protected]] offline.
# For non-empty directories, we remove all the fake directory object as they're
unnecessary. See {{S3AFileSystem::finishedWrite}}. That's said, if we create
one child in the test code as following, the fake directory parent object will
be deleted. That's a smart optimization.
{code:title=FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}
createFile(new Path(parentdir, "child"));
{code}
# However, the empty directories, we have to keep the fake directory object (of
0 length, with "/" ending name etc). If we listObject, we'll see this object as
the only content of object summaries.
Overall, the v1 patch is good IMO. Steve also suggested we add some logging
message for easy diagnosis. Plus, I also added new integration tests in the
base test class: non-empty directory and empty directory cases are both covered
now.
[[email protected]] Correct me if I understand you wrongly.
> S3Guard: intermittent duplicate item keys failure
> -------------------------------------------------
>
> Key: HADOOP-14036
> URL: https://issues.apache.org/jira/browse/HADOOP-14036
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Mingliang Liu
> Attachments: HADOOP-14036-HADOOP-13345.000.patch,
> HADOOP-14036-HADOOP-13345.001.patch, HADOOP-14036-HADOOP-13345.002.patch
>
>
> I see this occasionally when running integration tests with -Ds3guard
> -Ddynamo:
> {noformat}
> testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
> Time elapsed: 2.756 sec <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSServiceIOException: move:
> com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided
> list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status
> Code: 400; Error Code: ValidationException; Request ID:
> QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item
> keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error
> Code: ValidationException; Request ID:
> QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
