[
https://issues.apache.org/jira/browse/HADOOP-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925374#comment-15925374
]
Mingliang Liu commented on HADOOP-14036:
----------------------------------------
For empty directories, they will be added twice, while for non-empty
directories the problem does not show up to me. So in test if we create one
file under the source directory, we can pass the test. I tried multiple times.
try this if it works.
{code:title=FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}
createFile(new Path(parentdir, "child"));
{code}
I had a patch that uses Set instead of Collection as the type of {{srcPaths}}
and {{dstMetas}}, which I don't like in the sense that it doesn't care why/when
the items are added twice. Unless we're sure why the return object summaries
contain parent path, we should not blindly ignore the parent path in the code.
> S3Guard: intermittent duplicate item keys failure
> -------------------------------------------------
>
> Key: HADOOP-14036
> URL: https://issues.apache.org/jira/browse/HADOOP-14036
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Mingliang Liu
> Attachments: HADOOP-14036-HADOOP-13345.000.patch
>
>
> I see this occasionally when running integration tests with -Ds3guard
> -Ddynamo:
> {noformat}
> testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
> Time elapsed: 2.756 sec <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSServiceIOException: move:
> com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided
> list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status
> Code: 400; Error Code: ValidationException; Request ID:
> QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item
> keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error
> Code: ValidationException; Request ID:
> QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
