[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798844#comment-15798844
]
Mingliang Liu commented on HADOOP-13934:
----------------------------------------
{{FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}} fails
occasionally with the following errors:
{code}
$ mvn -Dtest=none -Dit.test='ITestS3AFileSystemContract' -q -Dscale clean verify
Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 74.348 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
Time elapsed: 0.556 sec <<< FAILURE!
junit.framework.AssertionFailedError: Rename result expected:<true> but
was:<false>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:174)
at junit.framework.TestCase.assertEquals(TestCase.java:333)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
{code}
It seems the working directories in {{FileSystemContractBaseTest}} is shared
across different test cases. Then I applied the following patch to use unique
test directory for test {{testRenameToDirWithSamePrefixAllowed}}.
{code:title=hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java}
@@ -663,9 +663,9 @@ public void testRenameChildDirForbidden() throws Exception {
*/
public void testRenameToDirWithSamePrefixAllowed() throws Throwable {
if (!renameSupported()) return;
- Path parentdir = path("test/parentdir");
+ Path parentdir = path("testRenameToDirWithSamePrefixAllowed/parentdir");
fs.mkdirs(parentdir);
- Path dest = path("test/parentdirdest");
+ Path dest = path("testRenameToDirWithSamePrefixAllowed/parentdirdest");
rename(parentdir, dest, true, false, true);
}
{code}
After that, the test
{{FileSystemContractBaseTest#testRenameToDirWithSamePrefixAllowed}} is failing
occasionally with different error. The reason was that the parameters created
in {{S3AFileSystem#innerRename}} passing to {{DynamoDBMetadataStore#move()}}
are lists that may contain duplicate paths (and thus duplicate DDB primary
key). Per offline discussion with [[email protected]] and [~rajesh.balamohan],
the v4 patch de-duplicates items of the same primary key in
DynamoDBMetadataStore side, but it is using auxiliary hashset so memory usage
is higher.
{code}
$ mvn -Dtest=none -Dit.test='ITestS3AFileSystemContract' -q -Dscale clean verify
Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Tests run: 43, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 70.52 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
Time elapsed: 1.347 sec <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: move:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list
of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID:
3MED8L2MSBF0SH8S4U5OK0TGQ3VV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item
keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error
Code: ValidationException; Request ID:
3MED8L2MSBF0SH8S4U5OK0TGQ3VV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:363)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
{code}
This problem is not related to the scope of this patch though. We can address
the problem in {{S3AFileSystem#innerRename}} or in test separately instead. We
can stick with the v3 patch I think.
> S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to
> BatchWriteItem limits
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Rajesh Balamohan
> Assignee: Mingliang Liu
> Priority: Minor
> Attachments: HADOOP-13934-HADOOP-13345.000.patch,
> HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch,
> HADOOP-13934-HADOOP-13345.003.patch, HADOOP-13934-HADOOP-13345.004.patch
>
>
> When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it
> started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with
> the following exception, it is relatively hard to debug on the real issue in
> DynamoDB side.
> {noformat}
> Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1
> validation error detected: Value
> '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
> ...
> ...
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
> at
> com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
