[
https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870696#comment-15870696
]
Aaron Fabbri commented on HADOOP-14041:
---------------------------------------
Just recording results from my test runs last night:
mvn clean verify -Ds3guard -Ddynamo -Dscale
Tests run: 366, Failures: 3, Errors: 2, Skipped: 70
{noformat}
Failed tests:
(1)
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:222->Assert.assertTrue:41->Assert.fail:88
files mismatch: "s3a://fabbri-dev/user/fabbri/test/file"
"s3a://fabbri-dev/user/fabbri/test/parentdir/child"
(2)
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:95->Assert.fail:88
After 1 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://fabbri-dev/Users; isDirectory=true;
modification_time=0; access_time=0; owner=fabbri; group=fabbri;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
(3)
ITestS3AContractRootDir.testListEmptyRootDirectory:63->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:186->Assert.fail:88
Deleted file: unexpectedly found s3a://fabbri-dev/user as
S3AFileStatus{path=s3a://fabbri-dev/user; isDirectory=true;
modification_time=0; access_time=0; owner=fabbri; group=fabbri;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
Tests in error:
(4) ITestS3ACredentialsInURL.testInstantiateFromURL:86 » InterruptedIO
initTable: ...
(5) ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore:145 » IO S3Guard
tab...
{noformat}
1-3 are root directory test failures which have been flaky.. one is leftover
files from FileSystemContractBaseTest, the other two are something creating a
user/ directory while test is running?
4 is expected: s3guard will not use URI credentials. (We should skip this if
we don't already do that in pending patch)
5 is this: S3Guard table lacks version marker. Table:
destroyDynamoDBMetadataStore-1546206104
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.verifyVersionCompatibility(DynamoDBMetadataStore.java:667)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:630)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:288)
I don't think any of these are related, except maybe the last one?
As for testing the prune command itself, the first thing I notice is that it
behaves a bit differently than, say, diff. Diff appears to use bucket name as
table name if one is not set, but prune requires setting the table name.
{noformat}
$ hadoop s3a prune -H 1 s3a://fabbri-bucket
No DynamoDB table name configured!
{noformat}
> CLI command to prune old metadata
> ---------------------------------
>
> Key: HADOOP-14041
> URL: https://issues.apache.org/jira/browse/HADOOP-14041
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
> Attachments: HADOOP-14041-HADOOP-13345.001.patch,
> HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch,
> HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch
>
>
> Add a CLI command that allows users to specify an age at which to prune
> metadata that hasn't been modified for an extended period of time. Since the
> primary use-case targeted at the moment is list consistency, it would make
> sense (especially when authoritative=false) to prune metadata that is
> expected to have become consistent a long time ago.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
