[
https://issues.apache.org/jira/browse/HADOOP-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HADOOP-16586:
--------------------------------------
Status: Patch Available (was: Open)
> ITestS3GuardFsck, others fails when run using a local metastore
> ---------------------------------------------------------------
>
> Key: HADOOP-16586
> URL: https://issues.apache.org/jira/browse/HADOOP-16586
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Reporter: Siddharth Seth
> Assignee: Masatake Iwasaki
> Priority: Major
>
> Most of these tests fail if running against a local metastore with a
> ClassCastException.
> Not sure if these tests are intended to work with dynamo only. The fix
> (either ignore in case of other metastores or fix the test) would depend on
> the original intent.
> {code}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
> -------------------------------------------------------------------------------
> Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s
> <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
> testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.237 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)
> testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 1.827 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)
> testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.819 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)
> testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time
> elapsed: 2.832 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)
> testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.752 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)
> testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 4.103 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)
> testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.017 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)
> testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.821 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)
> testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time elapsed:
> 4.493 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)
> testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.782 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)
> testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.008 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)
> {code}
> ITestPartialRenamesDeletes
> {code}
> [ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
> FileNotFound Path ...
> [ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
> FileNotFound Path ...
> [ERROR]
> ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
> » FileNotFound
> [ERROR]
> ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
> » FileNotFound
> {code}
> {code}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
> -------------------------------------------------------------------------------
> Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
> testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.947 s <<< ERROR!
> java.io.FileNotFoundException: Expected file: not found
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
> in
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> Caused by: java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.825 s <<< ERROR!
> java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
> testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.938 s <<< ERROR!
> java.io.FileNotFoundException: Expected file: not found
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
> in
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> Caused by: java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.834 s <<< ERROR!
> java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
> {code}
> I also see AccessDenied exceptions in the test logs. Have not debugged this
> further.
> ITestS3GuardToolLocal
> {code}
> [ERROR] Errors:
> [ERROR]
> ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137
> » IllegalArgument
> {code}
> {code}
> Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.577 s <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)
> Time elapsed: 0.182 s <<< ERROR!
> java.lang.IllegalArgumentException: bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.testInitNegativeRead(ITestS3GuardToolLocal.java:176)
> {code}
> ITestS3GuardToolLocal ends up skipping all tests if running with -Ddynamo, or
> without -Ds3guard - so these tests are exercised with the -Ds3guard option
> only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
