[
https://issues.apache.org/jira/browse/HADOOP-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HADOOP-16586:
------------------------------------
Description:
Most of these tests fail if running against a local metastore with a
ClassCastException.
Not sure if these tests are intended to work with dynamo only. The fix (either
ignore in case of other metastores or fix the test) would depend on the
original intent.
{code}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
-------------------------------------------------------------------------------
Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.237 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)
testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 1.827 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)
testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.819 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)
testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time
elapsed: 2.832 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)
testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.752 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)
testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 4.103 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)
testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.017 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)
testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.821 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)
testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time elapsed:
4.493 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)
testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.782 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)
testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.008 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)
{code}
ITestPartialRenamesDeletes
{code}
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
FileNotFound Path ...
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
FileNotFound Path ...
[ERROR]
ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
[ERROR]
ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
{code}
{code}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
-------------------------------------------------------------------------------
Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.947 s <<< ERROR!
java.io.FileNotFoundException: Expected file: not found
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
in
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.825 s <<< ERROR!
java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.938 s <<< ERROR!
java.io.FileNotFoundException: Expected file: not found
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
in
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.834 s <<< ERROR!
java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
{code}
I also see AccessDenied exceptions in the test logs. Have not debugged this
further.
ITestS3GuardToolLocal
{code}
[ERROR] Errors:
[ERROR]
ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137
» IllegalArgument
{code}
{code}
Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.577 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)
Time elapsed: 0.182 s <<< ERROR!
java.lang.IllegalArgumentException: bucket
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.testInitNegativeRead(ITestS3GuardToolLocal.java:176)
{code}
ITestS3GuardToolLocal ends up skipping all tests if running with -Ddynamo, or
without -Ds3guard - so these tests are exercised with the -Ds3guard option only.
was:
Most of these tests fail if running against a local metastore with a
ClassCastException.
Not sure if these tests are intended to work with dynamo only. The fix (either
ignore in case of other metastores or fix the test) would depend on the
original intent.
{code}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
-------------------------------------------------------------------------------
Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.237 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)
testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 1.827 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)
testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.819 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)
testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time
elapsed: 2.832 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)
testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.752 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)
testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 4.103 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)
testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.017 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)
testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.821 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)
testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time elapsed:
4.493 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)
testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 2.782 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)
testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
Time elapsed: 3.008 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)
{code}
ITestPartialRenamesDeletes
{code}
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
FileNotFound Path ...
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
FileNotFound Path ...
[ERROR]
ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
[ERROR]
ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
{code}
{code}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
-------------------------------------------------------------------------------
Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.947 s <<< ERROR!
java.io.FileNotFoundException: Expected file: not found
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
in
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.825 s <<< ERROR!
java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.938 s <<< ERROR!
java.io.FileNotFoundException: Expected file: not found
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
in
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
Caused by: java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
Time elapsed: 0.834 s <<< ERROR!
java.io.FileNotFoundException: Path
s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir
is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
{code}
I also see AccessDenied exceptions in the test logs. Have not debugged this
further.
ITestS3GuardToolLocal
{code}
[ERROR] Errors:
[ERROR]
ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137
» IllegalArgument
{code}
> ITestS3GuardFsck, others fails when run using a local metastore
> ---------------------------------------------------------------
>
> Key: HADOOP-16586
> URL: https://issues.apache.org/jira/browse/HADOOP-16586
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Reporter: Siddharth Seth
> Priority: Major
>
> Most of these tests fail if running against a local metastore with a
> ClassCastException.
> Not sure if these tests are intended to work with dynamo only. The fix
> (either ignore in case of other metastores or fix the test) would depend on
> the original intent.
> {code}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
> -------------------------------------------------------------------------------
> Tests run: 12, Failures: 0, Errors: 11, Skipped: 1, Time elapsed: 34.653 s
> <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck
> testIDetectParentTombstoned(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.237 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentTombstoned(ITestS3GuardFsck.java:190)
> testIDetectDirInS3FileInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 1.827 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectDirInS3FileInMs(ITestS3GuardFsck.java:214)
> testIDetectLengthMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.819 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectLengthMismatch(ITestS3GuardFsck.java:311)
> testIEtagMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time
> elapsed: 2.832 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIEtagMismatch(ITestS3GuardFsck.java:373)
> testIDetectFileInS3DirInMs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.752 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectFileInS3DirInMs(ITestS3GuardFsck.java:238)
> testIDetectModTimeMismatch(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 4.103 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectModTimeMismatch(ITestS3GuardFsck.java:346)
> testIDetectNoMetadataEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.017 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoMetadataEntry(ITestS3GuardFsck.java:113)
> testIDetectNoParentEntry(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.821 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectNoParentEntry(ITestS3GuardFsck.java:136)
> testINoEtag(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck) Time elapsed:
> 4.493 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testINoEtag(ITestS3GuardFsck.java:403)
> testIDetectParentIsAFile(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 2.782 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testIDetectParentIsAFile(ITestS3GuardFsck.java:163)
> testTombstonedInMsNotDeletedInS3(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck)
> Time elapsed: 3.008 s <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore cannot be cast to
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck.testTombstonedInMsNotDeletedInS3(ITestS3GuardFsck.java:432)
> {code}
> ITestPartialRenamesDeletes
> {code}
> [ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
> FileNotFound Path ...
> [ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:510 »
> FileNotFound Path ...
> [ERROR]
> ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
> » FileNotFound
> [ERROR]
> ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:411->AbstractFSContractTestBase.assertIsFile:316
> » FileNotFound
> {code}
> {code}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
> -------------------------------------------------------------------------------
> Tests run: 24, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 19.043 s <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
> testRenameParentPathNotWriteable[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.947 s <<< ERROR!
> java.io.FileNotFoundException: Expected file: not found
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
> in
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> Caused by: java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=false]-single-3272/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:20:53.835Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> testRenameDirFailsInDelete[bulk-delete=false](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.825 s <<< ERROR!
> java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=false]-single-5232/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:20:55.696Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
> testRenameParentPathNotWriteable[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.938 s <<< ERROR!
> java.io.FileNotFoundException: Expected file: not found
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
> in
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> Caused by: java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameParentPathNotWriteable[bulk-delete=true]-multi-1609/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:21:02.152Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameParentPathNotWriteable(ITestPartialRenamesDeletes.java:411)
> testRenameDirFailsInDelete[bulk-delete=true](org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes)
> Time elapsed: 0.834 s <<< ERROR!
> java.io.FileNotFoundException: Path
> s3a://<bucket>/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-3392/writableDir
> is recorded as deleted by S3Guard at 2019-09-24T07:21:03.843Z
> at
> org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.testRenameDirFailsInDelete(ITestPartialRenamesDeletes.java:510)
> {code}
> I also see AccessDenied exceptions in the test logs. Have not debugged this
> further.
> ITestS3GuardToolLocal
> {code}
> [ERROR] Errors:
> [ERROR]
> ITestS3GuardToolLocal.testInitNegativeRead:176->AbstractS3GuardToolTestBase.runToFailure:150->AbstractS3GuardToolTestBase.run:137
> » IllegalArgument
> {code}
> {code}
> Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.577 s <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)
> Time elapsed: 0.182 s <<< ERROR!
> java.lang.IllegalArgumentException: bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.testInitNegativeRead(ITestS3GuardToolLocal.java:176)
> {code}
> ITestS3GuardToolLocal ends up skipping all tests if running with -Ddynamo, or
> without -Ds3guard - so these tests are exercised with the -Ds3guard option
> only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
