[
https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055269#comment-16055269
]
Aaron Fabbri commented on HADOOP-14548:
---------------------------------------
I need to re-test this stuff on trunk to isolate any differences vs. the
feature branch. For now collecting details on HADOOP-13345 branch:
#1 I did not reproduce the second time.
#2 failed because root was non-empty: s3a://bucket/Users existed.
#3 failed because it expected to only find a single path "testListStatus" in
the listing, but it got much more:
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testInputStreamClosedTwice
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testListStatus
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testMkdirsFailsForSubdirectoryOfExistingFile
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testMoveFileUnderParent
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testMultiByteFilesAreFiles
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testOverWriteAndRead
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testRenameDirectoryAsExistingFile
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testRenameDirectoryAsExistingFileNew
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testRenameDirectoryMoveToExistingDirectory
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testRenameDirectoryMoveToExistingDirectoryNew
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/testRenameFileMoveToNonExistentDirectory
s3n://fabbri-dev/user/fabbri/FileSystemContractBaseTest/writeReadAndDelete
#4 failure, looks like someone deleted file (1) from underneath us, or (2)
without using S3Guard (so a path was listed in MetadataStore still but we
failed to stat it on S3).
Tests run: 9, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 224.52 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)
Time elapsed: 53.834 sec <<< ERROR!
java.io.FileNotFoundException: No such file or directory:
s3a://fabbri-dev/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/target/test-dir/3/kkrpL8TPXx/test
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1939)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1834)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1777)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1557)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1533)
at
org.apache.hadoop.fs.contract.ContractTestUtils.treeWalk(ContractTestUtils.java:1168)
...
#6 failure looks like this:
testLSRootDir(org.apache.hadoop.fs.s3native.ITestJets3tNativeS3FileSystemContract)
Time elapsed: 24.063 sec <<< ERROR!
java.io.FileNotFoundException: File
s3n://fabbri-dev/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/target/test/data/2OvFym0hEP
does not exist.
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:603)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1824)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1866)
at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:2028)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2027)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2010)
at
org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2168)
at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2145)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.assertListFilesFinds(FileSystemContractBaseTest.java:869)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testLSRootDir(FileSystemContractBaseTest.java:851)
> S3Guard: issues running parallel tests w/ S3N
> ----------------------------------------------
>
> Key: HADOOP-14548
> URL: https://issues.apache.org/jira/browse/HADOOP-14548
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
>
> In general, running S3Guard and parallel tests with S3A and S3N contract
> tests enabled is asking for trouble: S3Guard code assumes there are not
> other non-S3Guard clients modifying the bucket.
> Goal of this JIRA is to:
> - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard
> -Ddynamo` with S3A and S3N contract tests configured.
> - Identify any failures here that are worth looking into.
> - Document (or enforce) that people should not do this, or should expect
> failures if they do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
