[GitHub] [hadoop] mukund-thakur commented on pull request #2038: HADOOP-17022 Tune S3AFileSystem.listFiles() api.
mukund-thakur commented on pull request #2038: URL: https://github.com/apache/hadoop/pull/2038#issuecomment-650206234 Fixed the raw test failures. Also ran the scale tests. All went fine apart from some timeout failures. I think this is almost done but I would like others @bgaborg and @mehakmeet as well to run tests on their configs to achieve good confidence in merging this. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2038: HADOOP-17022 Tune S3AFileSystem.listFiles() api.
mukund-thakur commented on pull request #2038: URL: https://github.com/apache/hadoop/pull/2038#issuecomment-648702503 Latest commit fixes the above test failures. After multiple debugging efforts I thought I should run the tests without s3guard as well. So when I did that I see 8 tests failing all with file not found exception. I will debug these further. `[ERROR] Errors: [ERROR] ITestS3AContractGetFileStatus>AbstractContractGetFileStatusTest.testListFilesEmptyDirectoryNonrecursive:99->AbstractContractGetFileStatusTest.listFilesOnEmptyDir:119 » FileNotFound [ERROR] ITestS3AContractGetFileStatus>AbstractContractGetFileStatusTest.testListFilesEmptyDirectoryRecursive:104->AbstractContractGetFileStatusTest.listFilesOnEmptyDir:119 » FileNotFound [ERROR] ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListFilesEmptyDirectoryNonrecursive:99->AbstractContractGetFileStatusTest.listFilesOnEmptyDir:119 » FileNotFound [ERROR] ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListFilesEmptyDirectoryRecursive:104->AbstractContractGetFileStatusTest.listFilesOnEmptyDir:119 » FileNotFound [ERROR] ITestMagicCommitProtocol>AbstractITCommitProtocol.testCommitJobButNotTask:1004->AbstractITCommitProtocol.executeWork:552->AbstractITCommitProtocol.executeWork:568->AbstractITCommitProtocol.lambda$testCommitJobButNotTask$9:1010 » FileNotFound [INFO] [ERROR] Tests run: 1204, Failures: 0, Errors: 5, Skipped: 342` `[ERROR] Errors: [ERROR] ITestS3AContractRootDir.testListEmptyRootDirectory:82->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:199 » FileNotFound [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testSimpleRootListing:239 » FileNotFound [ERROR] ITestS3AEncryptionSSEC.testListEncryptedDir:197 » FileNotFound No such file or... [INFO] [ERROR] Tests run: 110, Failures: 0, Errors: 3, Skipped: 87 ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2038: HADOOP-17022 Tune S3AFileSystem.listFiles() api.
mukund-thakur commented on pull request #2038:
URL: https://github.com/apache/hadoop/pull/2038#issuecomment-642775318
Full stack traces for test failures:
`java.lang.AssertionError: Recently renamed dir should not be visible:
[s3a://mthakur-data/test/a/b/dir3-DELAY_LISTING_ME, s3a://mthakur-data/test/a]
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentRenameAfterDelete(ITestS3GuardListConsistency.java:286)`
`java.lang.AssertionError: Expected a java.io.FileNotFoundException to be
thrown, but got the result: : "MagicCommitter{}"
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at
org.apache.hadoop.fs.s3a.commit.AbstractITCommitProtocol.expectFNFEonTaskCommit(AbstractITCommitProtocol.java:876)
at
org.apache.hadoop.fs.s3a.commit.AbstractITCommitProtocol.testCommitterWithDuplicatedCommit(AbstractITCommitProtocol.java:806)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2038: HADOOP-17022 Tune S3AFileSystem.listFiles() api.
mukund-thakur commented on pull request #2038:
URL: https://github.com/apache/hadoop/pull/2038#issuecomment-642125722
The reason for failure of
`ITestMagicCommitProtocol>AbstractITCommitProtocol.testCommitterWithDuplicatedCommit:806->AbstractITCommitProtocol.expectFNFEonTaskCommit:876
Expected a java.io.FileNotFoundException to be thrown, but got the result: :
"MagicCommitter{}"`
seem similar to the other test failure of
ITestS3GuardListConsistency.testConsistentRenameAfterDelete.
In this one, once the task is already committed the listing during next
commit still list the older file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
