steveloughran commented on a change in pull request #1815: HADOOP-16801:
S3Guard-listFiles will not query s3 if all listings are…
URL: https://github.com/apache/hadoop/pull/1815#discussion_r370306157
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java
##########
@@ -291,6 +295,102 @@ public void testListStatusMakesEmptyDirAuth() throws
Throwable {
assertListDoesNotUpdateAuth(dir);
}
+ @Test
+ public void testListFilesRecursiveWhenAllListingsAreAuthoritative()
+ throws Exception {
+ describe("listFiles does not make further calls to the fs when"
+ + "all nested directory listings are authoritative");
+ Set<Path> files = new HashSet<>();
+
+ Path parentDir = dir;
+ Path parentFile = dirFile;
+ Path nestedDir1 = new Path(dir, "nested1");
+ Path nestedFile1 = new Path(nestedDir1, "nestedFile1");
+ Path nestedDir2 = new Path(nestedDir1, "nested2/");
+ Path nestedFile2 = new Path(nestedDir2, "nestedFile2");
+
+ files.add(parentFile);
+ files.add(nestedFile1);
+ files.add(nestedFile2);
+
+ authFS.mkdirs(parentDir);
+ authFS.mkdirs(nestedDir1);
+ authFS.mkdirs(nestedDir2);
+ touchFile(parentFile);
+ touchFile(nestedFile1);
+ touchFile(nestedFile2);
+
+ // making listStatus call to mark directories authoritative
+ authFS.listStatus(parentDir);
+ authFS.listStatus(nestedDir1);
+ authFS.listStatus(nestedDir2);
+
+ S3AStorageStatistics statistics = authFS.getStorageStatistics();
+ statistics.reset();
+
+ RemoteIterator<LocatedFileStatus> statusIterator =
+ authFS.listFiles(dir, true);
+
+ while (statusIterator.hasNext()) {
+ LocatedFileStatus locatedFileStatus = statusIterator.next();
+ assertTrue("This path does not exist in original listing: " +
Review comment:
This test suite has made the leap to AssertJ assertions -please us them
unless you can make the case against them. They're better, really
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
