steveloughran commented on a change in pull request #1761: HADOOP-16759.
Filesystem openFile() builder to take a FileStatus param
URL: https://github.com/apache/hadoop/pull/1761#discussion_r358215296
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -432,6 +433,42 @@ public void testReadFileChangedOutOfSyncMetadata() throws
Throwable {
}
}
+ /**
+ * Verifies that when the openFile builder is passed in a status,
+ * then that is used to eliminate the getFileStatus call in open();
+ * thus the version and etag passed down are still used.
+ */
+ @Test
+ public void testOpenFileWithStatus() throws Throwable {
+ final Path testpath = path("testOpenFileWithStatus.dat");
+ final byte[] dataset = TEST_DATA_BYTES;
+ S3AFileStatus originalStatus =
+ writeFile(testpath, dataset, dataset.length, true);
+
+ // forge a file status with a different tag
+ S3AFileStatus forgedStatus =
+ S3AFileStatus.fromFileStatus(originalStatus, Tristate.FALSE,
+ originalStatus.getETag() + "-fake",
+ originalStatus.getVersionId() + "");
+ fs.getMetadataStore().put(
+ new PathMetadata(forgedStatus, Tristate.FALSE, false));
+
+ // By passing in the status open() doesn't need to check s3guard
+ // And hence the existing file is opened
+ try (FSDataInputStream instream = fs.openFile(testpath)
+ .withFileStatus(originalStatus)
+ .build().get()) {
+ instream.read();
+ }
+
+ // and this holds for S3A Located Status
+ try (FSDataInputStream instream = fs.openFile(testpath)
+ .withFileStatus(new S3ALocatedFileStatus(originalStatus, null))
+ .build().get()) {
+ instream.read();
+ }
+ }
Review comment:
you mean with the file version changed? It shouldn't notice the version
issues without s3guard, will fai on read() after a number of retriesl when
s3guard version doesn't match that in the store
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
