clesaec commented on code in PR #1807:
URL: https://github.com/apache/avro/pull/1807#discussion_r938651175
##########
lang/java/mapred/src/main/java/org/apache/avro/mapred/FsInput.java:
##########
@@ -41,7 +43,15 @@ public FsInput(Path path, Configuration conf) throws
IOException {
/** Construct given a path and a {@code FileSystem}. */
public FsInput(Path path, FileSystem fileSystem) throws IOException {
this.len = fileSystem.getFileStatus(path).getLen();
- this.stream = fileSystem.open(path);
+ // use the hadoop 3.3.0 openFile API and specify length
+ // and read policy. object stores can use these to
+ // optimize read performance.
+ // the read policy "adaptive" means "start sequential but
+ // go to random IO after backwards seeks"
+ // Filesystems which don't recognize the options will ignore them
+
+ this.stream =
awaitFuture(fileSystem.openFile(path).opt("fs.option.openfile.read.policy",
"adaptive")
+ .opt("fs.option.openfile.length", Long.toString(len)).build());
Review Comment:
ok, thanks for explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
