steveloughran commented on code in PR #1807:
URL: https://github.com/apache/avro/pull/1807#discussion_r938627811
##########
lang/java/mapred/src/main/java/org/apache/avro/mapred/FsInput.java:
##########
@@ -41,7 +43,15 @@ public FsInput(Path path, Configuration conf) throws
IOException {
/** Construct given a path and a {@code FileSystem}. */
public FsInput(Path path, FileSystem fileSystem) throws IOException {
this.len = fileSystem.getFileStatus(path).getLen();
- this.stream = fileSystem.open(path);
+ // use the hadoop 3.3.0 openFile API and specify length
+ // and read policy. object stores can use these to
+ // optimize read performance.
+ // the read policy "adaptive" means "start sequential but
+ // go to random IO after backwards seeks"
+ // Filesystems which don't recognize the options will ignore them
+
+ this.stream =
awaitFuture(fileSystem.openFile(path).opt("fs.option.openfile.read.policy",
"adaptive")
+ .opt("fs.option.openfile.length", Long.toString(len)).build());
Review Comment:
we only added those options explicitly in branch-3.3; the code wouldn't
compile/link with 3.3.0. hence the strings. Unfortunately it also means you
don't get that speedup until 3.3.5 ships ....but the release will be ready.
3.3.0 did add the `withFileStatus(FileStatus)` param which was my original
design for passing in all filestatus info, inc etag and maybe version, so you
can go straight from listing to opening.
first in s3a, added abfs in 3.3.5. but it is too brittle because the path
checking requires status.getPath to equal the path opened. and hive with its
wrapper fs doesn't always do that.
Passing in file length is guaranteed to be ignored or actually used...no
brittleness. it also suits hive where workers know the length of the file but
don't have a status.
One thing i can add with immediate benefit in 3.3.0 is the initial the
fs.s3a.experimental.fadvise option, which again can mandate be adaptive, even
on hive clusters where they explicitly set read policy to be random (which some
do for max orc/parquet performance). The new opt fs.option.openfile.read.policy
is an evolution of that (you can now specify a list of policies and the first
one recognised is understood. if someone ever implemented explicit "parquet",
"orc", and "avro" for example), you could say the read policy is "orc, random,
adaptive" and get the first one known.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
