[
https://issues.apache.org/jira/browse/HIVE-24225?focusedWorklogId=497198&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497198
]
ASF GitHub Bot logged work on HIVE-24225:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Oct/20 09:24
Start Date: 08/Oct/20 09:24
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #1547:
URL: https://github.com/apache/hive/pull/1547#issuecomment-705446922
you actually want to use the openFile() API Call which is already in
hadoop-3.3, but which is being tweaked
(https://github.com/apache/hadoop/pull/2168) to have a standard option for seek
policy for all object stores, and the ability to pass in file length -which
will let the S3A & ABFS connectors skip a head probe
```java
@Override
public SeekableInputStream newStream() throws IOException {
FutureDataInputStreamBuilder builder = fs.openFile(getPath())
.opt("fs.s3a.experimental.input.fadvise", "random") // supported in
Hadoop 3.3.0
.opt("fs.option.openfile.fadvise", "random"); // this will be the
new standard one
if (length > 0) {
builder.opt("fs.option.openfile.length", length); // also in the
PR. Pass in the end of the split & s3a will be happy too
}
CompletableFuture<FSDataInputStream> streamF = builder.build();
return HadoopStreams.wrap(FutureIOSupport.awaitFuture(streamF));
}
```
That API is there today, if you want to try playing with it,.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 497198)
Time Spent: 1h 10m (was: 1h)
> FIX S3A recordReader policy selection
> -------------------------------------
>
> Key: HIVE-24225
> URL: https://issues.apache.org/jira/browse/HIVE-24225
> Project: Hive
> Issue Type: Bug
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Dynamic S3A recordReader policy selection can cause issues on lazy
> initialized FS objects
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
