[
https://issues.apache.org/jira/browse/HDFS-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091685#comment-18091685
]
ASF GitHub Bot commented on HDFS-17593:
---------------------------------------
steveloughran commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3476392775
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,85 @@ public FSDataInputStream open(PathHandle fd, int
bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ try {
+ return dfs.createWrappedInputStream(dfsis);
+ } catch (IOException e) {
+ dfsis.close();
+ throw e;
+ }
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
Review Comment:
add one more check here, filenames. From the spec.
```
The final `status.getPath().getName()` element of the supplied status MUST
equal
the name value of the path supplied to the `openFile(path)` call.
Filesystems MUST NOT validate the rest of the path.
This is needed to support viewfs and other mount-point wrapper filesystems
where schemas and paths are different. These often create their own
FileStatus results
```
not quite so critical given you are already gating on class, but you don't
want to have the status of a different file passed in, do you?
> Allow setting block locations when opening streams
> --------------------------------------------------
>
> Key: HDFS-17593
> URL: https://issues.apache.org/jira/browse/HDFS-17593
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: pull-request-available
>
> The HDFS client seems to always get block locations from the namenode when
> opening a file:
> https://github.com/apache/hadoop/blob/4525c7e35ea22d7a6350b8af10eb8d2ff68376e7/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1099
> This leads to unnecessary RPCs in Apache Impala when doing reads, as the
> block locations are cached globally and the executors already have a good
> guess about the block locations when opening a stream. Unless the cached
> block locations are stale ideally no RPC should be made to the namenode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
