[
https://issues.apache.org/jira/browse/PARQUET-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated PARQUET-2493:
------------------------------------
Description:
In the current version of the HadoopInputFile implementation:
[https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java]
When performing a newStream, the reference to the FileStatus is lost, which has
already been previously consulted to create this class. This means that when
you go to the implementation of each FileSystem, it will surely have to be
requested again, since you have requested the reference of whether the file
exists, when the file is weighed or relevant information to be able to open the
file.
Hadoop's openFile() builder API does support this, but it is not on older
releases, so until Parquet moves to Hadoop 3.2.0+ only it cannot use the API.
And because its a complex and extensible design, it's very hard to use
reflection.
HADOOP-19131 adds reflection-friendly entry points for this and other
operations, so for releases with the new class, Parquet can pick up the speedup.
was:
In the current version of the HadoopInputFile implementation:
[https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java]
When performing a newStream, the reference to the FileStatus is lost, which has
already been previously consulted to create this class. This means that when
you go to the implementation of each FileSystem, it will surely have to be
requested again, since you have requested the reference of whether the file
exists, when the file is weighed or relevant information to be able to open the
file.
To do this, it is necessary to change the Hadoop implementation, including that
in the open function, the Path object does not necessarily have to be passed
but the FileStatus can be passed directly.
return HadoopStreams.wrap(fs.open(stat.getPath()));
to
return HadoopStreams.wrap(fs.open(stat));
The implementation change is proposed in:
https://issues.apache.org/jira/browse/HADOOP-19199
> HadoopInputFile to pass down FileStatus when opening file.
> ----------------------------------------------------------
>
> Key: PARQUET-2493
> URL: https://issues.apache.org/jira/browse/PARQUET-2493
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-hadoop
> Affects Versions: 1.14.0
> Reporter: Oliver Caballero Alvarez
> Priority: Major
>
> In the current version of the HadoopInputFile implementation:
> [https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java]
>
> When performing a newStream, the reference to the FileStatus is lost, which
> has already been previously consulted to create this class. This means that
> when you go to the implementation of each FileSystem, it will surely have to
> be requested again, since you have requested the reference of whether the
> file exists, when the file is weighed or relevant information to be able to
> open the file.
> Hadoop's openFile() builder API does support this, but it is not on older
> releases, so until Parquet moves to Hadoop 3.2.0+ only it cannot use the API.
> And because its a complex and extensible design, it's very hard to use
> reflection.
> HADOOP-19131 adds reflection-friendly entry points for this and other
> operations, so for releases with the new class, Parquet can pick up the
> speedup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
