Wan Kun created PARQUET-2415:
--------------------------------
Summary: Reuse hadoop file status and footer in ParquetRecordReader
Key: PARQUET-2415
URL: https://issues.apache.org/jira/browse/PARQUET-2415
Project: Parquet
Issue Type: Improvement
Components: parquet-hadoop
Affects Versions: 1.14.0
Reporter: Wan Kun
h2. DESCRIPTION
Spark will send a listStatus RPC to get hadoop file status and read the parquet
file footer before reading the parquet file. And send a same listStatus RPC to
get the same hadoop file status and read the footer again in
ParquetRecordReader. We can reuse the file status and the footer.
h2. PLANS
Save the hadoop file status in the ParquetMetadata and save the ParquetMetadata
in the input split, so we can reuse them when init a new ParquetRecordReader.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
