steveloughran commented on code in PR #1242:
URL: https://github.com/apache/parquet-mr/pull/1242#discussion_r1537928482
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetRecordReader.java:
##########
@@ -155,8 +156,13 @@ private void initializeInternalReader(ParquetInputSplit
split, Configuration con
}
// open a reader with the metadata filter
- ParquetFileReader reader =
- ParquetFileReader.open(HadoopInputFile.fromPath(path, configuration),
optionsBuilder.build());
+ InputFile inputFile;
+ if (split.getFooter() != null && split.getFooter().getInputFile() != null)
{
+ inputFile = split.getFooter().getInputFile();
+ } else {
+ inputFile = HadoopInputFile.fromPath(path, configuration);
Review Comment:
if the filestatus (or at least file length) can get down here then it
becomes possible to skip a HEAD request when opening a file against cloud
storage. the api you need is in 3.3.0, and not very reflection friendly. we
could add something to assist there.
what is key is: get as much info as possible into HadoopInputFile,
especially expected length
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
