tustvold commented on code in PR #4892:
URL: https://github.com/apache/arrow-rs/pull/4892#discussion_r1346383547
##########
parquet/src/arrow/async_reader/metadata.rs:
##########
@@ -95,16 +94,14 @@ impl<F: MetadataFetch> MetadataLoader<F> {
// Did not fetch the entire file metadata in the initial read, need to
make a second request
let (metadata, remainder) = if length > suffix_len - 8 {
let metadata_start = file_size - length - 8;
- let remaining_metadata =
fetch.fetch(metadata_start..footer_start).await?;
-
- let reader =
remaining_metadata.as_ref().chain(&suffix[..suffix_len - 8]);
- (read_metadata(reader)?, None)
+ let meta = fetch.fetch(metadata_start..file_size - 8).await?;
Review Comment:
This does mean we re-read data, however, in almost all cases this will be
irrelevant in the grand scheme of things.
Another option would be to concatenate the data here, but this might be less
efficient, e.g. the underlying AsyncFileReader is just slicing a Bytes from
memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
