etseidl commented on code in PR #8714:
URL: https://github.com/apache/arrow-rs/pull/8714#discussion_r2471056853
##########
parquet/src/file/metadata/thrift/mod.rs:
##########
@@ -669,9 +696,46 @@ fn read_row_group(
Ok(row_group)
}
+/// Extract the metadata index from the footer bytes. `buf` should contain the
entire footer.
+pub(crate) fn get_metadata_index(buf: &[u8]) -> Result<Option<MetaIndex>> {
+ // TODO(ets): need constants to get rid of magic numbers
+ if buf.len() < 13 {
+ return Ok(None);
+ }
+ // check the last 4 bytes to see if we have the full footer or not
+ let magic = &buf[buf.len() - 4..];
+ let buf = if magic == "PAR1".as_bytes() {
+ &buf[0..buf.len() - 8]
+ } else {
+ buf
+ };
+
+ // check for PARI followed by 0.
+ if buf[buf.len() - 1] != 0 {
+ return Ok(None);
+ }
+ let magic = &buf[buf.len() - 5..buf.len() - 1];
+ if magic != "PARI".as_bytes() {
Review Comment:
> So the argument goes something like even if you have optimistically
fetched a bunch of bytes in the hopes of reading the entire footer in the first
read, you could not guarantee that a second fetch would get the index too (you
would have to do a second optimistic fetch or something)
Yes. Worst case would be two fetches for the file metadata (one for the 8
byte footer, a second for the thrift encoded file meta), followed by two for
the index. If you get really lucky you might get everything in a single fetch,
less lucky you'd most likely do two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
