alamb commented on code in PR #8714:
URL: https://github.com/apache/arrow-rs/pull/8714#discussion_r2467030949
##########
parquet/src/file/metadata/thrift/mod.rs:
##########
@@ -669,9 +696,46 @@ fn read_row_group(
Ok(row_group)
}
+/// Extract the metadata index from the footer bytes. `buf` should contain the
entire footer.
+pub(crate) fn get_metadata_index(buf: &[u8]) -> Result<Option<MetaIndex>> {
+ // TODO(ets): need constants to get rid of magic numbers
+ if buf.len() < 13 {
+ return Ok(None);
+ }
+ // check the last 4 bytes to see if we have the full footer or not
+ let magic = &buf[buf.len() - 4..];
+ let buf = if magic == "PAR1".as_bytes() {
+ &buf[0..buf.len() - 8]
+ } else {
+ buf
+ };
+
+ // check for PARI followed by 0.
+ if buf[buf.len() - 1] != 0 {
+ return Ok(None);
+ }
+ let magic = &buf[buf.len() - 5..buf.len() - 1];
+ if magic != "PARI".as_bytes() {
Review Comment:
Seems like everyone is racing to grab field 32767 lol
> I'm modifying the current reader to search backwards some for the "PARI",
and if found then parse that field first to get the index. That part seems to
be working 😄
Another approach would be to place it right before the current metadata --
since you know where the current footer metadata starts, you could check for
`PARI` at an offset you know after you read the last 8 bytes 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
