etseidl commented on issue #10129: URL: https://github.com/apache/arrow-rs/issues/10129#issuecomment-4692473296
Thanks for raising this issue. This isn't a bug, but the result of a deliberate decision to make use of the row group `ordinal` field internally (see #8715). Changing this behavior would require breaking changes to the `ParquetMetaData`, and would also require special handling on write (don't emit `ordinal` if `row_groups.len() > 32767`). I'm not saying we can't do this, just that it will require discussion of the ramifications. It's worth noting that `ordinal` was originally added to the spec to aid with encryption support (it allows for decoding row groups in parallel), but the encryption in this crate does not appear to use the field, opting instead for a `usize` row group index. cc @alamb @adamreeve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
