zhangfengcdt opened a new issue, #3480: URL: https://github.com/apache/iceberg-rust/issues/3480
### Is your feature request related to a problem or challenge? For a Parquet file with no field ids and no name mapping, [`get_arrow_projection_mask_fallback`](https://github.com/apache/iceberg-rust/blob/f83bec42298a1cf51a42ea2aee26c041a15db5b4/crates/iceberg/src/arrow/reader/projection.rs#L243) projects each requested top-level column whole, so the reader decodes every leaf under it. After #3437 (fix for #3436), fields inside a struct of such a file have no id and read as null, as in iceberg-java. Their leaves are still decoded and then discarded by `RecordBatchTransformer`. For wide or deeply nested struct columns this is wasted I/O and decoding. Raised in review of #3437. ### Describe the solution you'd like Project only the leaves the result needs: - Primitive top-level columns, and the element of a list and the key and value of a map, which match by position and keep their values. - One leaf per struct. A struct keeps the file's validity, which Parquet stores in the definition levels of its leaves, so at least one leaf must still be read. Picking a cheap leaf (for example the narrowest primitive) would keep the cost low. The read result should not change; only the projection mask does. ### Willingness to contribute I would be willing to contribute to this feature with guidance from the Iceberg community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
