mbutrovich opened a new issue, #1749: URL: https://github.com/apache/iceberg-rust/issues/1749
### What's the feature are you trying to implement? Apache DataFusion Comet is an Apache Spark accelerator with Apache Iceberg support. We would like to enhance that support by leveraging Iceberg-Rust. You can find the details of this effort in the POC PR https://github.com/apache/datafusion-comet/pull/2528 and in [slides presented at the 10/9/25 Iceberg-Rust community call](https://github.com/user-attachments/files/22930897/iceberg-rust.pdf). The short version is the Comet will rely on Apache Iceberg's Java integration with Apache Spark for planning, and then pass those generated `FileScanTask`s to Iceberg-Rust via a new DataFusion `IcebergScan` operator in Comet. We need a lot of new (or just public) APIs in the `ArrowReader` since we are bypassing the `Table` interface to avoid redundant (and possibly incorrect partitioned) planning. I will start to accumulate those efforts here. - [ ] Make `ArrowReaderBuilder::new` `pub` instead of `pub(crate)`. - [ ] Expose decryption options in `ArrowReaderBuilder`. This likely requires a new Iceberg-Rust Cargo feature like in DataFusion to enable the `encryption` feature for the Parquet crate. - [ ] Expose `ArrowReaderOptions` in `ArrowReaderBuilder`. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
