unikdahal commented on issue #6737: URL: https://github.com/apache/datafusion-comet/issues/6737#issuecomment-6129447709
A small update before I disappear for about a month: I don't want this design question, or the related epic #6641, to block anyone while I'm away. My summary of where the discussion landed (please correct me if I've misread it): - **Near term:** follow @mbutrovich's suggestion. Build a `DataSource` in datafusion-iceberg that still reads through iceberg-rust's `ArrowReader`, making whatever needs to be `pub` public, and keep the deep Iceberg semantics in iceberg-rust. @alexanderbianchi's apache/datafusion-iceberg#50 already moved `IcebergTableScan` onto `DataSourceExec`, so runtime filters have a standard place to land. - **Longer term:** either run data files through DataFusion's Parquet scan with the Iceberg semantics wrapped around it, or add narrower hooks to `ArrowReader`, such as an injectable `AsyncFileReader` factory plus a way to pass engine-built row filters. Both are still open, and Alex laid out the trade-offs well above. - **For Comet specifically:** my prototype for #6641 runs on the iceberg-rust `ArrowReader` path. If a datafusion-iceberg source that accepts pre-planned `FileScanTask`s shows up, I'd expect the producer side (join / Top-K / MIN-MAX filters) to carry over with little change, because it only publishes dynamic filters. apache/datafusion-iceberg#68 now tracks the dynamic-filter piece with a concrete design, which seems like the natural place for that part to continue. The broader `ParquetSource`-vs-`ArrowReader` question can stay open here. If anyone wants to pick up the Comet side or prototype a pre-planned-tasks entry point, please go ahead; I'll catch up when I'm back. Thanks @mbutrovich, @comphead and @alexanderbianchi for the thoughtful replies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
