16pierre commented on issue #9929: URL: https://github.com/apache/datafusion/issues/9929#issuecomment-3435802315
Thanks a lot for these iterations as well as the crystal-clear examples, these really help. The `TableProvider` approach in `advanced_parquet_index.rs` is powerful, particularly appealing when clients want to optimize predicates pushed down by DF query planner. For other use-cases of `ParquetAccessPlan`, such as static masks applied to Parquet files (deletion vectors etc.), `TableProvider` feels a bit too low-level. I'm personally a bit scared of the maintenance overhead when re-implementing the instantiation of `ParquetSource` etc: I'd much rather delegate to Datafusion default Parquet codepaths & standard configuration, so that we don't miss out on newer changes upon DF upgrades / accidentally regress. I'm currently prototyping some alternative wiring of the `ParquetAccessPlan` via a `FileFormat` wrapper: slightly hacky but I hope to intercept and patch the `FileScanConfig` passed in `FileFormat#create_physical_plan` calls. This comes with the advantage of being a higher-level entrypoint. I'm still curious if we could ease up the process of passing the `ParquetAccessPlan`. Unless I missed some codepaths, patching the `extensions` of `PartitionedFile` felt surprisingly difficult, I would love some easier wiring via `ParquetReadOptions` & cie. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
