yjshen commented on issue #944: URL: https://github.com/apache/arrow-datafusion/issues/944#issuecomment-907614249
Thanks @houqp for the detailed explanation. After checking delta-rs code as well as what we are doing currently in DataFusion. I have several thoughts: - It's more natural to have the TableProvider for dealing with filters since it's the abstraction over a table, therefore also a suitable entity for table partition (inferred or user-provided). we could enhance TableProvider with per partition laziness for partition file listing and file metadata extraction. - `max_partition` shouldn't be related to a Table. It's more reasonable to be deduced or set manually through the planning phase and passed to `ParquetExec` where it actually takes effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
