alamb commented on PR #27210: URL: https://github.com/apache/datafusion/pull/27210#issuecomment-6975388874
> I think this is more or less the alternative solution you're talking about. I see a couple of possible issues with that approach. Yes, that is basically what I was thinking https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38 > * `RelationPlanner` is not async. As a consequence all metadata you might possible need must have been fetched already during prefetch and there's no clear path to optimising this. This means you pay the cost of fetching all historic metadata regardless of whether you'll actually need it or not. This is a good point. I think we could use the approach shown here for remote catalogs to mostly avoid this: https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38 However that API might need some changes to actually support AS OF times > * The returned scan logical plan is in no way different from a scan of the same table without any or with different options. In other words, at the logical plan level `from t version as of 1` and `from t version as of 2` are identical. I'm not sure if this is currently the case, but something like `select * from t version as of 1 union select * from t version as of 2` could end up being optimised incorrectly because of this. I thought the model was that (effectively) there were different `Arc<dyn TableProvider>`s for each union input > * Very practically, time travel queries are quite common across formats (not just Iceberg). It seems practical for the query planner to know how to handle these so that each extension doesn't have to redo this. The alternative is that every extension implements their own 'versioned scan' logical plan. Maybe this is the key thing we should push on -- if this feature is needed and could be used by more than one consumer then I think there is a strong case for including this in the core engine. For example, I would love to help ensure that https://github.com/apache/datafusion-iceberg ends up supporting time travel queries too So as a practical step forward, what do you think about making an example, in the spirit of https://github.com/apache/datafusion/blob/b7b014e731/datafusion-examples/examples/data_io/remote_catalog.rs#L38-L96 (or maybe extend it) to show how to connect in AS OF type queries? @alexanderbianchi I wonder if you could maybe try a POC integration of this API with datafusion-iceberg to verify this API would work for that project? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
