alamb commented on PR #27210:
URL: https://github.com/apache/datafusion/pull/27210#issuecomment-6975388874

   > I think this is more or less the alternative solution you're talking 
about. I see a couple of possible issues with that approach.
   
   Yes, that is basically what I was thinking
   
   
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38
   
   > * `RelationPlanner` is not async. As a consequence all metadata you might 
possible need must have been fetched already during prefetch and there's no 
clear path to optimising this. This means you pay the cost of fetching all 
historic metadata regardless of whether you'll actually need it or not.
   
   This is a good point. I think we could use the approach shown here for 
remote catalogs to mostly avoid this: 
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38
   
   However that API might need some changes to actually support AS OF times
   
   
   > * The returned scan logical plan is in no way different from a scan of the 
same table without any or with different options. In other words, at the 
logical plan level `from t version as of 1` and `from t version as of 2` are 
identical. I'm not sure if this is currently the case, but something like 
`select * from t version as of 1 union select * from t version as of 2` could 
end up being optimised incorrectly because of this.
   
   I thought the model was that (effectively) there were different `Arc<dyn 
TableProvider>`s for each union input
   
   > * Very practically, time travel queries are quite common across formats 
(not just Iceberg). It seems practical for the query planner to know how to 
handle these so that each extension doesn't have to redo this. The alternative 
is that every extension implements their own 'versioned scan' logical plan.
   
   Maybe this is the key thing we should push on -- if this feature is needed 
and could be used by more than one consumer then I think there is a strong case 
for including this in the core engine. 
   
   For example, I would love to help ensure that  
https://github.com/apache/datafusion-iceberg ends up supporting time travel 
queries too
   
   So as a practical step forward, what do you think about making an example, 
in the spirit of 
https://github.com/apache/datafusion/blob/b7b014e731/datafusion-examples/examples/data_io/remote_catalog.rs#L38-L96
 (or maybe extend it) to show how to connect in AS OF type queries?
   
   @alexanderbianchi  I wonder if you could maybe try a POC integration of this 
API with datafusion-iceberg to verify this API would work for that project?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to