pepijnve commented on PR #27210:
URL: https://github.com/apache/datafusion/pull/27210#issuecomment-6923562626

   > Basically I am trying to figure out what the core rationale is for 
including time travel queries in the core.
   
   I started from what Paimon's DataFusion integration does which can be seen 
at 
https://github.com/apache/paimon-rust/blob/main/crates/integrations/datafusion/src/relation_planner.rs#L121.
 What's done there is that the table name is first looked up using the bare 
table reference in the prefetched table map. Then a downcast is done, and a new 
TableProvider is derived that takes the options from the TableFactor into 
account. This is returned as a logical scan of the bare table reference.
   
   I think this is more or less the alternative solution you're talking about. 
I see a couple of possible issues with that approach.
   - `RelationPlanner` is not async. As a consequence all metadata you might 
possible need must have been fetched already during prefetch and there's no 
clear path to optimising this. This means you pay the cost of fetching all 
historic metadata regardless of whether you'll actually need it or not.
   - The returned scan logical plan is in no way different from a scan of the 
same table without any or with different options. In other words, at the 
logical plan level `from t version as of 1` and `from t version as of 2` are 
identical. I'm not sure if this is currently the case, but something like 
`select * from t version as of 1 union select * from t version as of 2` could 
end up being optimised incorrectly because of this.
   - Very practically, time travel queries are quite common across formats (not 
just Iceberg). It seems practical for the query planner to know how to handle 
these so that each extension doesn't have to redo this. The alternative is that 
every extension implements their own 'versioned scan' logical plan.
   
   What I'm experimenting with here is if there's some way to add an extension 
point that lets us improve upon the above without restricting ourselves to just 
time travel. What came out of a bunch of experimenting is two needs:
   - During the async prefetch phase, you want some way to extract information 
out of the `TableFactor` which can then be passed to the `SchemaProvider` so 
that it knows which metadata is actually needed. It only needs to return a 
`TableProvider` that can read exactly what was asked for.
   - If I perform two different lookups like this I need some way to tell the 
two `TableProvider` instances apart. We could key this on the `TableReference` 
along with the request value (i.e. the info extracted from the `TableFactor`), 
but then it's no longer possible for multiple requests that resolve to the same 
thing to actually be identical. When using `timestamp as of` for instance it's 
quite possible that two distinct timestamps resolve to the same snapshot. If we 
want to be able to detect this situation, then we need something that 
identifies a particular version (or more generally aspect) of a table that may 
be different from the request parameters.
   
   In @alexanderbianchi's suggestion the request and the identity values use 
two distinct types. Currently in this PR I'm using the same type as 
representation for both, but the request and identity values are allowed to be 
different. For lack of a better name this is currently called a 'table read 
selector'.
   
   The 'table read selector planner' is the bit that converts `TableFactor` to 
something independent of the SQL AST. 
`SchemaProvider::table_with_read_selector` performs the lookup during prefetch. 
`TableProvider::read_selector` is the identity part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to