NoahKusaba opened a new issue, #2881:
URL: https://github.com/apache/datafusion-ballista/issues/2881

   **Is your feature request related to a problem or challenge? Please describe 
what you are trying to do.**
   
   `ballista-iceberg` (#2638) pins each catalog-backed Iceberg table to one 
version when a query looks it up, in `PinningSchema::table`, so the plan can be 
encoded without I/O. #2784 extends that to views by re-pinning a view's stored 
plan on each lookup. Choosing the version at lookup still leaves gaps that 
single-process DataFusion doesn't have, since there 
`IcebergCatalogTableProvider` loads its table in `scan()`:
   
   - A reused `DataFrame`, or a Flight SQL prepared statement run more than 
once, reads the version from when it was planned. Neither looks its tables up 
again.
   - A query that reads a table both through a view and directly looks it up 
twice, so a commit landing between the two lookups gives them different 
versions.
   - Lookups do catalog I/O, so anything that only needs a table's name, type 
or schema (`DROP`, `DESCRIBE`, `information_schema.columns`) loads it, and 
fails if it can't be loaded. #2784 works around this for views with 
`RepinnedView`, and the same problem for base tables is tracked separately.
   
   **Describe the solution you'd like**
   
   Pin when a query runs: in the async `QueryPlanner::create_physical_plan`, 
walk the whole plan once, load each distinct catalog-backed table once, and 
swap in a pinned version of each scan. Lookups would then do no I/O, every scan 
of a table in a query would read one version, and each run of a reused 
`DataFrame` or prepared statement would read the latest version. The re-pinning 
in `PinningSchema::table` and `RepinnedView` would go away.
   
   This needs a hook in Ballista's core first. 
`SessionStateExt::upgrade_for_ballista` (`ballista/core/src/extension.rs`) only 
lets `ballista_query_planner()` replace `BallistaQueryPlanner`, so an 
integration can't wrap it. Either a composable query planner, or a hook that 
runs on the logical plan before `DistributedQueryExec` is built, would do. The 
scheduler plans Flight SQL queries itself, so it needs the same hook.
   
   **Describe alternatives you've considered**
   
   - Pin at lookup and re-pin views (#2638, #2784): what exists now, with the 
gaps above.
   - Re-plan a view from its SQL: `SchemaProvider::table` gets no session state 
to plan with, and a view created from a `DataFrame` has no SQL (#2726).
   
   **Additional context**
   
   Follow-up to #2638 and #2784.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to