backkem commented on issue #970:
URL:
https://github.com/apache/arrow-datafusion/issues/970#issuecomment-1885112386
Based on the discussion above and in #7871 and #8699, I wanted to further
explore the idea of the `TableProvider` 'consuming' part of a plan. I wrote up
the following sudo-code to help illustrate this:
```Rust
// Optimizer rule that facilitates federation.
struct FederationOptimizerRule {}
impl OptimizerRule for FederationOptimizerRule {
fn try_optimize(
&self,
plan: &LogicalPlan,
config: &dyn OptimizerConfig,
) -> Result<Option<LogicalPlan>> {
// Walk over the plan, look for the largest subtrees with only
// FederatedTableSource's all using the same FederationProvider.
// Pass each subtrees to its respective FederationProvider.optimizer
// The job of this optimizer is to 'absorb' part of the plan and
replace it with
// one or more virtual TableScan's that wrap the piece of the plan
it will federate.
}
}
trait FederationProvider {
fn optimizer(&self) -> Option<Arc<Optimizer>>;
// Add Analyzer & PhysicalOptimizer if needed.
}
// FederatedTableSource that helps the FederationOptimizerRule find the
'largest subtrees'
// in a plan and run the corresponding FederationProvider.optimizer
trait FederatedTableSource: TableSource {
fn federation_provider(&self) -> Arc<dyn FederationProvider>;
fn table_type(&self) -> TableType {
TableType::Temporary
}
}
// TableProvider (or a new trait) is simplified since the virtual TableScan
// injected into the plan by the FederationProvider Optimizer already knows
what to do.
trait TableProvider {
async fn scan(&self, state: &SessionState) -> Result<Arc<dyn
ExecutionPlan>>;
}
```
It should be feasible to create a wrapper between the old and the new
`TableProvider` traits to port all existing implementations forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
