jorgecarleitao commented on a change in pull request #8097:
URL: https://github.com/apache/arrow/pull/8097#discussion_r482968858
##########
File path: rust/datafusion/src/physical_plan/planner.rs
##########
@@ -331,6 +355,16 @@ impl DefaultPhysicalPlanner {
let schema_ref = Arc::new(schema.as_ref().clone());
Ok(Arc::new(ExplainExec::new(schema_ref, stringified_plans)))
}
+ LogicalPlan::Extension { node } => {
+ let inputs = node
+ .inputs()
+ .into_iter()
+ .map(|input_plan| self.create_physical_plan(input_plan,
ctx_state))
+ .collect::<Result<Vec<_>>>()?;
+
+ self.extension_planner
Review comment:
I am sorry, I did not mean to say that we should do it in this PR - it
does not. I just got too excited about this ^_^
The use case:
You publish the `TopK` extension as a (library) crate called
`datafusion-topk`, and I publish a crate `datafusion-s3` with another extension.
A user wants to use both extensions. It installs them by:
1. adding each crate to `Cargo.toml`
2. initialize the default planner with both of them
3. plan them
4. execute them
I.e. freaking easy!
Broadly speaking, this allows the existence of an ecosystem of
extensions/user-defined plans: people can share hand-crafted plans and plans
can be added as dependencies to the crate and registered to the planner to be
used by other people. 🤯
This also reduces the pressure of placing everything in DataFusion's
codebase: if we offer an API to extend DataFusion in this way, people can just
distribute libraries with the extension/user-defined plan without having to go
through the decision process of whether X is part of DataFusion's core or not
(e.g. a scan of format Y, or a scan over protocol Z).
For me, this use case does require an easy way to achieve `2. initialize the
default planner with both of them`. But again, this PR is definitely a major
step in this direction!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
