thinkharderdev commented on pull request #1881:
URL:
https://github.com/apache/arrow-datafusion/pull/1881#issuecomment-1068866173
> @Ted-Jiang @alamb
>
> In fact, after the external UDF/UDAF is registered with datafusion's
`ExecutionContext` via the `register_udf` function, As long as we know the name
of UDF/UDAF, we can obtain the implementation of UDF/UDAF from ExecutionContext
in the following ways to complete the deserialization of UDF/UDAF.
>
> ```
> impl FunctionRegistry for ExecutionContext {
> fn udfs(&self) -> HashSet<String> {
> self.state.lock().udfs()
> }
>
> fn udf(&self, name: &str) -> Result<Arc<ScalarUDF>> {
> self.state.lock().udf(name)
> }
>
> fn udaf(&self, name: &str) -> Result<Arc<AggregateUDF>> {
> self.state.lock().udaf(name)
> }
> }
> ```
>
> So we can put name in proto when we serialize UDF/UDAF, and then we can
deserialize UDF/UDAF using this name and ExecutionContext. But that need add a
function like this:
>
> ```
> fn from_proto_to_logicalplan(logical_plan: &protobuf::LogicalExprNode,
ctx: &ExecutionContext) -> Result<LogicalPlan, Error>
> ```
>
> There is no doubt that it is also a good way to add a ballista-plugin
crate and make both datafusion-proto and ballista-core dependent on
ballista-plugin. This approach may require fewer changes to the code.
I like this approach. In our project we plan on building a custom ballista
binary so we can register our `ExecutionContext` at startup. I think using the
`ExecutionContext` rather than the plugin manager directly in the serde should
support both approaches.
In general I think the design with respect to plugins is that they are
loaded into the `ExecutionContext` (or `SessionContext` when that works is
merged) at startup and then internally the context is used. This should support
both use cases as well as allow users to build Ballista without the plugin
manager if they do not need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
