comphead commented on PR #4459: URL: https://github.com/apache/datafusion-comet/pull/4459#issuecomment-5295287707
> FYI: One potentially related Spark effort is [SPARK-55278: Language-agnostic UDF Protocol for Spark](https://issues.apache.org/jira/browse/SPARK-55278), an accepted SPIP currently under implementation. Its [design](https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8/edit) standardizes external UDF execution around Arrow, a structured IPC protocol, capability-driven planning, and a declarative [worker specification](https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs/edit). This may be relevant to the longer-term direction and worth referencing, but it should not block the current implementation. Thanks @wForget for the hint. Any idea if it lands in Spark 5 without backports to Spark 4? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
