Holden Karau created SPARK-55213:
------------------------------------
Summary: Developer hook API for accelerator-specific UDF
transpilation
Key: SPARK-55213
URL: https://issues.apache.org/jira/browse/SPARK-55213
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3
Reporter: Holden Karau
Let accelerators add their own UDF translations safely.
Add an internal API that allows:
* Registration of transpilers
* Safe fallback semantics
This enables CUDA, cuDF, Rust, or future engines to participate.
We can likely do this through adding more hooks to the existing extension
mechanism but since we might want to support it Python side not just Scala side
we might need to do something more from the ground up (the majority of
extension work only functions in JVM land from memory)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
