Tim Meehan created SPARK-61466:
----------------------------------

             Summary: Add native scalar functions: the C interface spark_udf.h 
and USING PACKAGE
                 Key: SPARK-61466
                 URL: https://issues.apache.org/jira/browse/SPARK-61466
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Tim Meehan


Add native scalar functions: scalar functions implemented in a native library, 
for example in C++ or Rust, that Spark calls in the executor JVM through the 
Arrow C data interface, with no Python worker. They ship in the native packages 
(.sparkpkg) of native data sources and follow the conventions of 
spark_datasource.h version 1. A library can implement both interfaces.

Proposed changes:
- A C interface, spark_udf.h, version 1, next to spark_datasource.h: an init 
that runs once per JVM and negotiates the version, and a create / eval / close 
lifecycle for each call of a function in each task. A batch is an Arrow struct 
array with one child per argument. Errors are int return codes with an optional 
last_error.
- Two new manifest fields, udfAbiVersion and functions. Each function has a 
name, a kind (scalar), argument and result types as DDL strings, and a 
deterministic flag. A package may contain only functions.
- Registration with CREATE [OR REPLACE] TEMPORARY FUNCTION f AS '<name in the 
manifest>' USING PACKAGE '<package>'. The types come from the manifest, so the 
driver never loads the library. Persistent functions are not supported yet.
- Evaluation in the task thread through the Foreign Function and Memory API, on 
Java 22 or later. The functions are called from SQL, the DataFrame API 
(call_function) and Spark Connect, with no protocol change.
- A static setting, spark.sql.udf.native.enabled, true by default, and new 
error classes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to