Tim Meehan created SPARK-61466:
----------------------------------
Summary: Add native scalar functions: the C interface spark_udf.h
and USING PACKAGE
Key: SPARK-61466
URL: https://issues.apache.org/jira/browse/SPARK-61466
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Tim Meehan
Add native scalar functions: scalar functions implemented in a native library,
for example in C++ or Rust, that Spark calls in the executor JVM through the
Arrow C data interface, with no Python worker. They ship in the native packages
(.sparkpkg) of native data sources and follow the conventions of
spark_datasource.h version 1. A library can implement both interfaces.
Proposed changes:
- A C interface, spark_udf.h, version 1, next to spark_datasource.h: an init
that runs once per JVM and negotiates the version, and a create / eval / close
lifecycle for each call of a function in each task. A batch is an Arrow struct
array with one child per argument. Errors are int return codes with an optional
last_error.
- Two new manifest fields, udfAbiVersion and functions. Each function has a
name, a kind (scalar), argument and result types as DDL strings, and a
deterministic flag. A package may contain only functions.
- Registration with CREATE [OR REPLACE] TEMPORARY FUNCTION f AS '<name in the
manifest>' USING PACKAGE '<package>'. The types come from the manifest, so the
driver never loads the library. Persistent functions are not supported yet.
- Evaluation in the task thread through the Foreign Function and Memory API, on
Java 22 or later. The functions are called from SQL, the DataFrame API
(call_function) and Spark Connect, with no protocol change.
- A static setting, spark.sql.udf.native.enabled, true by default, and new
error classes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]