[
https://issues.apache.org/jira/browse/SPARK-61466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61466:
-----------------------------------
Labels: pull-request-available (was: )
> Add native scalar functions: the C interface spark_udf.h and USING PACKAGE
> --------------------------------------------------------------------------
>
> Key: SPARK-61466
> URL: https://issues.apache.org/jira/browse/SPARK-61466
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Tim Meehan
> Priority: Major
> Labels: pull-request-available
>
> Add native scalar functions: scalar functions implemented in a native
> library, for example in C++ or Rust, that Spark calls in the executor JVM
> through the Arrow C data interface, with no Python worker. They ship in the
> native packages (.sparkpkg) of native data sources and follow the conventions
> of spark_datasource.h version 1. A library can implement both interfaces.
> Proposed changes:
> - A C interface, spark_udf.h, version 1, next to spark_datasource.h: an init
> that runs once per JVM and negotiates the version, and a create / eval /
> close lifecycle for each call of a function in each task. A batch is an Arrow
> struct array with one child per argument. Errors are int return codes with an
> optional last_error.
> - Two new manifest fields, udfAbiVersion and functions. Each function has a
> name, a kind (scalar), argument and result types as DDL strings, and a
> deterministic flag. A package may contain only functions.
> - Registration with CREATE [OR REPLACE] TEMPORARY FUNCTION f AS '<name in the
> manifest>' USING PACKAGE '<package>'. The types come from the manifest, so
> the driver never loads the library. Persistent functions are not supported
> yet.
> - Evaluation in the task thread through the Foreign Function and Memory API,
> on Java 22 or later. The functions are called from SQL, the DataFrame API
> (call_function) and Spark Connect, with no protocol change.
> - A static setting, spark.sql.udf.native.enabled, true by default, and new
> error classes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]