[ 
https://issues.apache.org/jira/browse/SPARK-61466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61466:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add native scalar functions: the C interface spark_udf.h and USING PACKAGE
> --------------------------------------------------------------------------
>
>                 Key: SPARK-61466
>                 URL: https://issues.apache.org/jira/browse/SPARK-61466
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Tim Meehan
>            Priority: Major
>              Labels: pull-request-available
>
> Add native scalar functions: scalar functions implemented in a native 
> library, for example in C++ or Rust, that Spark calls in the executor JVM 
> through the Arrow C data interface, with no Python worker. They ship in the 
> native packages (.sparkpkg) of native data sources and follow the conventions 
> of spark_datasource.h version 1. A library can implement both interfaces.
> Proposed changes:
> - A C interface, spark_udf.h, version 1, next to spark_datasource.h: an init 
> that runs once per JVM and negotiates the version, and a create / eval / 
> close lifecycle for each call of a function in each task. A batch is an Arrow 
> struct array with one child per argument. Errors are int return codes with an 
> optional last_error.
> - Two new manifest fields, udfAbiVersion and functions. Each function has a 
> name, a kind (scalar), argument and result types as DDL strings, and a 
> deterministic flag. A package may contain only functions.
> - Registration with CREATE [OR REPLACE] TEMPORARY FUNCTION f AS '<name in the 
> manifest>' USING PACKAGE '<package>'. The types come from the manifest, so 
> the driver never loads the library. Persistent functions are not supported 
> yet.
> - Evaluation in the task thread through the Foreign Function and Memory API, 
> on Java 22 or later. The functions are called from SQL, the DataFrame API 
> (call_function) and Spark Connect, with no protocol change.
> - A static setting, spark.sql.udf.native.enabled, true by default, and new 
> error classes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to