Huaxin Gao created SPARK-59283:
----------------------------------
Summary: SPIP: Catalog-backed Code-Literal Functions (SQL and
Python) with Catalog SPI and CRUD
Key: SPARK-59283
URL: https://issues.apache.org/jira/browse/SPARK-59283
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.4.0
Reporter: Huaxin Gao
*SPIP: Catalog-backed Code-Literal Functions (SQL and Python) with Catalog SPI
and CRUD*
Full proposal (Google Doc):
https://docs.google.com/document/d/186cTAZxoXp1p8vaSunIaJmVLXcPR-FxSiLiDUl8kK8A/edit?tab=t.0
Discussion thread:
https://www.mail-archive.com/[email protected]/msg34809.html
*Summary*
Let Spark call functions whose "code" is stored as text in an external catalog
(a SQL expression or Python code), managed through normal SQL DDL (CREATE /
CREATE OR REPLACE / DROP FUNCTION) and callable in queries without manual
session registration.
The proposal introduces a language-agnostic Java SPI
(CodeLiteralFunctionCatalog + CodeFunctionSpec) for catalogs to expose
code-literal functions as metadata:
* Scalar functions with spark-sql, python, or python-pandas implementations.
* SQL table-valued functions (SQL body as text).
* Full CRUD delegated through existing SQL DDL.
Spark inlines deterministic SQL bodies during analysis (so they benefit from
the optimizer, e.g. constant folding) and binds Python bodies to the existing
Python/pandas UDF runtimes (opaque to the optimizer).
*Scope (non-goals)*
Function overloads, named/default arguments, caching/eventing, cross-function
references/recursion, non-Spark SQL dialects, and Python TVF/UDTF execution are
out of scope for the first version.
*Key risks*
* Security/governance: executing catalog-hosted code requires trust boundaries
and guardrails (permissions, auditing, allowlists).
* Function-resolution precedence must be clearly defined against built-ins,
temp/session, and DSv2 functions.
* Safe SQL inlining (parameter substitution, type coercion, determinism) and
minimal, explicit Python environment handling.
*Plan*
Phased: (1) core SPI + SQL scalar + DDL delegation, (2) Python/pandas scalar
binding, (3) SQL TVF, (4) hardening (errors, docs, tests). See the linked doc
for API details (CodeLiteralFunctionCatalog, CodeFunctionSpec), resolution
precedence, error mapping, and configuration flags.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
