haiyangsun-db opened a new pull request, #58988: URL: https://github.com/apache/spark/pull/58988
### What changes were proposed in this pull request? This PR adds the language-neutral initialization and execution bridge for scalar external UDFs: * Extend worker and session specifications with declarative properties, environment variables, and resource-directory requirements, and build complete `Init` messages from them. * Preserve input metadata, including named arguments, and add a versioned PySpark UDF payload adapter without changing the existing Python UDF execution path. * Execute scalar external UDFs through dispatcher-managed worker sessions using Arrow batches and `HybridRowQueue` to pair worker output with the original input rows. * Limit input batches by both row count and byte size, expose additive Arrow conversion helpers, and report output cardinality mismatches with a structured error. * Treat external UDF results as nullable until the worker protocol can enforce declared non-nullability. The dispatcher is intentionally not configured with a worker implementation in this PR. Worker creation therefore fails before starting a process; the worker server will be added separately. ### Why are the changes needed? The external UDF framework can describe and plan UDFs, but it does not yet have a generic contract for per-session initialization or a physical execution path that exchanges data with a worker. This leaves `ExecuteExternalUDFExec` unable to execute and would otherwise require language-specific behavior in the physical operator. The proposed bridge keeps worker requirements declarative, resolves only engine-owned context, and provides the Arrow data path independently of any particular worker server implementation. ### Does this PR introduce _any_ user-facing change? No. This extends unreleased external UDF infrastructure. Existing Python UDF execution is unchanged, and no external UDF worker implementation is enabled by this PR. ### How was this patch tested? Added and extended tests covering initialization metadata, property and resource validation, named arguments, planning and nullability, Arrow row- and byte-based batching, empty input, multiple partitions and batches, early iterator termination, session cleanup, and output cardinality mismatches. Focused SBT suites have not yet been run for this draft while its final combined scope is still being assembled. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
