zozo123 opened a new pull request, #59231: URL: https://github.com/apache/spark/pull/59231
Depends on #59155, #59230, and #58713. Do not review this diff and do not merge it before those pull requests merge. This branch currently points at `ec604460`, the same commit as #59155. After the dependencies merge, rebase this branch onto master and replace this placeholder with the Python worker commits. ### Dependencies - #59155 proves the scalar external UDF through a spawned gRPC worker on the sql test classpath. - #59230 shades the gRPC worker runtime and proves the query from the assembly classpath. - #58713 defines the payload and Init messages. Haiyang owns that work. - SPARK-59083 is the first worker that computes instead of echoing Arrow batches. ### Sandbox boundary Spark keeps one generic boundary. `DirectWorkerDispatcher` runs the worker command and appends `--id` and `--connection`. The CPython worker binds that Unix socket and speaks the external UDF protocol. A local sandbox wraps the worker command. A container runtime mounts the socket directory, runs with the executor uid, and stays in the foreground. Spark does not gain a Docker, Monty, Boat, or sandbox-provider API. A remote sandbox needs an external launcher. The launcher binds Spark's local Unix socket, starts the remote worker, forwards gRPC bytes over an authenticated tunnel, and cleans up the remote machine when Spark terminates it. That launcher is not part of Spark. Monty can be an optional evaluator behind the CPython or Rust worker. It cannot own the socket or decode Arrow because it has no sockets, subprocesses, or third-party packages. ### What changes were proposed in this pull request? Nothing yet. The follow-up will add a CPython worker that decodes Arrow, executes a scalar function, and returns Arrow through the existing gRPC Unix socket. ### Why are the changes needed? The current echo worker proves transport only. It does not prove that a language worker can deserialize a function and compute output rows. ### Does this PR introduce _any_ user-facing change? No. This draft has no Python worker commit. ### How was this patch tested? Not tested. There is no new commit beyond #59155. The eventual verification will run a non-identity scalar function from a packaged Spark distribution and assert literal output rows. It will also verify worker termination and socket cleanup after task cancellation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.7 Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
