zozo123 opened a new pull request, #59155: URL: https://github.com/apache/spark/pull/59155
Depends on #59153. The commits before `00ece1b3` are that pull request. Please review `00ece1b3` only. ### What changes were proposed in this pull request? `PlanExternalUDFsSuite` runs a scalar external UDF through the real `DIRECT` path. `ExecuteExternalUDFExec` asks SparkEnv for the built-in dispatcher. `DirectGrpcDispatcher` starts `EchoGrpcWorkerMain` and talks to it over a Unix domain socket. The echo worker returns each Arrow batch unchanged, so a bigint column is an identity function. The test input is `Range(0, 6)` in two partitions. The collected rows are `(0, 0)` through `(5, 5)`. `sql/core` gains a test-scoped dependency on `spark-udf-worker-grpc` and its test jar, so the reflective load of `DirectDispatcherFactory` succeeds in this suite. `core` still does not depend on gRPC. This replaces the suite's missing-runtime assertion. `SparkEnvUDFDispatcherSuite` still covers that error on the `core` classpath, which does not contain the gRPC module. ### Why are the changes needed? #59153 routes `DIRECT` to the built-in factory, and the SQL node can already stream Arrow batches. Nothing in `sql/core` checked that a query returns rows from a spawned worker. This test is that check. Shading and a Python worker stay in SPARK-57214 and the Python worker tickets. ### Does this PR introduce _any_ user-facing change? No. The new dependency is test scope. A packaged Spark still reports the missing runtime until SPARK-57214. ### How was this patch tested? - `sql/testOnly org.apache.spark.sql.execution.externalUDF.PlanExternalUDFsSuite` on macOS. 32 tests passed, including this one. The log shows one kqueue domain socket per task. - The same suite on Linux aarch64. 32 tests passed. The log shows epoll domain sockets. - `sql/Test/scalastyle` reported 0 errors. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.7 Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
