zozo123 opened a new pull request, #59153: URL: https://github.com/apache/spark/pull/59153
### What changes were proposed in this pull request? Replace the arbitrary `spark.udf.worker.dispatcherFactory` class hook introduced by SPARK-59822 with Spark-owned routing from `UDFWorkerSpecification.worker`. The protobuf `oneof worker` already generates the closed `WorkerCase` enum, so it is the single source of truth for worker creation mode. `DIRECT` maps to a hard-coded Spark implementation, `DirectDispatcherFactory`, in `spark-udf-worker-grpc`. `core` loads only that Spark-owned class name reflectively because it intentionally has no compile-time dependency on the gRPC module. User-provided class names are never loaded. The old class-name setting is removed and explicitly rejected if present. The lazy dispatcher-manager lifecycle coordination from SPARK-59822 is preserved. ### Why are the changes needed? An arbitrary factory class executes operator-provided JVM code inside Spark and makes engine behavior and security policy unpredictable. Worker-specific commands, environment setup, capabilities, and connection requirements already belong in `UDFWorkerSpecification` and execute outside the Spark JVM. A separate dispatcher enum config would duplicate `WorkerCase` and could disagree with the specification. Routing from the existing closed `oneof` keeps Spark in control of dispatch while leaving worker customization in the worker-spec contract. ### Does this PR introduce _any_ user-facing change? Yes, relative to unreleased master and branch-4.x. `spark.udf.worker.dispatcherFactory` is removed; custom dispatcher classes are no longer supported. A `DirectWorker` specification selects Spark's built-in direct dispatcher automatically. The gRPC implementation is still outside Spark's runtime classpath until SPARK-57214 packages and shades it, so direct execution currently reports an actionable missing-runtime error. This PR does not make gRPC a dependency of `core`. ### Follow-up direction Indirect or remote worker creation should be a generic new arm of `UDFWorkerSpecification.worker` (for example `ProvisionedWorker`) with provisioning, connection, lifecycle, reuse, and security semantics. It should not add vendor-specific dispatchers or restore arbitrary JVM plugins. If this direction is aligned with the SPIP, could a maintainer please create a sibling sub-task to track that worker-spec contract separately from SPARK-59886? ### How was this patch tested? - `core/testOnly org.apache.spark.SparkEnvUDFDispatcherSuite` - `udf-worker-grpc/testOnly org.apache.spark.udf.worker.grpc.DirectGrpcDispatcherSuite` - `sql/testOnly org.apache.spark.sql.execution.externalUDF.PlanExternalUDFsSuite` - `dev/scalafmt` The suites cover unset worker types, `DIRECT` routing, rejection of the removed class setting without class initialization, shutdown racing with lazy manager creation, concrete factory construction, and the SQL execution boundary. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Auto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
