zozo123 commented on code in PR #59230: URL: https://github.com/apache/spark/pull/59230#discussion_r4291862754
########## udf/worker/runtime/pom.xml: ########## @@ -0,0 +1,200 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one or more + ~ contributor license agreements. See the NOTICE file distributed with + ~ this work for additional information regarding copyright ownership. + ~ The ASF licenses this file to You under the Apache License, Version 2.0 + ~ (the "License"); you may not use this file except in compliance with + ~ the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 + http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.spark</groupId> + <artifactId>spark-parent_2.13</artifactId> + <version>5.0.0-SNAPSHOT</version> + <relativePath>../../../pom.xml</relativePath> + </parent> + + <artifactId>spark-udf-worker-runtime_2.13</artifactId> + <packaging>jar</packaging> + <name>Spark Project UDF Worker Runtime</name> + <url>https://spark.apache.org/</url> + + <properties> + <sbt.project.name>udf-worker-runtime</sbt.project.name> + </properties> + + <!-- + Packages spark-udf-worker-grpc for the Spark runtime classpath. This module has + no main sources: its jar is spark-udf-worker-grpc plus the gRPC runtime, + with gRPC, Guava and their small helper libraries relocated under + org.sparkproject.spark_udf_worker so they cannot clash with Spark Connect's + relocated gRPC or with the Guava that Spark exposes to applications. + + Two dependencies are deliberately left unrelocated and unbundled: + - Netty. gRPC's Netty transport links to the Netty that Spark ships, including + the epoll and kqueue native transports that Unix domain sockets need. + Relocating Netty would change the native library names it loads. + - protobuf-java. The generated stubs and gRPC's protobuf marshallers must use + the same protobuf classes as the spark-udf-worker-proto messages, which are + not part of this jar. + + The relocations must stay consistent with UDFWorkerRuntime in + project/SparkBuild.scala. + --> + <dependencies> + <dependency> + <groupId>org.apache.spark</groupId> + <artifactId>spark-udf-worker-grpc_${scala.binary.version}</artifactId> + <version>${project.version}</version> + </dependency> + <!-- Bundled and relocated below; the parent manages both as provided. --> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + <scope>compile</scope> + </dependency> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>failureaccess</artifactId> + <scope>compile</scope> + </dependency> + <!-- + The jar links to classes it does not bundle: the UDF worker proto and core + modules, Netty and protobuf-java. Spark provides them, and declaring Spark core Review Comment: @HyukjinKwon the follow-up you asked for is now on this branch as `fcc1f160b6b`. The previous head `7669c969` stays in the history. This commit does not rewrite it. **POM comments.** Your classpath reading of `7669c969` was right. I measured that Maven distribution and posted the `NoClassDefFoundError` and the `setData` signature split. `udf/worker/runtime/pom.xml` now states the contract those measurements require. The runtime jar bundles proto, core, grpc, protobuf-java, and Guava, and relocates them under `org.sparkproject.spark_udf_worker`. Engine classes call `WorkerMessages` and do not name `com.google.protobuf`. Netty stays unrelocated. **Scaladoc.** The sentence that said worker creation fails because no dispatcher is configured is replaced. `SparkEnv` loads the direct dispatcher from `spark-udf-worker-runtime`. An empty runner command fails before a process starts. The flow-control sentence stays. **CI.** The packaged guard is `UdfWorkerRuntimeSparkSubmitSuite` in `sql/core`, because the test spark-submits and the SQL extended job builds the assembly. `udf-worker` stays selected in the core and utils group for the module tests. I did not add a second copy of that suite to the connect group. Fork CI run https://github.com/zozo123/spark/actions/runs/38798924119 passed the SQL extended tests on the parent of this tree. **Flow control.** Shipping behind `spark.sql.execution.udf.unified.execution.enabled`, which defaults to false, is not the production path in the `GrpcWorkerSession` TODO. That TODO and the `ExecuteExternalUDFExec` note now say the transport ships behind the flag, and that flow control is still required before the flag leaves experimental status. Flow control itself is not in this PR. **Maven test.** `./build/mvn -B -pl assembly -am -DskipTests package` at `d4ec6f12a0c` on JDK 21 produced the assembly. `UdfWorkerRuntimeSparkSubmitSuite` passed in 14.360 seconds. The runtime jar checksum stayed `dba42225fbcee6a78c299f1872108411d5909b3d7f89ad1010724ed3d5a7caa0`. The rows were the `Range(0, 6)` echo rows. `fcc1f160b6b` has the same tree for the jar, plus the comments above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
