HyukjinKwon commented on code in PR #59230: URL: https://github.com/apache/spark/pull/59230#discussion_r4265503170
########## udf/worker/runtime/pom.xml: ########## @@ -0,0 +1,200 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one or more + ~ contributor license agreements. See the NOTICE file distributed with + ~ this work for additional information regarding copyright ownership. + ~ The ASF licenses this file to You under the Apache License, Version 2.0 + ~ (the "License"); you may not use this file except in compliance with + ~ the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 + http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.spark</groupId> + <artifactId>spark-parent_2.13</artifactId> + <version>5.0.0-SNAPSHOT</version> + <relativePath>../../../pom.xml</relativePath> + </parent> + + <artifactId>spark-udf-worker-runtime_2.13</artifactId> + <packaging>jar</packaging> + <name>Spark Project UDF Worker Runtime</name> + <url>https://spark.apache.org/</url> + + <properties> + <sbt.project.name>udf-worker-runtime</sbt.project.name> + </properties> + + <!-- + Packages spark-udf-worker-grpc for the Spark runtime classpath. This module has + no main sources: its jar is spark-udf-worker-grpc plus the gRPC runtime, + with gRPC, Guava and their small helper libraries relocated under + org.sparkproject.spark_udf_worker so they cannot clash with Spark Connect's + relocated gRPC or with the Guava that Spark exposes to applications. + + Two dependencies are deliberately left unrelocated and unbundled: + - Netty. gRPC's Netty transport links to the Netty that Spark ships, including + the epoll and kqueue native transports that Unix domain sockets need. + Relocating Netty would change the native library names it loads. + - protobuf-java. The generated stubs and gRPC's protobuf marshallers must use + the same protobuf classes as the spark-udf-worker-proto messages, which are + not part of this jar. + + The relocations must stay consistent with UDFWorkerRuntime in + project/SparkBuild.scala. + --> + <dependencies> + <dependency> + <groupId>org.apache.spark</groupId> + <artifactId>spark-udf-worker-grpc_${scala.binary.version}</artifactId> + <version>${project.version}</version> + </dependency> + <!-- Bundled and relocated below; the parent manages both as provided. --> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + <scope>compile</scope> + </dependency> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>failureaccess</artifactId> + <scope>compile</scope> + </dependency> + <!-- + The jar links to classes it does not bundle: the UDF worker proto and core + modules, Netty and protobuf-java. Spark provides them, and declaring Spark core Review Comment: **Nit (P3):** This says Spark provides protobuf-java, but a Maven-built distribution does not ship unrelocated protobuf-java. It is managed as `provided` in the root pom, it is absent from `dev/deps/spark-deps-hadoop-3-hive-2.3`, and spark-core only bundles a relocated copy. The PR description calls this out as a known gap. The same claim is in `udf/worker/grpc/pom.xml` ("leaves Netty and protobuf-java to the rest of the Spark classpath"). Could these comments say that unrelocated protobuf-java is currently present only in sbt-built jars directories, and that this jar's protobuf relocation must stay identical to `spark-udf-worker-proto`'s? That coupling is recorded only in the PR description right now, and the planned protobuf follow-up will need to find it here. **Verification:** - **Inspection:** The comments' classpath claims agree with what the Maven assembly ships (no unrelocated protobuf-java) and state the proto/runtime relocation coupling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
