This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bb5bc50a203f [SPARK-60773][CONNECT] Make server-side Maven resolution
opt-in
bb5bc50a203f is described below
commit bb5bc50a203faf485cf3df71dfc0b765c45bae37
Author: qianlan717 <[email protected]>
AuthorDate: Thu Nov 5 07:06:43 2026 +0900
[SPARK-60773][CONNECT] Make server-side Maven resolution opt-in
### What changes were proposed in this pull request?
Add `spark.connect.session.serverSideMavenResolution.enabled`, defaulting
to `false`.
The Spark Connect JVM client uses server-side Maven resolution only when
this session setting
is enabled and the server advertises support. URIs with `repos` remain
client-resolved.
This is a routing preference, not server-side enforcement.
Client resolution uses the session's `spark.jars.ivySettings` when the
server exposes its value
and the file is readable locally. Configuration and file checks run on each
client-resolving
call. Missing or unavailable settings retain the built-in defaults;
unusable paths produce a
warning without printing the path. Server redaction is unchanged.
### Why are the changes needed?
Server capability should describe support without automatically changing
dependency resolution
for every session. Client resolution should honor locally available
repository settings.
JIRA: https://issues.apache.org/jira/browse/SPARK-60773
### Does this PR introduce _any_ user-facing change?
Yes, relative to unreleased `master`: Spark Connect JVM Ivy resolution
defaults to the client.
Sessions can opt in to server resolution with:
```scala
spark.conf.set("spark.connect.session.serverSideMavenResolution.enabled",
"true")
```
Client-resolved dependencies use a readable session-provided Ivy settings
file. Spark Classic
is unaffected by the routing setting.
### How was this patch tested?
`ArtifactSuite`: all 34 tests passed, covering routing, runtime config
changes, Ivy settings
substitution and URI escaping, file creation/removal, unavailable config
reads, propagated RPC
failures, and single/batch uploads.
```text
build/sbt 'connect-client-jvm/testOnly
org.apache.spark.sql.connect.client.ArtifactSuite'
```
Scalafmt validation passed for SQL API and all four Connect modules checked
by CI.
`git diff --check`, changed-source line-length checks, and ASCII checks
passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5 and GPT-6)
Closes #59982 from qianlan717/serverside-resolution.
Authored-by: qianlan717 <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/internal/config/package.scala | 3 +-
docs/configuration.md | 16 +-
.../main/scala/org/apache/spark/sql/Artifact.scala | 9 +-
.../scala/org/apache/spark/sql/SparkSession.scala | 21 +-
.../spark/sql/connect/client/ArtifactSuite.scala | 349 ++++++++++++++++++++-
.../connect/client/SparkConnectClientSuite.scala | 4 +
.../spark/sql/connect/client/ArtifactManager.scala | 28 +-
.../sql/connect/client/SparkConnectClient.scala | 69 +++-
.../sql/connect/common/config/ConnectCommon.scala | 2 +
.../apache/spark/sql/connect/config/Connect.scala | 13 +
.../service/SparkConnectConfigHandlerSuite.scala | 8 +
11 files changed, 480 insertions(+), 42 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index f1e5828e5899..bcd1a13fea5b 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -3087,7 +3087,8 @@ package object config {
"of the built-in defaults, such as maven central. " +
"For spark.jars.packages, additional repositories from
spark.jars.repositories will " +
"also be included. " +
- "Client-resolved Spark Connect Ivy URIs do not use this setting. " +
+ "Client-resolved Spark Connect Ivy URIs use this setting only when the
settings file is " +
+ "readable on the client. " +
"The spark-submit --repositories option applies to submission-time
resolution. " +
"Useful for allowing Spark to resolve artifacts from behind a firewall
" +
"e.g. via an in-house artifact server like Artifactory. " +
diff --git a/docs/configuration.md b/docs/configuration.md
index 8c87ebede11d..9fab8e62b802 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1105,10 +1105,10 @@ Apart from these, the following properties are also
available, and may be useful
For <code>spark.jars.packages</code>, additional repositories from
<code>spark.jars.repositories</code> will also be included.
The <code>spark-submit --repositories</code> option applies to
submission-time resolution.
- In Spark Connect, Ivy URIs with a <code>repos</code> query parameter or
sent to a server without
- server-side Maven resolution are resolved by the client and do not use
this setting. Other Ivy
- URIs are resolved by the server, so Maven and Ivy repositories local to
the client, such as
- <code>~/.m2/repository</code> and <code>~/.ivy2.5.2/local</code>, are not
searched.
+ Spark Connect JVM clients use the server session's value only if Config
RPC exposes it and
+ the file is readable locally; otherwise, they use the built-in defaults.
+ Enable
<code>spark.connect.session.serverSideMavenResolution.enabled</code> to resolve
Ivy URIs
+ without <code>repos</code> on a supporting server.
Useful for allowing Spark to resolve artifacts from behind a firewall e.g.
via an in-house
artifact server like Artifactory. Details on the settings file format can
be
found at <a
href="http://ant.apache.org/ivy/history/latest-milestone/settings.html">Settings
Files</a>.
@@ -3896,6 +3896,14 @@ Command types in proto.</td>
<td>The interval at which the progress of a query is reported to the client.
If the value is set to a negative value the progress reports will be
disabled.</td>
<td>4.0.0</td>
</tr>
+<tr>
+ <td><code>spark.connect.session.serverSideMavenResolution.enabled</code></td>
+ <td>
+ false
+ </td>
+ <td>When true, the Spark Connect JVM client resolves eligible Maven
dependencies on the server when it advertises support. Dependencies that
specify repositories are always resolved on the client. When false, the JVM
client resolves all Maven dependencies locally. This is a client routing
preference, not a server-side enforcement setting.</td>
+ <td>4.4.0</td>
+</tr>
</table>
### Security
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/Artifact.scala
b/sql/api/src/main/scala/org/apache/spark/sql/Artifact.scala
index e4fb52e92839..db80292fc6a7 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/Artifact.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/Artifact.scala
@@ -94,7 +94,9 @@ private[sql] object Artifact {
newArtifact(CACHE_PREFIX, "", Paths.get(id), storage)
}
- def newIvyArtifacts(uri: URI): Seq[Artifact] = {
+ def newIvyArtifacts(uri: URI): Seq[Artifact] = newIvyArtifacts(uri, None)
+
+ def newIvyArtifacts(uri: URI, ivySettingsPath: Option[String]):
Seq[Artifact] = {
implicit val printStream: PrintStream = System.err
val authority = uri.getAuthority
@@ -118,7 +120,10 @@ private[sql] object Artifact {
Nil
}
- val ivySettings = MavenUtils.buildIvySettings(Some(repos), None)
+ val ivySettings = ivySettingsPath match {
+ case Some(path) => MavenUtils.loadIvySettings(path, Some(repos), None)
+ case None => MavenUtils.buildIvySettings(Some(repos), None)
+ }
val jars = MavenUtils.resolveMavenCoordinates(
authority,
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
b/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
index 7f9f403b8983..5d51dab6811b 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -562,11 +562,10 @@ abstract class SparkSession extends Serializable with
Closeable {
* Add a single artifact to the current session.
*
* Currently it supports local files with extensions .jar and .class and
Apache Ivy URIs. Ivy
- * URIs are resolved using `spark.jars.ivySettings` when it is configured.
When using Spark
- * Connect, Ivy URIs with repositories in the `repos` query parameter, or
sent to servers that
- * do not support server-side Maven resolution, are resolved by the client
without the server's
- * Ivy settings. Otherwise, Ivy URIs are resolved by the server, which does
not search Maven or
- * Ivy repositories local to the client.
+ * URIs use `spark.jars.ivySettings` when configured. Spark Connect resolves
them on the client
+ * by default, using the session's settings file if its path is exposed and
readable locally.
+ * Enable `spark.connect.session.serverSideMavenResolution.enabled` to
resolve URIs without
+ * `repos` on a supporting server.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
* .sparkpkg, which make the native data sources that they contain available
to the session. It
@@ -584,11 +583,7 @@ abstract class SparkSession extends Serializable with
Closeable {
* Add a single artifact to the current session.
*
* Currently it supports local files with extensions .jar and .class and
Apache Ivy URIs. Ivy
- * URIs are resolved using `spark.jars.ivySettings` when it is configured.
When using Spark
- * Connect, Ivy URIs with repositories in the `repos` query parameter, or
sent to servers that
- * do not support server-side Maven resolution, are resolved by the client
without the server's
- * Ivy settings. Otherwise, Ivy URIs are resolved by the server, which does
not search Maven or
- * Ivy repositories local to the client.
+ * resolution follows the same settings as `addArtifact(String)`.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
* .sparkpkg, which make the native data sources that they contain available
to the session. It
@@ -657,11 +652,7 @@ abstract class SparkSession extends Serializable with
Closeable {
* Add one or more artifacts to the session.
*
* Currently it supports local files with extensions .jar and .class and
Apache Ivy URIs. Ivy
- * URIs are resolved using `spark.jars.ivySettings` when it is configured.
When using Spark
- * Connect, Ivy URIs with repositories in the `repos` query parameter, or
sent to servers that
- * do not support server-side Maven resolution, are resolved by the client
without the server's
- * Ivy settings. Otherwise, Ivy URIs are resolved by the server, which does
not search Maven or
- * Ivy repositories local to the client.
+ * resolution follows the same settings as `addArtifact(String)`.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
* .sparkpkg, which make the native data sources that they contain available
to the session. A
diff --git
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/ArtifactSuite.scala
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/ArtifactSuite.scala
index d4db6958319c..c340ea0ee3a9 100644
---
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/ArtifactSuite.scala
+++
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/ArtifactSuite.scala
@@ -26,15 +26,19 @@ import java.util.zip.CRC32
import scala.jdk.CollectionConverters._
import com.google.protobuf.ByteString
-import io.grpc.{ManagedChannel, Server, Status}
+import com.google.rpc.ErrorInfo
+import io.grpc.{ManagedChannel, Server, Status, StatusRuntimeException}
import io.grpc.inprocess.{InProcessChannelBuilder, InProcessServerBuilder}
+import io.grpc.protobuf.StatusProto
+import org.apache.spark.SparkException
import org.apache.spark.connect.proto.AddArtifactsRequest
import org.apache.spark.network.util.JavaUtils.sha256Hex
import org.apache.spark.sql.Artifact
import org.apache.spark.sql.connect.client.SparkConnectClient.Configuration
+import org.apache.spark.sql.connect.common.config.ConnectCommon
import org.apache.spark.sql.connect.test.ConnectFunSuite
-import org.apache.spark.util.{IvyTestUtils, SparkFileUtils}
+import org.apache.spark.util.{IvyTestUtils, MavenUtils, SparkFileUtils}
import org.apache.spark.util.MavenUtils.MavenCoordinate
class ArtifactSuite extends ConnectFunSuite {
@@ -172,6 +176,27 @@ class ArtifactSuite extends ConnectFunSuite {
assert(artifactChunk.getCrc == expectedCrc)
}
+ // Include a space and a literal percent sequence to catch incorrect URI
encoding.
+ // The artifact is available only in the repository referenced by these
settings.
+ private def writeClientIvySettings(
+ main: MavenCoordinate,
+ directory: String = "client settings %20"): Path = {
+ val dir = Files.createDirectories(ivyHome.resolve(directory))
+ val repo = IvyTestUtils.createLocalRepositoryForTests(
+ main,
+ dependencies = None,
+ rootDir = Some(Files.createTempDirectory(ivyHome,
"client-repository").toFile))
+ Files.writeString(
+ dir.resolve("ivysettings.xml"),
+ s"""<ivysettings>
+ | <settings defaultResolver="client"/>
+ | <resolvers>
+ | <ibiblio name="client" m2compatible="true" root="${repo.toURI}"/>
+ | </resolvers>
+ |</ivysettings>""".stripMargin)
+ dir
+ }
+
private def singleChunkArtifactTest(path: String): Unit = {
test(s"Single Chunk Artifact - $path") {
val artifactPath = artifactFilePath.resolve(path)
@@ -314,8 +339,9 @@ class ArtifactSuite extends ConnectFunSuite {
}
}
- test("Spark Connect uses server-side Maven resolution when advertised") {
+ test("Spark Connect uses server-side Maven resolution when enabled and
advertised") {
service.serverCapabilities =
Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+
service.setSparkConfig(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
" true ")
client = new SparkConnectClient(Configuration(), channel)
val ivyUri = URI.create("ivy://my.artifactsuite.lib:mylib:0.1")
@@ -325,12 +351,310 @@ class ArtifactSuite extends ConnectFunSuite {
assert(requests.size == 1)
assert(requests.head.getBatch.getEntriesCount == 1)
assert(requests.head.getBatch.getEntries(0).getMavenDependency.getUri ==
ivyUri.toString)
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(configRequests.size == 1)
+ assert(
+ configRequests.head.getOperation.getGetOption.getKeysList.asScala ==
+ Seq(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED))
+ }
+
+ Seq(
+ "raw path" -> ((dir: Path) => dir.toString),
+ "escaped file URI" -> ((dir: Path) =>
dir.toUri.toString.stripSuffix("/"))).foreach {
+ case (label, settingsDir) =>
+ Seq("addArtifact", "addArtifacts").foreach { entryPoint =>
+ test(
+ s"Spark Connect client resolution uses locally readable Ivy
settings: " +
+ s"$label, $entryPoint") {
+ val main = new MavenCoordinate("my.artifactsuite.clientsettings",
"mylib", "0.1")
+ val dir = writeClientIvySettings(main)
+ val settingsDirKey = "spark.test.artifact.clientIvySettingsDir"
+ service.serverCapabilities =
+ Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+ service.setSparkConfig(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ "false")
+ service.setSparkConfig(settingsDirKey, settingsDir(dir))
+ service.setSparkConfig(
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ s"$${$settingsDirKey}/ivysettings.xml")
+ client = new SparkConnectClient(Configuration(), channel)
+ val ivyUri = URI.create(s"ivy://${main.toString}")
+
+ if (entryPoint == "addArtifact") {
+ client.addArtifact(ivyUri)
+ } else {
+ client.addArtifacts(
+ Seq(
+ artifactFilePath.resolve("smallClassFile.class").toUri,
+ ivyUri,
+ artifactFilePath.resolve("smallJar.jar").toUri))
+ }
+
+ val names = service
+ .getAndClearLatestAddArtifactRequests()
+ .flatMap(_.getBatch.getArtifactsList.asScala)
+ .map(_.getName)
+ val ivyName = "my.artifactsuite.clientsettings_mylib-0.1"
+ if (entryPoint == "addArtifact") {
+ assert(names.size == 1 && names.head.contains(ivyName))
+ } else {
+ assert(names.size == 3)
+ assert(names(0) == "classes/smallClassFile.class")
+ assert(names(1).contains(ivyName))
+ assert(names(2) == "jars/smallJar.jar")
+ }
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
configRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ settingsDirKey))
+ }
+ }
+ }
+
+ Seq(false, true).foreach { initiallyPresent =>
+ val change = if (initiallyPresent) "removed" else "created"
+ test(s"Spark Connect rechecks Ivy settings when the file is $change") {
+ val custom = new MavenCoordinate("my.artifactsuite.customsettings",
"mylib", "0.1")
+ val default = new MavenCoordinate("my.artifactsuite.defaultsettings",
"mylib", "0.1")
+ val settingsFile =
writeClientIvySettings(custom).resolve("ivysettings.xml")
+ val settings = Files.readAllBytes(settingsFile)
+ if (!initiallyPresent) {
+ Files.delete(settingsFile)
+ }
+ service.setSparkConfig(MavenUtils.JAR_IVY_SETTING_PATH_KEY,
settingsFile.toString)
+ client = new SparkConnectClient(Configuration(), channel)
+
+ IvyTestUtils.withRepository(
+ default,
+ dependencies = None,
+ rootDir = Some(ivyHome.resolve("local").toFile),
+ useIvyLayout = true) { _ =>
+ val first = if (initiallyPresent) custom else default
+ val second = if (initiallyPresent) default else custom
+ client.addArtifact(URI.create(s"ivy://${first.toString}"))
+ if (initiallyPresent) {
+ Files.delete(settingsFile)
+ } else {
+ Files.write(settingsFile, settings)
+ }
+ client.addArtifacts(Seq(URI.create(s"ivy://${second.toString}")))
+
+ val names = service
+ .getAndClearLatestAddArtifactRequests()
+ .flatMap(_.getBatch.getArtifactsList.asScala)
+ .map(_.getName)
+ assert(names.size == 2)
+
assert(names.head.contains(s"${first.groupId}_${first.artifactId}-${first.version}"))
+
assert(names(1).contains(s"${second.groupId}_${second.artifactId}-${second.version}"))
+ assert(service.getAndClearLatestConfigRequests().size == 2)
+ }
+ }
+ }
+
+ test("Spark Connect rereads Ivy settings substitution for each call") {
+ val settingsDirKey = "spark.test.artifact.clientIvySettingsDir"
+ service.setSparkConfig(
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ s"$${$settingsDirKey}/ivysettings.xml")
+ client = new SparkConnectClient(Configuration(), channel)
+
+ Seq("first", "second").foreach { name =>
+ val main = new MavenCoordinate(s"my.artifactsuite.$name", "mylib", "0.1")
+ val dir = writeClientIvySettings(main, name)
+ service.setSparkConfig(settingsDirKey, dir.toString)
+
+ client.addArtifact(URI.create(s"ivy://${main.toString}"))
+
+ val requests = service.getAndClearLatestAddArtifactRequests()
+ assert(requests.flatMap(_.getBatch.getArtifactsList.asScala).exists {
artifact =>
+ artifact.getName.contains(s"my.artifactsuite.${name}_mylib-0.1")
+ })
+ assert(service.getAndClearLatestConfigRequests().size == 2)
+ }
+ }
+
+ Seq("SQL_CONF_NOT_FOUND", "CONFIG_NOT_AVAILABLE").foreach { condition =>
+ Seq(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ "spark.test.artifact.settingsDir").foreach { key =>
+ test(s"Spark Connect uses Maven defaults when $key is unavailable:
$condition") {
+ val main = new MavenCoordinate("my.artifactsuite.unavailableconfig",
"mylib", "0.1")
+ service.serverCapabilities =
+ Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+ service.setSparkConfig(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ "false")
+ if (key == "spark.test.artifact.settingsDir") {
+ service.setSparkConfig(MavenUtils.JAR_IVY_SETTING_PATH_KEY,
s"$${$key}/ivysettings.xml")
+ }
+ val info = ErrorInfo
+ .newBuilder()
+ .putMetadata("classes", """["org.apache.spark.SparkException"]""")
+ .putMetadata("errorClass", condition)
+ .putMetadata("sqlState", "42K0I")
+ .putMetadata("messageParameters", s"""{"sqlConf":"$key"}""")
+ .build()
+ val error = com.google.rpc.Status
+ .newBuilder()
+ .setCode(Status.Code.INTERNAL.value())
+ .setMessage("Configuration is unavailable")
+ .addDetails(com.google.protobuf.Any.pack(info))
+ .build()
+ service.setErrorToThrowOnConfig(key,
StatusProto.toStatusRuntimeException(error))
+ client = new SparkConnectClient(Configuration(), channel)
+
+ IvyTestUtils.withRepository(
+ main,
+ dependencies = None,
+ rootDir = Some(ivyHome.resolve("local").toFile),
+ useIvyLayout = true) { _ =>
+ client.addArtifact(URI.create(s"ivy://${main.toString}"))
+
+ val requests = service.getAndClearLatestAddArtifactRequests()
+ assert(requests.forall(_.getBatch.getEntriesCount == 0))
+ assert(requests.flatMap(_.getBatch.getArtifactsList.asScala).exists
{ artifact =>
+
artifact.getName.contains("my.artifactsuite.unavailableconfig_mylib-0.1")
+ })
+ }
+ }
+ }
+ }
+
+ Seq(Status.PERMISSION_DENIED, Status.UNAVAILABLE).foreach { status =>
+ test(s"Spark Connect propagates Maven config RPC failures:
${status.getCode}") {
+ service.serverCapabilities =
Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+ service.setErrorToThrowOnConfig(
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ status.asRuntimeException())
+ client = new SparkConnectClient(Configuration(retryPolicies = Nil),
channel)
+
+ val error = intercept[SparkException] {
+ client.addArtifact(URI.create("ivy://my.artifactsuite.lib:mylib:0.1"))
+ }
+ assert(
+ error.getCause.asInstanceOf[StatusRuntimeException].getStatus.getCode
== status.getCode)
+ assert(service.getAndClearLatestAddArtifactRequests().isEmpty)
+ }
+ }
+
+ test("Spark Connect batch applies Ivy settings only to client-resolved Ivy
URIs") {
+ val main = new MavenCoordinate("my.artifactsuite.clientsettings", "mylib",
"0.1")
+ val dir = writeClientIvySettings(main)
+ // The requested repository does not contain `main`, so resolution
succeeds only through the
+ // settings file.
+ val unrelatedRepo =
Files.createDirectories(ivyHome.resolve("unrelated-repository"))
+ service.serverCapabilities =
Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+
service.setSparkConfig(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
"true")
+ service.setSparkConfig(
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ dir.resolve("ivysettings.xml").toString)
+ client = new SparkConnectClient(Configuration(), channel)
+ val serverIvyUri = URI.create("ivy://my.artifactsuite.server:mylib:0.1")
+
+ client.addArtifacts(
+ Seq(
+ artifactFilePath.resolve("smallClassFile.class").toUri,
+ URI.create(s"ivy://${main.toString}?repos=${unrelatedRepo.toUri}"),
+ serverIvyUri,
+ artifactFilePath.resolve("smallJar.jar").toUri))
+
+ val requests = service.getAndClearLatestAddArtifactRequests()
+ assert(requests.size == 1)
+ val entries = requests.head.getBatch.getEntriesList.asScala
+ assert(entries.size == 4)
+ assert(entries(0).getArtifact.getName == "classes/smallClassFile.class")
+
assert(entries(1).getArtifact.getName.contains("my.artifactsuite.clientsettings_mylib-0.1"))
+ assert(entries(2).getMavenDependency.getUri == serverIvyUri.toString)
+ assert(entries(3).getArtifact.getName == "jars/smallJar.jar")
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(
+ configRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala)
==
+ Seq(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY))
+ }
+
+ test("Spark Connect defaults to client-side Maven resolution") {
+ val main = new MavenCoordinate("my.artifactsuite.defaultclient", "mylib",
"0.1")
+ IvyTestUtils.withRepository(
+ main,
+ dependencies = None,
+ rootDir = Some(ivyHome.resolve("local").toFile),
+ useIvyLayout = true) { _ =>
+ service.serverCapabilities =
Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+ client = new SparkConnectClient(Configuration(), channel)
+
+ client.addArtifact(URI.create(s"ivy://${main.toString}"))
+
+ val requests = service.getAndClearLatestAddArtifactRequests()
+ assert(requests.nonEmpty)
+ assert(requests.forall(_.getBatch.getEntriesCount == 0))
+ assert(requests.flatMap(_.getBatch.getArtifactsList.asScala).exists {
artifact =>
+ artifact.getName.contains("my.artifactsuite.defaultclient_mylib-0.1")
+ })
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
configRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY))
+ }
+ }
+
+ test("Spark Connect server-side Maven resolution follows the session
configuration") {
+ val main = new MavenCoordinate("my.artifactsuite.sessionconfig", "mylib",
"0.1")
+ IvyTestUtils.withRepository(
+ main,
+ dependencies = None,
+ rootDir = Some(ivyHome.resolve("local").toFile),
+ useIvyLayout = true) { _ =>
+ service.serverCapabilities =
Seq(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+
service.setSparkConfig(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
"false")
+ client = new SparkConnectClient(Configuration(), channel)
+ val ivyUri = URI.create(s"ivy://${main.toString}")
+
+ client.addArtifact(ivyUri)
+
+ val clientRequests = service.getAndClearLatestAddArtifactRequests()
+ assert(clientRequests.nonEmpty)
+ assert(clientRequests.forall(_.getBatch.getEntriesCount == 0))
+
assert(clientRequests.flatMap(_.getBatch.getArtifactsList.asScala).exists {
artifact =>
+ artifact.getName.contains("my.artifactsuite.sessionconfig_mylib-0.1")
+ })
+ val clientConfigRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
clientConfigRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(
+ ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY))
+
+
service.setSparkConfig(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
"true")
+ client.addArtifact(ivyUri)
+
+ val serverRequests = service.getAndClearLatestAddArtifactRequests()
+ assert(serverRequests.size == 1)
+ assert(serverRequests.head.getBatch.getEntriesCount == 1)
+ assert(
+ serverRequests.head.getBatch.getEntries(0).getMavenDependency.getUri
== ivyUri.toString)
+ val serverConfigRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
serverConfigRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED))
+ }
}
test("Spark Connect resolves requested Maven repositories on the client") {
val main = new MavenCoordinate("my.artifactsuite.clientcapability",
"mylib", "0.1")
IvyTestUtils.withRepository(main, None, None) { repo =>
service.errorToThrowOnAnalyze =
Some(Status.UNAVAILABLE.asRuntimeException())
+ service.setSparkConfig(
+ MavenUtils.JAR_IVY_SETTING_PATH_KEY,
+ ivyHome.resolve("missing-ivysettings.xml").toString)
client = new SparkConnectClient(Configuration(), channel)
client.addArtifact(URI.create(s"ivy://${main.toString}?repos=$repo"))
@@ -342,6 +666,10 @@ class ArtifactSuite extends ConnectFunSuite {
assert(batch.getArtifactsCount == 1)
assert(
batch.getArtifacts(0).getName.contains("my.artifactsuite.clientcapability_mylib-0.1"))
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
configRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(MavenUtils.JAR_IVY_SETTING_PATH_KEY))
}
}
@@ -358,12 +686,17 @@ class ArtifactSuite extends ConnectFunSuite {
assert(requests(1).getBatch.getEntries(0).getMavenDependency.getUri ==
otherUri.toString)
}
- test("Spark Connect falls back only when the Maven capability is absent") {
+ test("Spark Connect resolves Maven dependencies on the client without the
capability") {
val main = new MavenCoordinate("my.artifactsuite.fallback", "mylib", "0.1")
- IvyTestUtils.withRepository(main, None, None) { repo =>
+ IvyTestUtils.withRepository(
+ main,
+ dependencies = None,
+ rootDir = Some(ivyHome.resolve("local").toFile),
+ useIvyLayout = true) { _ =>
+
service.setSparkConfig(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED,
"true")
client = new SparkConnectClient(Configuration(), channel)
- client.addArtifact(URI.create(s"ivy://${main.toString}?repos=$repo"))
+ client.addArtifact(URI.create(s"ivy://${main.toString}"))
val requests = service.getAndClearLatestAddArtifactRequests()
assert(requests.nonEmpty)
@@ -371,6 +704,10 @@ class ArtifactSuite extends ConnectFunSuite {
assert(requests.flatMap(_.getBatch.getArtifactsList.asScala).exists {
artifact =>
artifact.getName.contains("my.artifactsuite.fallback_mylib-0.1")
})
+ val configRequests = service.getAndClearLatestConfigRequests()
+ assert(
+
configRequests.flatMap(_.getOperation.getGetOption.getKeysList.asScala) ==
+ Seq(MavenUtils.JAR_IVY_SETTING_PATH_KEY))
}
service.errorToThrowOnAnalyze =
Some(Status.UNAVAILABLE.asRuntimeException())
diff --git
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala
index 7e99a6e06328..fab2e073887f 100644
---
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala
+++
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala
@@ -1439,6 +1439,10 @@ class DummySparkConnectService() extends
SparkConnectServiceGrpc.SparkConnectSer
errorToThrowOnConfig = errorToThrowOnConfig + (key -> error)
}
+ private[sql] def setSparkConfig(key: String, value: String): Unit =
synchronized {
+ sparkConfigs.put(key, value)
+ }
+
private[sql] def getAndClearLatestInputPlan(): proto.Plan = synchronized {
val plan = inputPlan
inputPlan = null
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala
index 08bc4f2a8741..c829755e8baf 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala
@@ -96,7 +96,7 @@ class ArtifactManager(
addArtifact(SparkFileUtils.resolveURI(path))
}
- private def parseArtifacts(uri: URI): Seq[Artifact] = {
+ private def parseArtifacts(uri: URI, ivySettingsPath: Option[String]):
Seq[Artifact] = {
// Currently only local files with extensions .jar and .class are
supported.
uri.getScheme match {
case "file" =>
@@ -108,7 +108,7 @@ class ArtifactManager(
Seq[Artifact](artifact)
case "ivy" =>
- newIvyArtifacts(uri)
+ newIvyArtifacts(uri, ivySettingsPath)
case other =>
throw new UnsupportedOperationException(s"Unsupported scheme: $other")
@@ -128,10 +128,14 @@ class ArtifactManager(
*
* Currently it supports local files with extensions .jar and .class and
Apache Ivy URIs
*/
- def addArtifact(uri: URI): Unit = addArtifact(uri, serverSideMavenArtifacts
= false)
+ def addArtifact(uri: URI): Unit =
+ addArtifact(uri, serverSideMavenArtifacts = false, ivySettingsPath = None)
- private[client] def addArtifact(uri: URI, serverSideMavenArtifacts:
Boolean): Unit =
- addArtifacts(Seq(uri), serverSideMavenArtifacts)
+ private[client] def addArtifact(
+ uri: URI,
+ serverSideMavenArtifacts: Boolean,
+ ivySettingsPath: Option[String]): Unit =
+ addArtifacts(Seq(uri), serverSideMavenArtifacts, ivySettingsPath)
/**
* Add a single in-memory artifact to the session while preserving the
directory structure
@@ -188,21 +192,27 @@ class ArtifactManager(
* Currently it supports local files with extensions .jar and .class and
Apache Ivy URIs
*/
def addArtifacts(uris: Seq[URI]): Unit =
- addArtifacts(uris, serverSideMavenArtifacts = false)
+ addArtifacts(uris, serverSideMavenArtifacts = false, ivySettingsPath =
None)
- private[client] def addArtifacts(uris: Seq[URI], serverSideMavenArtifacts:
Boolean): Unit = {
+ private[client] def addArtifacts(uris: Seq[URI], serverSideMavenArtifacts:
Boolean): Unit =
+ addArtifacts(uris, serverSideMavenArtifacts, ivySettingsPath = None)
+
+ private[client] def addArtifacts(
+ uris: Seq[URI],
+ serverSideMavenArtifacts: Boolean,
+ ivySettingsPath: Option[String]): Unit = {
if (serverSideMavenArtifacts) {
val entries = uris.flatMap { uri =>
uri.getScheme match {
// Resolve explicit repositories on the client so arbitrary
repositories are never
// accessed from the server.
case "ivy" if !hasRequestedRepositories(uri) => MavenDependency(uri)
:: Nil
- case _ => parseArtifacts(uri).map(UploadedArtifact)
+ case _ => parseArtifacts(uri, ivySettingsPath).map(UploadedArtifact)
}
}
addArtifactEntries(entries, useOrderedEntries = true)
} else {
- addArtifacts(uris.flatMap(parseArtifacts))
+ addArtifacts(uris.flatMap(uri => parseArtifacts(uri, ivySettingsPath)))
}
}
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
index e60f4a83b694..d47572fe156e 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.connect.client
import java.net.URI
import java.nio.charset.StandardCharsets.UTF_8
+import java.nio.file.{Files, Paths}
import java.util.{Base64, Locale, UUID}
import java.util.concurrent.{Executor, TimeUnit}
@@ -39,11 +40,12 @@ import org.apache.spark.connect.proto
import org.apache.spark.connect.proto.UserContext
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.{ERROR, RATIO, SIZE, TIME}
+import org.apache.spark.internal.config.ConfigReader
import org.apache.spark.sql.connect.RuntimeConfig
import org.apache.spark.sql.connect.common.ProtoUtils
import org.apache.spark.sql.connect.common.config.ConnectCommon
import org.apache.spark.sql.util.CloseableIterator
-import org.apache.spark.util.{SparkFileUtils, SparkSystemUtils}
+import org.apache.spark.util.{MavenUtils, SparkFileUtils, SparkSystemUtils}
/**
* Conceptually the remote spark session that communicates with the server.
@@ -104,6 +106,10 @@ private[sql] class SparkConnectClient(
private def supportsServerSideMavenArtifacts: Boolean =
serverCapabilities.contains(SparkConnectClient.SERVER_SIDE_MAVEN_ARTIFACTS_CAPABILITY)
+ private def serverSideMavenResolutionEnabled: Boolean =
+
getArtifactConfigOption(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED)
+ .exists(_.trim.toBoolean)
+
// Cached plan compression options.
private var _planCompressionOptions: Option[Option[PlanCompressionOptions]]
= None
@@ -601,6 +607,55 @@ private[sql] class SparkConnectClient(
*/
def preferredArrowChunkSize: Option[Int] =
configuration.preferredArrowChunkSize
+ private def getArtifactConfigOption(key: String): Option[String] = {
+ try {
+ conf.getOption(key)
+ } catch {
+ case _: NoSuchElementException => None
+ case e: SparkThrowable
+ if e.getCondition == "SQL_CONF_NOT_FOUND"
+ || e.getCondition == "CONFIG_NOT_AVAILABLE" =>
+ None
+ }
+ }
+
+ private def localIvySettingsPath: Option[String] =
+ new ConfigReader((key: String) => getArtifactConfigOption(key))
+ .get(MavenUtils.JAR_IVY_SETTING_PATH_KEY)
+ .filter(_.trim.nonEmpty)
+ .flatMap { settingsPath =>
+ val localPath =
+ try {
+ val settingsUri = SparkFileUtils.resolveURI(settingsPath)
+ if (settingsUri.getScheme == "file") {
+ val path = Paths.get(settingsUri)
+ Option.when(Files.isRegularFile(path) && Files.isReadable(path))(
+ settingsUri.toString)
+ } else {
+ None
+ }
+ } catch {
+ case NonFatal(_) => None
+ }
+ if (localPath.isEmpty) {
+ logWarning(
+ "Client-side Maven resolution cannot use the session's
spark.jars.ivySettings; " +
+ "using default Ivy settings. Provide a readable local file or
enable " +
+ "spark.connect.session.serverSideMavenResolution.enabled for
URIs without repos.")
+ }
+ localPath
+ }
+
+ private def ivySettingsPathForClientResolution(
+ uris: Seq[URI],
+ serverSideMavenArtifacts: Boolean): Option[String] = {
+ val resolvesIvyOnClient = uris.exists { uri =>
+ uri.getScheme == "ivy" &&
+ (!serverSideMavenArtifacts ||
!artifactManager.isServerSideMavenCandidate(uri))
+ }
+ if (resolvesIvyOnClient) localIvySettingsPath else None
+ }
+
/**
* Add a single artifact to the client session.
*
@@ -615,8 +670,10 @@ private[sql] class SparkConnectClient(
*/
def addArtifact(uri: URI): Unit = {
val serverSide = artifactManager.isServerSideMavenCandidate(uri) &&
- supportsServerSideMavenArtifacts
- artifactManager.addArtifact(uri, serverSide)
+ supportsServerSideMavenArtifacts &&
+ serverSideMavenResolutionEnabled
+ val ivySettingsPath = ivySettingsPathForClientResolution(Seq(uri),
serverSide)
+ artifactManager.addArtifact(uri, serverSide, ivySettingsPath)
}
/**
@@ -663,8 +720,10 @@ private[sql] class SparkConnectClient(
*/
def addArtifacts(uri: Seq[URI]): Unit = {
val serverSide = uri.exists(artifactManager.isServerSideMavenCandidate) &&
- supportsServerSideMavenArtifacts
- artifactManager.addArtifacts(uri, serverSide)
+ supportsServerSideMavenArtifacts &&
+ serverSideMavenResolutionEnabled
+ val ivySettingsPath = ivySettingsPathForClientResolution(uri, serverSide)
+ artifactManager.addArtifacts(uri, serverSide, ivySettingsPath)
}
/**
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala
index ba2aa09435d4..304c40a936f4 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala
@@ -21,6 +21,8 @@ private[sql] object ConnectCommon {
val CONNECT_GRPC_PORT_MAX_RETRIES: Int = 0
val CONNECT_GRPC_MAX_MESSAGE_SIZE: Int = 128 * 1024 * 1024
val CONNECT_GRPC_MARSHALLER_RECURSION_LIMIT: Int = 1024
+ val CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED: String =
+ "spark.connect.session.serverSideMavenResolution.enabled"
// Detects a silently-dead connection (e.g. a NAT gateway/load balancer
dropping an idle
// connection mapping without sending a TCP RST/FIN) via gRPC/HTTP2
keepalive PINGs, so a
// blocked RPC surfaces as UNAVAILABLE instead of hanging forever. Guarded by
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
index 9a9d15d197bd..213f652978dd 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
@@ -20,6 +20,7 @@ import java.util.Locale
import java.util.concurrent.TimeUnit
import org.apache.spark.SparkEnv
+import org.apache.spark.internal.config.ConfigBindingPolicy
import org.apache.spark.network.util.ByteUnit
import org.apache.spark.sql.connect.common.config.ConnectCommon
import org.apache.spark.sql.internal.SQLConf.buildConf
@@ -343,6 +344,18 @@ object Connect {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("2s")
+ val CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED =
+ buildConf(ConnectCommon.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED)
+ .doc(
+ "When true, the Spark Connect JVM client resolves eligible Maven
dependencies on the " +
+ "server when it advertises support. Dependencies that specify
repositories are always " +
+ "resolved on the client. When false, the JVM client resolves all
Maven dependencies " +
+ "locally. This is a client routing preference, not a server-side
enforcement setting.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
val CONNECT_SESSION_PLAN_CACHE_SIZE =
buildStaticConf("spark.connect.session.planCache.maxSize")
.doc("Sets the maximum number of cached resolved logical plans in Spark
Connect Session." +
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectConfigHandlerSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectConfigHandlerSuite.scala
index c17441b4119c..8055a61e0353 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectConfigHandlerSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectConfigHandlerSuite.scala
@@ -29,6 +29,7 @@ import org.apache.spark.{SparkEnv, SparkException,
SparkNoSuchElementException}
import org.apache.spark.connect.proto
import org.apache.spark.internal.config.{ConfigEntry, SECRET_REDACTION_PATTERN}
import org.apache.spark.sql.connect.SparkConnectTestUtils
+import org.apache.spark.sql.connect.config.Connect
import org.apache.spark.sql.execution.python.PythonWorkerEnvironment
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.sql.test.SharedSparkSession
@@ -129,6 +130,13 @@ class SparkConnectConfigHandlerSuite extends
SharedSparkSession {
}
}
+ test("GetOption returns the registered default for server-side Maven
resolution") {
+ val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
+ val key = Connect.CONNECT_SERVER_SIDE_MAVEN_RESOLUTION_ENABLED.key
+ val returned = pairs(sendConfigRequest(sessionHolder,
_.getGetOptionBuilder.addKeys(key)))
+ assert(returned(key) === Some("false"))
+ }
+
test("GetWithDefault returns the caller's default for a redacted key") {
val sessionHolder = SparkConnectTestUtils.createDummySessionHolder(spark)
withSQLConf(secretKey -> secretValue) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]