Qianlan Chen created SPARK-59553:
------------------------------------
Summary: Make SparkSession.addArtifact honor spark.jars.ivySettings
Key: SPARK-59553
URL: https://issues.apache.org/jira/browse/SPARK-59553
Project: Spark
Issue Type: New Feature
Components: Connect, SQL
Affects Versions: 4.2.0
Environment: SparkSession.addArtifact supports resolving Maven
coordinates through ivy:// URIs, but it does not currently honor
`spark.jars.ivySettings`.
For classic Spark, Artifact.newIvyArtifacts always builds the default Ivy
settings. For Spark Connect, Ivy coordinates are resolved by the client and the
resulting JARs are uploaded to the server. Consequently, the client's Ivy
configuration and network access are used instead of the server's
spark.jars.ivySettings.
This differs from other Maven-resolution surfaces such as spark-submit and
SparkContext.addJar. It also prevents Spark Connect from using repositories,
mirrors, credentials, or repository policies configured only on the server.
Proposed changes:
* Make SparkSession.addArtifact and addArtifacts use spark.jars.ivySettings
when resolving ivy:// coordinates.
* Move Spark Connect Ivy resolution from the client to the server:
* Add a protocol representation for an unresolved Ivy coordinate.
* Have the client send the coordinate instead of resolving and uploading
its JARs.
* Resolve the coordinate on the server and register the resulting JARs
through the existing artifact-management path.
* Preserve URI ordering and deduplicate shared transitive dependencies.
* Apply the behavior to both addArtifact and addArtifacts.
Compatibility and fallback:
* When spark.jars.ivySettings is configured, load it and use it for
resolution.
* When it is not configured, retain the existing default Ivy resolver
behavior.
* A client should use server-side resolution only when the server advertises
support. With an older server, retain the existing client-side resolution and
upload behavior.
* Older clients must continue to work with newer servers.
* Preserve the existing handling of per-coordinate ?repos= repositories. If
these cannot safely be resolved by the server, fall back to the existing
client-side path.
* A failure to load or use an explicitly configured spark.jars.ivySettings
should be reported rather than silently bypassing it and resolving from default
repositories.
Reporter: Qianlan Chen
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
