Josh Rosen created SPARK-36835:
----------------------------------
Summary: Spark 3.2.0 POMs are no longer "dependency reduced"
Key: SPARK-36835
URL: https://issues.apache.org/jira/browse/SPARK-36835
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 3.2.0
Reporter: Josh Rosen
It looks like Spark 3.2.0's POMs are no longer "dependency reduced". As a
result, applications may pull in additional unnecessary dependencies when
depending on Spark.
Spark uses the Maven Shade plugin to create effective POMs and to bundle shaded
versions of certain libraries with Spark (namely, Jetty, Guava, and JPPML). [By
default|https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#createDependencyReducedPom],
the Maven Shade plugin generates simplified POMs which remove dependencies on
artifacts that have been shaded.
SPARK-33212 /
[b6f46ca29742029efea2790af7fdefbc2fcf52de|https://github.com/apache/spark/commit/b6f46ca29742029efea2790af7fdefbc2fcf52de]
changed the configuration of the Maven Shade plugin, setting
{{createDependencyReducedPom}} to {{false}}.
As a result, the generated POMs now include compile-scope dependencies on the
shaded libraries. For example, compare the {{org.eclipse.jetty}} dependencies
in:
* Spark 3.1.2:
[https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.2/spark-core_2.12-3.1.2.pom]
* Spark 3.2.0 RC2:
[https://repository.apache.org/content/repositories/orgapachespark-1390/org/apache/spark/spark-core_2.12/3.2.0/spark-core_2.12-3.2.0.pom]
I think we should revert back to generating "dependency reduced" POMs to ensure
that Spark declares a proper set of dependencies and to avoid "unknown unknown"
consequences of changing our generated POM format.
/cc [~csun]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
