[
https://issues.apache.org/jira/browse/SPARK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260096#comment-15260096
]
Sean Owen commented on SPARK-14956:
-----------------------------------
I think the issues are: Spark can accomplish the same things by shading anyway,
and this doesn't solve problems with other runtime dependencies like Hadoop.
> Spark dependencies conflicts
> ----------------------------
>
> Key: SPARK-14956
> URL: https://issues.apache.org/jira/browse/SPARK-14956
> Project: Spark
> Issue Type: Brainstorming
> Components: Spark Core
> Affects Versions: 1.6.1
> Reporter: Michel Lemay
> Priority: Minor
> Labels: dependencies
>
> Since Spark lives with ages-old dependencies, it's all too often a problem
> that we must downgrade one of our dependencies just to make it "not explode"
> in Spark. And most importantly, this is something we only encounter at
> runtime and that makes it an even worse problem.
> So the usual solution when we depend directly on a package (but not one of
> our transitive dependencies) is to relocate with a shade plugin like this:
> {code}
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-shade-plugin</artifactId>
> ...
> <configuration>
> <relocations>
> <relocation>
> <pattern>scopt</pattern>
> <shadedPattern>hidden.scopt</shadedPattern>
> </relocation>
> {code}
> Other times, we must exclude transitive dependencies like that:
> {code}
> <dependency>
> <groupId>com.twitter.penguin</groupId>
> <artifactId>korean-text</artifactId>
> <version>4.1.2</version>
> <exclusions>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-nop</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> {code}
> Everything related to Guava, Log4j, Databind, scopt and even Java8 fall into
> this category.
> I wonder if that would be possible to use OSGi style of plugins when running
> code inside Spark.. That would shield us from all theses things.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
