[
https://issues.apache.org/jira/browse/SPARK-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183833#comment-15183833
]
Marcelo Vanzin edited comment on SPARK-13575 at 3/9/16 10:44 PM:
-----------------------------------------------------------------
After some playing around, I'm starting to feel that it's better to leave these
as is.
Removing the streaming assemblies makes it harder for people who don't want to
use "--packages" to distribute them, because now instead of one jar they have
to download a bunch of jars.
Adding to that, Maven's dependency mechanism is sort of broken when you have
test jars (such as the shared spark-core test classes used all over Spark, e.g.
"SparkFunSuite"). Transitive dependencies of the test jars end up being
promoted to "compile" scope, meaning you still need explicit excludes like we
have today in the flume-assembly module to clean things up. Fixing that
properly would mean getting rid of the spark-core tests jar - which is not
entirely a bad idea, just more work than I'm willing to put into this right
now. And you may still have to do some manual exclusions...
Since these assemblies are rather small and well-curated currently, I'll just
leave this task in the backburner in case people think it might still be
beneficial to do this.
was (Author: vanzin):
After some playing around, I'm starting to feel that it's better to leave these
as is.
Removing the streaming assemblies makes it harder for people who don't want to
use "--packages" to distribute them, because now instead of one jar they have
to download a bunch of jars.
Adding to that, Maven's dependency mechanism is sort of broken when you have
test jars (such as the shared spark-core test classes used all over Spark, e.g.
"SparkFunSuite"). Transitive dependencies of the test jars end up being
promoted to "compile" scope, meaning you still need explicit excludes like we
have today in the flume-assembly module to clean things up. Fixing that
properly would mean getting rid of the spark-core tests jar - which is not
entirely a bad idea, just more work than I'm willing to put into this right
now. And you may still have to do some manual exclusions...
Since these assemblies are rather small and well-curated currently, I'll just
leave this task in the backburner in case people thing it might still be
beneficial to do this.
> Remove streaming backends' assemblies
> -------------------------------------
>
> Key: SPARK-13575
> URL: https://issues.apache.org/jira/browse/SPARK-13575
> Project: Spark
> Issue Type: Sub-task
> Components: Build, Streaming
> Reporter: Marcelo Vanzin
>
> See parent bug for details. This task covers removing assemblies for
> streaming backends.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
