[
https://issues.apache.org/jira/browse/SPARK-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125147#comment-15125147
]
Josh Rosen commented on SPARK-11157:
------------------------------------
I'd love to start making progress towards removing the assemblies. Before we do
so, though, I think there are a few subtasks / obstacles that we need to clear
first:
- First, I think we should just completely remove the assembly rather than
giving both assembly and non-assembly options. Every additional option that we
provide / support adds lots of maintenance burden and it would be nice to
standardize on a single supported distribution technique.
- Prior to removing the assemblies, it would be great if we could reconfigure
our tests to not depend on the full assembly JAR in order to run. We already
have {{SPARK_PREPEND_CLASSPATH}} today, so this might be as simple as making
that behavior the default and reconfiguring our test scripts to skip the
assembly step.
- Building up a {{-classpath}} argument that lists hundreds of JARs is going to
be a debugging nightmare (lots of tools truncate process arguments past some
limit, etc.), so it would be good to investigate other techniques that we can
use to pass the classpath to {{java}} without bloating the CLI (maybe using an
environment variable or some file or something?).
- This is going to require changes to Launcher, shell scripts, and a few other
places; it would be good to scope out these changes to estimate how much work
it's going to be.
[~vanzin], are there any other obvious subtasks that I'm not thinking of? I'd
like to try to see whether we can break down this big task and scope out some
smaller pieces so we can make incremental progress and get this finished well
in time for 2.0.0 so we have lots of time to test.
> Allow Spark to be built without assemblies
> ------------------------------------------
>
> Key: SPARK-11157
> URL: https://issues.apache.org/jira/browse/SPARK-11157
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Spark Core, YARN
> Reporter: Marcelo Vanzin
> Attachments: no-assemblies.pdf
>
>
> For reasoning, discussion of pros and cons, and other more detailed
> information, please see attached doc.
> The idea is to be able to build a Spark distribution that has just a
> directory full of jars instead of the huge assembly files we currently have.
> Getting there requires changes in a bunch of places, I'll try to list the
> ones I identified in the document, in the order that I think would be needed
> to not break things:
> * make streaming backends not be assemblies
> Since people may depend on the current assembly artifacts in their
> deployments, we can't really remove them; but we can make them be dummy jars
> and rely on dependency resolution to download all the jars.
> PySpark tests would also need some tweaking here.
> * make examples jar not be an assembly
> Probably requires tweaks to the {{run-example}} script. The location of the
> examples jar would have to change (it won't be able to live in the same place
> as the main Spark jars anymore).
> * update YARN backend to handle a directory full of jars when launching apps
> Currently YARN localizes the Spark assembly (depending on the user
> configuration); it needs to be modified so that it can localize all needed
> libraries instead of a single jar.
> * Modify launcher library to handle the jars directory
> This should be trivial
> * Modify {{assembly/pom.xml}} to generate assembly or a {{libs}} directory
> depending on which profile is enabled.
> We should keep the option to build with the assembly on by default, for
> backwards compatibility, to give people time to prepare.
> Filing this bug as an umbrella; please file sub-tasks if you plan to work on
> a specific part of the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
