[
https://issues.apache.org/jira/browse/MAHOUT-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260221#comment-14260221
]
ASF GitHub Bot commented on MAHOUT-1636:
----------------------------------------
GitHub user pferrel opened a pull request:
https://github.com/apache/mahout/pull/69
MAHOUT-1636
Started out simplifying driver code and making changes to all drivers to
support that. Then ran into the fat job.jar issue of MAHOUT-1636 so created a
slimmed down version of the old job.jar by adding excludes to job.xml and
changing the name to "dependencies.jar"
The new jar works for spark-itemsimilarity and spark-row-similarity but
needs to be tested for the naive bayes drivers.
The dependencies.jar still contains a lot of stuff from mrlegacy, some is
in external projects, like jackson that can be excluded with this mechanism but
also a lot of mahout code that is unneeded in this jar. This later case would
require some other mechanism than a simple <exclude> clause in the assembly xml
file.
I believe the new dependencies.jar is the only thing that needs to be on
the classpath when running spark drivers or the spark-shell. I haven't changed
this but it is a further refinement we can try.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pferrel/mahout MAHOUT-1636
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/mahout/pull/69.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #69
----
commit d0f64205a116853aa471dd1361a635167da15fcc
Author: pferrel <[email protected]>
Date: 2014-12-27T23:43:41Z
simplified driver and made required changes to all, note: left job assembly
untouched
commit 5e32612b4ca9400287c43e4130478f04207fbcfa
Author: pferrel <[email protected]>
Date: 2014-12-29T16:49:10Z
creating a trimmed down all-deps dependencies.jar for spark drivers
----
> Class dependencies for the spark module are put in a job.jar, which is very
> inefficient
> ---------------------------------------------------------------------------------------
>
> Key: MAHOUT-1636
> URL: https://issues.apache.org/jira/browse/MAHOUT-1636
> Project: Mahout
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.0-snapshot
> Reporter: Pat Ferrel
> Assignee: Ted Dunning
> Fix For: 1.0-snapshot
>
>
> using a maven plugin and an assembly job.xml a job.jar is created with all
> dependencies including transitive ones. This job.jar is in
> mahout/spark/target and is included in the classpath when a Spark job is run.
> This allows dependency classes to be found at runtime but the job.jar include
> a great deal of things not needed that are duplicates of classes found in the
> main mrlegacy job.jar. If the job.jar is removed, drivers will not find
> needed classes. A better way needs to be implemented for including class
> dependencies.
> I'm not sure what that better way is so am leaving the assembly alone for
> now. Whoever picks up this Jira will have to remove it after deciding on a
> better method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
