[
https://issues.apache.org/jira/browse/MAHOUT-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257737#comment-14257737
]
Pat Ferrel edited comment on MAHOUT-1636 at 12/24/14 12:47 AM:
---------------------------------------------------------------
I was really hoping to stay on the topic of how to structure artifacts and make
sure classes can be found. For now can we assume that external classes will
occasionally be needed?
Talking to Andrew P and me your opinion was that anything non-engine specific
should go in math-scala. Renaming the module was discussed but no consensus to
change it was reached.
was (Author: pferrel):
I was really hoping to stay on the topic of how to structure artifacts and make
sure classes can be found. For now can we assume that external classes will
occasionally be needed?
Talking to Andrew P and I your opinion was that anything non-engine specific
should go in math-scala. Renaming the module was discussed but no consensus to
change it was reached.
> Class dependencies for the spark module are put in a job.jar, which is very
> inefficient
> ---------------------------------------------------------------------------------------
>
> Key: MAHOUT-1636
> URL: https://issues.apache.org/jira/browse/MAHOUT-1636
> Project: Mahout
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.0-snapshot
> Reporter: Pat Ferrel
> Fix For: 1.0-snapshot
>
>
> using a maven plugin and an assembly job.xml a job.jar is created with all
> dependencies including transitive ones. This job.jar is in
> mahout/spark/target and is included in the classpath when a Spark job is run.
> This allows dependency classes to be found at runtime but the job.jar include
> a great deal of things not needed that are duplicates of classes found in the
> main mrlegacy job.jar. If the job.jar is removed, drivers will not find
> needed classes. A better way needs to be implemented for including class
> dependencies.
> I'm not sure what that better way is so am leaving the assembly alone for
> now. Whoever picks up this Jira will have to remove it after deciding on a
> better method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
