[
https://issues.apache.org/jira/browse/MAHOUT-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261365#comment-14261365
]
ASF GitHub Bot commented on MAHOUT-1636:
----------------------------------------
Github user dlyubimov commented on the pull request:
https://github.com/apache/mahout/pull/69#issuecomment-68386183
I am against shaded jars in any form or flavor. I have already spoken
before about all arguments there. This is perhaps build antipattern #1 of
the industry. Speak to any experienced build engineer if in doubt.
On Dec 30, 2014 9:17 AM, "Pat Ferrel" <[email protected]> wrote:
> The minimum piece to push is in this PR. So it will go in soon but the PR
> will remain active until several issues are addressed:
>
> 1. can we change the scan for jars to only find the dependencies.jar
> for any spark driver or perhaps even the shell. I think this should
work
> but it will have to be rather well tested in a running system since
missing
> class errors are not detected by unit tests.
> 2. with any dependencies.jar we have the issue of what artifacts to
> publish. In theory the dependencies are everything needed for Spark
Mahout.
> So if #1 <https://github.com/apache/mahout/pull/1> proves viable
> perhaps the dependencies.jar could be renamed and treated as the
release
> artifact for Spark flavored of Mahout. This is a bit beyond my
> understanding so it's important to get someone who understands the
total
> release process to look at the question.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/mahout/pull/69#issuecomment-68375633>.
>
> Class dependencies for the spark module are put in a job.jar, which is very
> inefficient
> ---------------------------------------------------------------------------------------
>
> Key: MAHOUT-1636
> URL: https://issues.apache.org/jira/browse/MAHOUT-1636
> Project: Mahout
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.0-snapshot
> Reporter: Pat Ferrel
> Assignee: Ted Dunning
> Fix For: 1.0-snapshot
>
>
> using a maven plugin and an assembly job.xml a job.jar is created with all
> dependencies including transitive ones. This job.jar is in
> mahout/spark/target and is included in the classpath when a Spark job is run.
> This allows dependency classes to be found at runtime but the job.jar include
> a great deal of things not needed that are duplicates of classes found in the
> main mrlegacy job.jar. If the job.jar is removed, drivers will not find
> needed classes. A better way needs to be implemented for including class
> dependencies.
> I'm not sure what that better way is so am leaving the assembly alone for
> now. Whoever picks up this Jira will have to remove it after deciding on a
> better method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
