[
https://issues.apache.org/jira/browse/MAHOUT-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282454#comment-16282454
]
ASF GitHub Bot commented on MAHOUT-2023:
----------------------------------------
Github user andrewpalumbo commented on a diff in the pull request:
https://github.com/apache/mahout/pull/353#discussion_r155627595
--- Diff: spark/src/main/assembly/dependency-reduced.xml ---
@@ -34,16 +34,15 @@
</unpackOptions>
<scope>runtime</scope>
<outputDirectory>/</outputDirectory>
- <useTransitiveFiltering>true</useTransitiveFiltering>
+ <useProjectArtifact>true</useProjectArtifact>
<includes>
<!-- guava only included to get Preconditions in mahout-math and
mahout-hdfs -->
<include>com.google.guava:guava</include>
- <include>com.github.scopt_${scala.compat.version}</include>
+ <include>com.github.scopt:scopt_2.10</include>
+ <include>com.github.scopt:scopt_2.11</include>
<include>com.tdunning:t-digest</include>
<include>org.apache.commons:commons-math3</include>
<include>it.unimi.dsi:fastutil</include>
-
<include>org.apache.mahout:mahout-native-viennacl_${scala.compat.version}</include>
--- End diff --
I think after we discussed yesterday, if adding these jars to the classpath
is the better practice, and we want this jar to be a slim as possible, leaving
them out would be fine. I am unclear on how often its shipped around on the
backend.
I think that the spark method- just adding to the
`spark.executor.extraClassPath` and `spark.driver.extraClassPath` makes sense.
@rawkintrevo this is what you were saying we should to, correct and to just add
it to the documentation? I don't think that the `dependency-reduced.jar` is
the way we want to be shipping everything in the future.
I think that this is something that we as a project need to discuss at some
point. The original goal for these solver modules (and the somewhat current,
hacky implementation) was to probe the classpath for the existence of a solver,
and then to use e.g. viennacl-omp#Mmul if it is on the classpath. Not sure
that adding it to the dependency reduce jar is the way to go.
I say we just ship this as is- still need to do more testing on a cluster
due to pat's next scopt find:
```
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
```
> Drivers broken, scopt classes not found
> ---------------------------------------
>
> Key: MAHOUT-2023
> URL: https://issues.apache.org/jira/browse/MAHOUT-2023
> Project: Mahout
> Issue Type: Bug
> Components: build
> Affects Versions: 0.13.1
> Environment: any
> Reporter: Pat Ferrel
> Assignee: Trevor Grant
> Priority: Blocker
> Fix For: 0.13.1
>
>
> Type `mahout spark-itemsimilarity` after Mahout is installed properly and you
> get a fatal exception due to missing scopt classes.
> Probably a build issue related to incorrect versions of scopt being looked
> for.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
