[
https://issues.apache.org/jira/browse/SPARK-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972397#comment-13972397
]
Sean Owen commented on SPARK-1520:
----------------------------------
Madness. One wild guess is that the breeze .jar files have something in
META-INF that, when merged together into the assembly jar, conflicts with other
META-INF items. In particular I'm thinking of MANIFEST.MF entries. It's worth
diffing those if you can from before and after. However this would still
require that Java 7 and 6 behave differently with respect to the entries, to
explain your findings. It's possible.
Your last comment however suggests it's something strange with the byte code
that gets output for a few classes. Java 7 is stricter about byte code. For
example:
https://weblogs.java.net/blog/fabriziogiudici/archive/2012/05/07/understanding-subtle-new-behaviours-jdk-7
However I would think these would manifest as quite different errors.
What about running with -verbose:class to print classloading messages? it might
point directly to what's failing to load, if that's it.
Of course you can always build with Java 6 since that's supposed to be all
that's supported/required now (see my other JIRA about making Jenkins do this),
although I agree that it would be nice to get to the bottom of this, as there
is no obvious reason this shouldn't work.
> Inclusion of breeze corrupts assembly when compiled with JDK7 and run on JDK6
> -----------------------------------------------------------------------------
>
> Key: SPARK-1520
> URL: https://issues.apache.org/jira/browse/SPARK-1520
> Project: Spark
> Issue Type: Bug
> Components: MLlib, Spark Core
> Reporter: Patrick Wendell
> Priority: Blocker
> Fix For: 1.0.0
>
>
> This is a real doozie - when compiling a Spark assembly with JDK7, the
> produced jar does not work well with JRE6. I confirmed the byte code being
> produced is JDK 6 compatible (major version 50). What happens is that,
> silently, the JRE will not load any class files from the assembled jar.
> {code}
> $> sbt/sbt assembly/assembly
> $> /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -cp
> /home/patrick/Documents/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar
> org.apache.spark.ui.UIWorkloadGenerator
> usage: ./bin/spark-class org.apache.spark.ui.UIWorkloadGenerator [master]
> [FIFO|FAIR]
> $> /usr/lib/jvm/java-1.6.0-openjdk-amd64/bin/java -cp
> /home/patrick/Documents/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar
> org.apache.spark.ui.UIWorkloadGenerator
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/ui/UIWorkloadGenerator
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.ui.UIWorkloadGenerator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
> Could not find the main class: org.apache.spark.ui.UIWorkloadGenerator.
> Program will exit.
> {code}
> I also noticed that if the jar is unzipped, and the classpath set to the
> currently directory, it "just works". Finally, if the assembly jar is
> compiled with JDK6, it also works. The error is seen with any class, not just
> the UIWorkloadGenerator. Also, this error doesn't exist in branch 0.9, only
> in master.
> *Isolation*
> -I ran a git bisection and this appeared after the MLLib sparse vector patch
> was merged:-
> https://github.com/apache/spark/commit/80c29689ae3b589254a571da3ddb5f9c866ae534
> SPARK-1212
> -I narrowed this down specifically to the inclusion of the breeze library.
> Just adding breeze to an older (unaffected) build triggered the issue.-
> I've found that if I just unpack and re-pack the jar, it sometimes works:
> {code}
> $ cd assembly/target/scala-2.10/
> $ /usr/lib/jvm/java-1.6.0-openjdk-amd64/bin/java -cp
> ./spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar
> org.apache.spark.ui.UIWorkloadGenerator # fails
> $ jar xvf spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar
> $ jar cvf spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar *
> $ /usr/lib/jvm/java-1.6.0-openjdk-amd64/bin/java -cp
> ./spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar
> org.apache.spark.ui.UIWorkloadGenerator # succeeds
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
