[
https://issues.apache.org/jira/browse/SPARK-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567746#comment-14567746
]
Kannan Rajah commented on SPARK-4048:
-------------------------------------
This fix has caused a regression for Spark built using hadoop-provided mode
with Hadoop 2.5. Hadoop 2.5 does not contain curator jars. But spark class
ZooKeeperLeaderElectionAgent needs it. This causes spark master to fail.
15/06/01 06:35:24 ERROR ActorSystemImpl: Uncaught fatal error from thread
[sparkMaster-akka.actor.default-dispatcher-4] shutting down ActorSystem
[sparkMaster]
java.lang.NoClassDefFoundError:
org/apache/curator/framework/recipes/leader/LeaderLatchListener
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:69)
> Enhance and extend hadoop-provided profile
> ------------------------------------------
>
> Key: SPARK-4048
> URL: https://issues.apache.org/jira/browse/SPARK-4048
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.2.0
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
> Fix For: 1.3.0
>
>
> The hadoop-provided profile is used to not package Hadoop dependencies inside
> the Spark assembly. It works, sort of, but it could use some enhancements. A
> quick list:
> - It doesn't include all things that could be removed from the assembly
> - It doesn't work well when you're publishing artifacts based on it
> (SPARK-3812 fixes this)
> - There are other dependencies that could use similar treatment: Hive, HBase
> (for the examples), Flume, Parquet, maybe others I'm missing at the moment.
> - Unit tests, more specifically, those that use local-cluster mode, do not
> work when the assembly is built with this profile enabled.
> - The scripts to launch Spark jobs do not add needed "provided" jars to the
> classpath when this profile is enabled, leaving it for people to figure that
> out for themselves.
> - The examples assembly duplicates a lot of things in the main assembly.
> Part of this task is selfish since we build internally with this profile and
> we'd like to make it easier for us to merge changes without having to keep
> too many patches on top of upstream. But those feel like good improvements to
> me, regardless.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
