On 22 Oct 2015, at 02:47, Ajay Chander
<itsche...@gmail.com<mailto:itsche...@gmail.com>> wrote:
Thanks for your time. I have followed your inputs and downloaded
"spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a pie
test everything seems to be working fine, except that the spark-history -server
running on this node1 has gone down. It was complaining about missing class:
15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for [TERM,
HUP, INT]
15/10/21 16:41:28 WARN SparkConf: The configuration key
'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3
and and may be removed in the future. Please use the new key
'spark.yarn.am.waitTime' instead.
15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at
org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
at
org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
OK, that's happening because the version of spark we ship comes with
integration with the yarn history provider, and that's not on your classpath.
find the history server config and either unset spark.history.provider or set
it back to the filesystem one:
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
The 1.5.1-based binaries we'll push out will be set up for this and for the
spark apps to be able to publish to the timeline server.
note that irrespective of how histories are saved, a 1.3 or 1.4 history server
cannot view a 1.5 one. So you will need to run a 1.5.x history server to view
the most recent histories. But you can run a 1.4 history server talking to the
timeline service alongside a 1.5 history server working through HDFS.
see: SPARK-1537 for the details of what's happening
