-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49537/
-----------------------------------------------------------
Review request for Ambari and Alejandro Fernandez.
Bugs: AMBARI-17537
https://issues.apache.org/jira/browse/AMBARI-17537
Repository: ambari
Description
-------
After deploy Spark history server is stopped.
16/06/27 19:55:45 INFO HistoryServer: History provider class:
org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:224)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala).
Why this jira happened?
In the jira, the spark version is 1.5.1. which doesnot has the class
org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider. Instead, it
has "org.apache.spark.deploy.yarn.history.YarnHistoryProvider" and
"org.apache.spark.deploy.history.yarn.YarnHistoryProvider".
How to fix?
Actually, in spark internal logic,
"org.apache.spark.deploy.yarn.history.YarnHistoryProvider" is a wrapper of
"org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider" and
"org.apache.spark.deploy.history.yarn.YarnHistoryProvider".
The property "spark.history.provider" appears twice in the stack definition,
having two values: one is
"org.apache.spark.deploy.yarn.history.YarnHistoryProvider", and the other one
is "org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider". In this
patch, remove the one whose value is
"org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider".
The jira is originally reported by Denis Tarasyuk.
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
84eb805
Diff: https://reviews.apache.org/r/49537/diff/
Testing
-------
Tests passed in the cluster: Ambari 2.4 + HDP 2.3.2
Thanks,
Weiqing Yang
