hi Guys,
I am trying to run Zeppelin using Yarn as resource manager. I have made
following changes
1- I have specified master as 'yarn-client' in the interpreter settings
using UI
2. I have specified HADOOP_CONF_DIR as conf directory containing hadoop
configuration files
In my scenario I have three machines.
a- Client Machine where zeppelin is installed
b- Machine where YARN cluster manager along with nodemanager, namenode,
datanode, secondary namenode are running
c- Machine where only nodemanager and datanode is running
When I submit job from my client machine , it gets submitted to yarn but
fails with following exception -
5/08/04 15:08:05 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Failed to connect to driver!
at
org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:424)
at
org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:284)
at
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:146)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:575)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:573)
at
org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:596)
at
org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
15/08/04 15:08:05 INFO yarn.ApplicationMaster: Final app status:
FAILED, exitCode: 10, (reason: Uncaught exception: Failed to connect
to driver!)
Any help is much appreciated!
Regards
Monica
On Tue, Aug 4, 2015 at 10:57 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote:
> That worked. Why ?
> Can you share a comprehensive iist of examples.
>
>
> On Mon, Aug 3, 2015 at 4:59 PM, Alex <abezzu...@nflabs.com> wrote:
>
>> Hi,
>>
>> inside %spark you do not need to create SqlContext manually:
>> as with "sc" for SparkContext, Interpreter already have injected "sqlc"
>> val.
>>
>> Also AFAIK println statement should be in the separate paragraph.
>>
>> Can you try using that and see if it helps?
>>
>> --
>> Kind regards,
>> Alexander
>>
>> On 04 Aug 2015, at 05:58, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote:
>>
>> I am unable to see the visualization with Zeppelin from blog :
>> http://hortonworks.com/blog/introduction-to-data-science-with-apache-spark/
>>
>>
>> Notebook
>> %spark
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>> import sqlContext.implicits._
>> import java.sql.Date
>> import org.apache.spark.sql.Row
>>
>> case class Log(level: String, date: Date, fileName: String)
>>
>> import java.text.SimpleDateFormat
>>
>> val df = new SimpleDateFormat("yyyy-mm-dd HH:mm:ss,SSS")
>>
>> val ambari = ambariLogs.map { line =>
>> val s = line.split(" ")
>> val logLevel = s(0)
>> val dateTime = df.parse(s(1) + " " + s(2))
>> val fileName = s(3).split(":")(0)
>> Log(logLevel,new Date(dateTime.getTime()), fileName)}.toDF()
>> ambari.registerTempTable("ambari")
>>
>>
>> //ambari.groupBy("level").count()
>> sqlContext.sql("SELECT COUNT(*) from ambari")
>>
>> Output:
>>
>> sqlContext: org.apache.spark.sql.SQLContext =
>> org.apache.spark.sql.SQLContext@5ca68ee6 import sqlContext.implicits._
>> import java.sql.Date import org.apache.spark.sql.Row defined class Log
>> import java.text.SimpleDateFormat df: java.text.SimpleDateFormat =
>> java.text.SimpleDateFormat@98f267e7 ambari:
>> org.apache.spark.sql.DataFrame = [level: string, date: date, fileName:
>> string] res74: org.apache.spark.sql.DataFrame = [c0: bigint]
>>
>>
>> Hence the table ambari is created successfully.
>>
>> In a new note, i wrote this
>>
>> %spark
>> import org.apache.spark.sql.Row
>>
>> val result = sqlContext.sql("SELECT level, COUNT(1) from ambari group by
>> level").map {
>> case Row(level: String, count: Long) => {
>> level + "\t" + count
>> }
>> }.collect()
>>
>> println("%table Log Level\tCount\n" + result.mkString("\n"))
>>
>>
>> Output:
>> import org.apache.spark.sql.Row result: Array[String] = Array(INFO 2444,
>> WARNING 3) %table Log Level Count INFO 2444 WARNING 3
>>
>> I did not get graph rendering despite am outputing %table from println.
>>
>> Any suggestions ?
>>
>>
>> On Mon, Aug 3, 2015 at 1:47 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
>> wrote:
>>
>>> Fixed it
>>>
>>> mvn clean package -Pspark-1.3 -Dspark.version=1.3.1
>>> -Dhadoop.version=2.7.0 -Phadoop-2.6 -Pyarn -DskipTests
>>>
>>> Earlier i had
>>>
>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>> -Phadoop-2.7 -Pyarn
>>>
>>> On Mon, Aug 3, 2015 at 1:31 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
>>> wrote:
>>>
>>>> I have hadoop cluster up using Ambari. It also allowed me to install
>>>> Spark 1.3.1 and i can run sample spark application & Yarn application. So
>>>> cluster is up and running fine.
>>>>
>>>> I got Zeppelin setup on a new box and was able to launch UI.
>>>>
>>>> I modified spark interpreter to set
>>>>
>>>> masteryarn-clientspark.app.nameZeppelinspark.cores.max
>>>> spark.driver.extraJavaOptions-Dhdp.version=2.3.1.0-2574
>>>> spark.executor.memory512mspark.home/usr/hdp/2.3.1.0-2574/spark
>>>> spark.yarn.am.extraJavaOptions-Dhdp.version=2.3.1.0-2574spark.yarn.jar
>>>> /home/zeppelin/incubator-zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar
>>>> zeppelin.dep.localrepolocal-repo
>>>>
>>>> When i run a spark notebook
>>>> %spark
>>>> val ambariLogs =
>>>> sc.textFile("file:///var/log/ambari-agent/ambari-agent.log")
>>>> ambariLogs.take(10).mkString("\n")
>>>>
>>>> (The location exists)
>>>>
>>>> I see two exceptions in Zeppelin spark interpreter logs
>>>>
>>>> ERROR [2015-08-03 13:30:50,262] ({pool-1-thread-2}
>>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>>>
>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$
>>>>
>>>> at
>>>> org.apache.spark.deploy.yarn.ClientArguments.<init>(ClientArguments.scala:38)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:55)
>>>>
>>>> at
>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>>>
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>>>
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:301)
>>>>
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
>>>>
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:423)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:298)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1068)
>>>>
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1053)
>>>>
>>>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>>>
>>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>>>
>>>> at
>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> AND
>>>>
>>>>
>>>> WARN [2015-08-03 13:30:50,085] ({pool-1-thread-2}
>>>> Logging.scala[logWarning]:71) - Service 'SparkUI' could not bind on port
>>>> 4041. Attempting port 4042.
>>>>
>>>> INFO [2015-08-03 13:30:50,112] ({pool-1-thread-2}
>>>> Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT
>>>>
>>>> WARN [2015-08-03 13:30:50,123] ({pool-1-thread-2}
>>>> AbstractLifeCycle.java[setFailed]:204) - FAILED
>>>> SelectChannelConnector@0.0.0.0:4042: java.net.BindException: Address
>>>> already in use
>>>>
>>>> java.net.BindException: Address already in use
>>>>
>>>> at sun.nio.ch.Net.bind0(Native Method)
>>>>
>>>> at sun.nio.ch.Net.bind(Net.java:444)
>>>>
>>>> at sun.nio.ch.Net.bind(Net.java:436)
>>>>
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>>>>
>>>>
>>>> Any suggestions ?
>>>>
>>>>
>>>> On Mon, Aug 3, 2015 at 11:00 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks a lot for all these documents. Appreciate your effort & time.
>>>>>
>>>>> On Mon, Aug 3, 2015 at 10:15 AM, Christian Tzolov <ctzo...@pivotal.io>
>>>>> wrote:
>>>>>
>>>>>> ÐΞ€ρ@Ҝ (๏̯͡๏),
>>>>>>
>>>>>> I've successfully run Zeppelin with Spark on YARN. I'm using Ambari
>>>>>> and PivotalHD30. PHD30 is ODP compliant so you should be able to repeat
>>>>>> the
>>>>>> configuration for HDP (e.g. hortonworks).
>>>>>>
>>>>>> 1. Before you start with Zeppelin, make sure that your Spark/YARN
>>>>>> env. works from the command line (e.g run Pi test). If it doesn't work
>>>>>> make
>>>>>> sure that the hdp.version is set correctly or you can hardcode the
>>>>>> stack.name and stack.version properties as Ambari Custom yarn-site
>>>>>> properties (that is what i did).
>>>>>>
>>>>>> 2. Your Zeppelin should be build with proper Spark and Hadoop
>>>>>> versions and YARN support enabled. In my case used this build command:
>>>>>>
>>>>>> mvn clean package -Pspark-1.4 -Dspark.version=1.4.1
>>>>>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Pyarn -DskipTests -Pbuild-distr
>>>>>>
>>>>>> 3. Open the Spark interpreter configuration and set 'master' property
>>>>>> to 'yarn-client' ( e.g. master=yarn-client). then press Save.
>>>>>>
>>>>>> 4. In (conf/zeppelin-env.sh) set HADOOP_CONF_DIR for PHD and HDP it
>>>>>> will look like this:
>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>
>>>>>> 5. (optional) i've restarted the zeppelin daemon but i don't think
>>>>>> this is required.
>>>>>>
>>>>>> 6. Make sure that HDFS has /user/<zeppelin user> folder exists and
>>>>>> has HDFS write permissions. Otherwise you can create it like this:
>>>>>> sudo -u hdfs hdfs dfs -mkdir /user/<zeppelin user>
>>>>>> sudo -u hdfs hdfs dfs -chown -R <zeppelin user>t:hdfs
>>>>>> /user/<zeppelin user>
>>>>>>
>>>>>> Good to go!
>>>>>>
>>>>>> Cheers,
>>>>>> Christian
>>>>>>
>>>>>> On 3 August 2015 at 17:50, Vadla, Karthik <karthik.va...@intel.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Deepak,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have documented everything here.
>>>>>>>
>>>>>>> Please check published document.
>>>>>>>
>>>>>>>
>>>>>>> https://software.intel.com/sites/default/files/managed/bb/bf/Apache-Zeppelin.pdf
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Karthik Vadla
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From:* ÐΞ€ρ@Ҝ (๏̯͡๏) [mailto:deepuj...@gmail.com]
>>>>>>> *Sent:* Sunday, August 2, 2015 9:25 PM
>>>>>>> *To:* users@zeppelin.incubator.apache.org
>>>>>>> *Subject:* Yarn + Spark + Zepplin ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I would like to try out Zepplin and hence i got a 7 node Hadoop
>>>>>>> cluster with spark history server setup. I am able to run sample spark
>>>>>>> applications on my YARN cluster.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have no clue how to get zepplin to connect to this YARN cluster.
>>>>>>> Under
>>>>>>> https://zeppelin.incubator.apache.org/docs/install/install.html i
>>>>>>> see MASTER to point to spark master. I do not have a spark master
>>>>>>> running.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> How do i get Zepplin to be able to read data from YARN cluster ?
>>>>>>> Please share documentation.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Deepak
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution
>>>>>> Architect, EMEA Practice Team | Pivotal <http://pivotal.io/>
>>>>>> ctzo...@pivotal.io|+31610285517
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
