Sorry I missed your original response.
I export the SPARK_HOME to the install location and set the MASTER environment
variable to the mesos cluster. Yes we are using mesos as the scheduler.
E.g.
export SPARK_HOME=/etc/spark/spark-1.5.1
export MASTER="mesos://zk://maryann2:2181,maryann3:2181,maryann4:2181/mesos"
grep master /etc/spark/spark-1.5.1/conf/spark-defaults.conf
spark.master
mesos://zk://maryann2:2181,maryann3:2181,maryann4:2181/mesos
Not sure about the following which is (I think) the true error. I don't get any
interpreter logs, so I'm assuming it's not even getting started.
INFO [2015-11-09 12:31:48,272] ({qtp1241236050-99}
NotebookServer.java[onMessage]:101) - RECEIVE << PING
ERROR [2015-11-09 12:31:48,648] ({Thread-14}
RemoteInterpreterEventPoller.java[run]:63) - Can't get RemoteInterpreterEvent
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:61)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 7 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 8 more
--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.
From: Pablo Torre <pablotorr...@gmail.com<mailto:pablotorr...@gmail.com>>
Reply-To:
"users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>"
<users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>>
Date: Monday, November 9, 2015 8:35 AM
To:
"users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>"
<users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>>
Subject: Re: zeppelin startup warnings
Hi Moon,
I have the same problem. I am doing this in a script:
SPARK_PROFILE=1.4
SPARK_VERSION=1.4.1
HADOOP_PROFILE=2.4
HADOOP_VERSION=2.4.0
curl -sL --retry 3 \
"http://mirrors.ibiblio.org/apache/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz";
\
| gunzip \
| tar x -C /usr/ \
&& ln -s /usr/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE /usr/spark \
&& rm -rf /usr/spark/examples \
&& rm /usr/spark/lib/spark-examples*.jar
and then in my zeppelin configuration file I have this:
export SPARK_HOME="/usr/spark"
2015-11-08 7:09 GMT+01:00 moon soo Lee
<m...@apache.org<mailto:m...@apache.org>>:
This error may able to occur when SparkContext got some error to connect to
Spark cluster.
Could you share how you configured Zeppelin?
Have you export SPARK_HOME and set 'master' property in 'Interpreter' page?
Thanks,
moon
On Fri, Nov 6, 2015 at 1:03 AM Bill Sparks
<jspa...@cray.com<mailto:jspa...@cray.com>> wrote:
Hello.
I'm trying to run the master branch on CentOS6.5.
My compile completes and the server start successfully, in the log file I see
the following warnings. Question, are these expected? Also I'm uses spark with
Mesos, I've followed the wiki instruction s but get the following stack trace
on a simple inspection of the spark contect variable sc. Note the spark-shell
executes successfully. I'm thinking it has something to do with the thrift
server not running or … not even getting to executing the spark job.
Regards
-Bill
Startup Warnings (running as root):
[root@maryann4 incubator-zeppelin]# ./bin/zeppelin.sh
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support
was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/mnt/lustre/notebooks/zeppelin/incubator-zeppelin/zeppelin-server/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/mnt/lustre/notebooks/zeppelin/incubator-zeppelin/zeppelin-zengine/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/mnt/lustre/notebooks/zeppelin/incubator-zeppelin/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Nov 05, 2015 9:59:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.zeppelin.rest
Nov 05, 2015 9:59:13 AM com.sun.jersey.api.core.ScanningResourceConfig
logClasses
INFO: Root resource classes found:
class org.apache.zeppelin.rest.NotebookRestApi
class org.apache.zeppelin.rest.ZeppelinRestApi
class org.apache.zeppelin.rest.InterpreterRestApi
Nov 05, 2015 9:59:13 AM com.sun.jersey.api.core.ScanningResourceConfig init
INFO: No provider classes found.
Nov 05, 2015 9:59:13 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.13 06/29/2012 05:14 PM'
Nov 05, 2015 9:59:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or
provider classes:
WARNING: A sub-resource method, public javax.ws.rs.core.Response
org.apache.zeppelin.rest.NotebookRestApi.createNote(java.lang.String) throws
java.io.IOException, with URI template, "/", is treated as a resource method
WARNING: A sub-resource method, public javax.ws.rs.core.Response
org.apache.zeppelin.rest.NotebookRestApi.getNotebookList() throws
java.io.IOException, with URI template, "/", is treated as a resource method
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.zeppelin.rest.InterpreterRestApi.listInterpreter(java.lang.String),
should not consume any entity.
Error via spark interpreter …
java.net.ConnectException: Connection refused at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:589) at
org.apache.thrift.transport.TSocket.open(TSocket.java:182) at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:137)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:257)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:197) at
org.apache.zeppelin.scheduler.Job.run(Job.java:170) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:304)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
Jonathan (Bill) Sparks
Software Architecture
--
Pablo Torre.
Freelance software engineer and Ruby on Rails developer.
Oleiros (Coruña)
Personal site<http://www.pablotorrerodriguez.com>
My blog<http://www.aboutechnologies.com>
