for broken pipe error,
Zeppelin runs Spark driver (SparkContext) in separate JVM process and
communicate through thrift protocol. The error is about socket
disconnection between Zeppelin server daemon and Spark driver JVM process.
The reason can be vary. One possibility is, once SparkContext is terminated
by any reason, it sometimes terminates the process, too. Then Zeppelin
server could raise such exception.
Do you see any possibility of disconnection between Zeppelin server daemon
and Spark driver JVM process in your environment?
Thanks,
moon
On Mon, Jun 22, 2015 at 8:29 PM moon soo Lee <m...@apache.org> wrote:
> Hi,
>
> Really appreciate for sharing the problem.
> About the NPE, when do you get the exception? Are you getting the
> exception when you trying to access Spark application ui?
> If it is, any spark job (like .count(), .collect(), sparksql, etc) runs
> without error?
>
> Thanks!
> moon
>
> On Mon, Jun 22, 2015 at 5:19 AM Ophir Cohen <oph...@gmail.com> wrote:
>
>> OK,
>> I think we can seperate the issues.
>> Let's start from the NPE. For some reason (is that on perpuse?) when
>> Zeppelin starts Spark it seems that it does not open the web ui.
>> Trying to access port 4041 (this is the one it started on) yield the
>> above NPE with the "WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>> ServletHandler.java[doHandle]:561) - /jobs/
>> java.lang.NullPointerException" message.
>>
>> Any ideas?
>>
>> On Mon, Jun 22, 2015 at 12:34 PM, Ophir Cohen <oph...@gmail.com> wrote:
>>
>>> Hi Guys and great work!
>>> In our company we decided to relay on Zeppelin as our tool to access and
>>> visualize our data.
>>> I'm working with Spark 1.3.1 on top of Hadoop yarn 2.4
>>>
>>> Lately we started to see many NullPointerExcpetion on the Spark
>>> interpreter log (see below).
>>> Sometimes it does not affect Zeppelin but sometimes we start to see
>>> 'can't get status' error caused by broken pipe, this time on the
>>> *zeppelin* log file (see below).
>>> When the latest error starts - it stacks the Spark interpretor and it
>>> enforce us to restart Zeppelin.
>>>
>>> I couldn't find when and why it's happening, it just happened once
>>> awhile...
>>> Any help will be more than appreciated!
>>> 10x!
>>>
>>> *The NPE on zeppelin-interpreter-spark log file:*
>>> WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>> ServletHandler.java[doHandle]:561) - /jobs/
>>> java.lang.NullPointerException
>>> at
>>> org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
>>> at
>>> org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
>>> at
>>> org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
>>> at
>>> org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
>>> at
>>> org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
>>> at
>>> org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>> at
>>> org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>> at
>>> org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>> at org.spark-project.jetty.server.Server.handle(Server.java:370)
>>> at
>>> org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
>>> at
>>> org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
>>> at
>>> org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
>>> at
>>> org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>> at
>>> org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>> at
>>> org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>> at org.spark-project.jetty.io
>>> .nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>>> at org.spark-project.jetty.io
>>> .nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>> at
>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> at
>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> *The broken pipe on zeppelin log file:*
>>> ERROR [2015-06-21 13:10:31,368] ({Thread-120}
>>> RemoteScheduler.java[getStatus]:245) - Can't get status information
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.SocketException: Broken pipe
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>> at
>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.send_getStatus(RemoteInterpreterService.java:319)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getStatus(RemoteInterpreterService.java:311)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:232)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:183)
>>> Caused by: java.net.SocketException: Broken pipe
>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>> at
>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>> at
>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> at
>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>> ... 5 more
>>>
>>>
>>
