Rentaro Matsukata created ZEPPELIN-3361:
-------------------------------------------
Summary: 0.7.3 unable to register application in spark 2.2.x
Key: ZEPPELIN-3361
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3361
Project: Zeppelin
Issue Type: Bug
Affects Versions: 0.7.3
Reporter: Rentaro Matsukata
Zeppelin is unable to register itself as an application on Spark master in
Spark versions 2.2.x the same way it is unable to on 2.3.x
I was able to confirm that with the same job in the same deployment
configuration, zeppelin successfully submits workloads to Spark 2.1.2
Setup: Spark Standalone deployed in Kubernetes on Laptop (Minikube). 1 master
and 2 workers, each with 512MiB allocated
Zeppelin interpreter configs:
{code:java}
spark.cores.max = 2
spark.executor.memory = 512m
zeppelin.spark.useHiveContext = false{code}
Cell Code:
{code:java}
%spark.pyspark
from math import sqrt; from itertools import count, islice
def isprime(n):
return n > 1 and all(n%i for i in islice(count(2), int(sqrt(n)-1)))
nums = sc.parallelize(xrange(10000))
print nums.filter(isprime).count()
{code}
Spark Master Logs:
{code:java}
18/03/23 05:52:19 ERROR TransportRequestHandler: Error while invoking
RpcHandler#receive() on RPC id 9026752075421821168 java.io.EOFException at
java.io.DataInputStream.readFully(DataInputStream.java:197) at
java.io.DataInputStream.readUTF(DataInputStream.java:609) at
java.io.DataInputStream.readUTF(DataInputStream.java:564) at
org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:585)
at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:595) at
org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:654)
at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:639)
at
org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
at
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
{code}
Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
