Hi All, I am using Yarn and had set the following setting in livy.conf but Livy was unable to recover the session moreover the spark job spawned by Livy got killed when it is supposed to be running. Below is the stack trace. I am not sure what to deduce from it?
The following are the settings I set in livy.conf that are needed for recovery. I did not set anything for "livy.spark.deploy-mode" (It's actually commented out) livy.spark.master = yarn livy.server.recovery.mode = recovery livy.server.recovery.state-store = zookeeper livy.server.recovery.state-store.url = localhost:2181 Here is my stacktrace 18/03/20 09:15:57 INFO ZooKeeper: Client environment:user.dir=/home/ubuntu/livy-0.4.0-incubating-bin/bin 18/03/20 09:15:57 INFO ZooKeeper: Initiating client connection, connectString=x.x.x.x:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4ae9cfc1 18/03/20 09:15:57 INFO StateStore$: Using ZooKeeperStateStore for recovery. 18/03/20 09:15:57 INFO ClientCnxn: Opening socket connection to server x.x.x.x/x.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) 18/03/20 09:15:57 INFO ClientCnxn: Socket connection established to x.x.x.x/x.x.x.x:2181, initiating session 18/03/20 09:15:57 INFO ClientCnxn: Session establishment complete on server x.x.x.x/x.x.x.x:2181, sessionid = 0x162426cb5710007, negotiated timeout = 60000 18/03/20 09:15:57 INFO ConnectionStateManager: State change: CONNECTED 18/03/20 09:15:57 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0 18/03/20 09:15:58 ERROR RSCClient: Failed to connect to context. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO InteractiveSessionManager: Recovered 1 interactive sessions. Next session id: 4 18/03/20 09:15:58 INFO InteractiveSessionManager: Registering new session 3 18/03/20 09:15:58 INFO InteractiveSessionManager: Heartbeat watchdog thread started. 18/03/20 09:15:58 ERROR RSCClient: RPC error. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO WebServer: Starting server on http://x.x.x.x.us-west-2.compute.internal:8998 18/03/20 09:18:32 ERROR SessionServlet$: internal error java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.livy.server.interactive.InteractiveSession.statements(InteractiveSession.scala:471) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:69) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:67) at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionServlet.scala:205) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withViewAccessSession(SessionHeartbeat.scala:67) at org.apache.livy.server.interactive.InteractiveSessionServlet.withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7.apply(InteractiveSessionServlet.scala:102) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) at scala.Option.flatMap(Option.scala:170) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) at scala.collection.immutable.Stream.flatMap(Stream.scala:446) at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:34) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more 18/03/20 09:18:36 ERROR SessionServlet$: internal error java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.livy.server.interactive.InteractiveSession.statements(InteractiveSession.scala:471) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:69) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:67) at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionServlet.scala:205) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withViewAccessSession(SessionHeartbeat.scala:67) at org.apache.livy.server.interactive.InteractiveSessionServlet.withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7.apply(InteractiveSessionServlet.scala:102) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) at scala.Option.flatMap(Option.scala:170) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) at scala.collection.immutable.Stream.flatMap(Stream.scala:446) at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:34) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more