Jeff, I don't see any error on the RM logs. This works fine after restarting the zeppelin livy interpreter. It runs for 3 hours and it stars throwing session state error.
We are using spark 2 and what is the correct config for these ? Where can I find documentation for these ? # What spark master Livy sessions should use. livy.spark.master = yarn # What spark deploy mode Livy sessions should use. livy.spark.deployMode = cluster I am seeing this error on livy logs. 17/12/12 06:20:47 INFO ContextLauncher: 17/12/12 06:20:47 INFO ShutdownHookManager: Shutdown hook called 17/12/12 06:20:47 INFO ContextLauncher: 17/12/12 06:20:47 INFO ShutdownHookManager: Deleting directory /tmp/spark-cc7cbfb8-3af1-44de-9287-5c1331e20a01 17/12/12 06:20:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 17/12/12 06:20:48 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 17/12/12 06:20:49 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 17/12/12 06:20:50 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) On Mon, Dec 11, 2017 at 4:04 PM, Jeff Zhang <zjf...@gmail.com> wrote: > > Could you check the yarn app logs ? It seems livy session fails to start. > > zeppelin would not to recreate a new session when fails to create session > (session is dead), because usually it is due to user configuration or > cluster issue, recreating session may lead infinity loop. > > > Anandha L Ranganathan <analog.s...@gmail.com>于2017年12月12日周二 上午7:42写道： > >> We get this error from Zeppelin UI and logs. >> >> >> org.apache.zeppelin.livy.LivyException: >> org.springframework.web.client.HttpServerErrorException: >> 500 Internal Server Error >> at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI( >> BaseLivyInterpreter.java:552) >> at org.apache.zeppelin.livy.BaseLivyInterpreter.executeStatement( >> BaseLivyInterpreter.java:415) >> at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret( >> BaseLivyInterpreter.java:249) >> at org.apache.zeppelin.livy.BaseLivyInterpreter.interpret( >> BaseLivyInterpreter.java:168) >> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret( >> LazyOpenInterpreter.java:101) >> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$ >> InterpretJob.jobRun(RemoteInterpreterServer.java:602) >> at org.apache.zeppelin.scheduler.Job.run(Job.java:182) >> >> >> >> >> But on the Livy server, we see this error session is in dead state. >> Isn't that Livy should create a new session if it found a dead session ? >> Restarting of of Zeppelin interpreter resolves this problem but it is very >> annoying to restart the Zeppelin interpreter. >> >> >> 17/12/11 23:29:30 ERROR SessionServlet$: internal error >> java.lang.IllegalStateException: Session is in state dead >> at com.cloudera.livy.server.interactive.InteractiveSession. >> ensureRunning(InteractiveSession.scala:531) >> at com.cloudera.livy.server.interactive.InteractiveSession. >> executeStatement(InteractiveSession.scala:459) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet$$ >> anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:124) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet$$ >> anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:123) >> at com.cloudera.livy.server.SessionServlet.doWithSession( >> SessionServlet.scala:194) >> at com.cloudera.livy.server.SessionServlet.withSession( >> SessionServlet.scala:187) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet$$ >> anonfun$11.apply(InteractiveSessionServlet.scala:123) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet$$ >> anonfun$11.apply(InteractiveSessionServlet.scala:122) >> at com.cloudera.livy.server.JsonServlet.com$cloudera$livy$ >> server$JsonServlet$$doAction(JsonServlet.scala:114) >> at com.cloudera.livy.server.JsonServlet$$anonfun$jpost$1. >> apply(JsonServlet.scala:76) >> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$ >> liftAction(ScalatraBase.scala:270) >> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply( >> ScalatraBase.scala:265) >> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply( >> ScalatraBase.scala:265) >> at org.scalatra.ApiFormats$class.withRouteMultiParams( >> ApiFormats.scala:178) >> at com.cloudera.livy.server.JsonServlet.withRouteMultiParams( >> JsonServlet.scala:40) >> at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) >> at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) >> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$ >> apply$8.apply(ScalatraBase.scala:240) >> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$ >> apply$8.apply(ScalatraBase.scala:238) >> at scala.Option.flatMap(Option.scala:170) >> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply( >> ScalatraBase.scala:238) >> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply( >> ScalatraBase.scala:237) >> at scala.collection.immutable.Stream.flatMap(Stream.scala:446) >> at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) >> at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) >> at org.scalatra.ScalatraBase$class.runActions$1( >> ScalatraBase.scala:163) >> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$ >> mcV$sp(ScalatraBase.scala:175) >> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply( >> ScalatraBase.scala:175) >> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply( >> ScalatraBase.scala:175) >> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$ >> cradleHalt(ScalatraBase.scala:193) >> at org.scalatra.ScalatraBase$class.executeRoutes( >> ScalatraBase.scala:175) >> at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet. >> scala:49) >> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp( >> ScalatraBase.scala:113) >> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply( >> ScalatraBase.scala:113) >> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply( >> ScalatraBase.scala:113) >> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) >> at org.scalatra.DynamicScope$class.withResponse( >> DynamicScope.scala:80) >> at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet. >> scala:49) >> at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1. >> apply(DynamicScope.scala:60) >> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) >> at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) >> at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) >> at org.scalatra.DynamicScope$class.withRequestResponse( >> DynamicScope.scala:59) >> at org.scalatra.ScalatraServlet.withRequestResponse( >> ScalatraServlet.scala:49) >> at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) >> at org.scalatra.ScalatraServlet.org$scalatra$servlet$ >> ServletBase$$super$handle(ScalatraServlet.scala:49) >> at org.scalatra.servlet.ServletBase$class.handle( >> ServletBase.scala:43) >> at com.cloudera.livy.server.SessionServlet.org$scalatra$ >> MethodOverride$$super$handle(SessionServlet.scala:40) >> at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet.org >> $scalatra$servlet$FileUploadSupport$$super$handle( >> InteractiveSessionServlet.scala:42) >> at org.scalatra.servlet.FileUploadSupport$class. >> handle(FileUploadSupport.scala:93) >> at com.cloudera.livy.server.interactive.InteractiveSessionServlet. >> handle(InteractiveSessionServlet.scala:42) >> at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> at org.eclipse.jetty.servlet.ServletHolder.handle( >> ServletHolder.java:812) >> at org.eclipse.jetty.servlet.ServletHandler.doHandle( >> ServletHandler.java:587) >> at org.eclipse.jetty.server.handler.ContextHandler. >> doHandle(ContextHandler.java:1127) >> at org.eclipse.jetty.servlet.ServletHandler.doScope( >> ServletHandler.java:515) >> at org.eclipse.jetty.server.handler.ContextHandler. >> doScope(ContextHandler.java:1061) >> at org.eclipse.jetty.server.handler.ScopedHandler.handle( >> ScopedHandler.java:141) >> at org.eclipse.jetty.server.handler.HandlerCollection. >> handle(HandlerCollection.java:110) >> at org.eclipse.jetty.server.handler.HandlerWrapper.handle( >> HandlerWrapper.java:97) >> at org.eclipse.jetty.server.Server.handle(Server.java:499) >> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) >> at org.eclipse.jetty.server.HttpConnection.onFillable( >> HttpConnection.java:257) >> at org.eclipse.jetty.io.AbstractConnection$2.run( >> AbstractConnection.java:544) >> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( >> QueuedThreadPool.java:635) >> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( >> QueuedThreadPool.java:555) >> at java.lang.Thread.run(Thread.java:745) >> >
