cekicbaris edited a comment on pull request #167: URL: https://github.com/apache/incubator-livy/pull/167#issuecomment-698993880
> > Do you mind to give some advice to check the networking? > > I would leave it as the last effort, it's better to start from checking the Driver logs in the moment of the failure and/or more logs from Livy to get the idea on what was happening before and after the failure. Otherwise it is good to contact your network SREs to give the advice on that. They definitely know more about your setup. > > > if the session time'd out , the interactive session is deleted from livy but driver pods and executor pods are still running. > > I remember this bug in this PR, though I believe it has been fixed in #249 . This PR is out of sync and not supported anymore, but once I get some free time of my work I plan to backport it to this PR and the issue should be gone then. Also once I finish with Spark 3 support in the Helm chart I will also update the images which will solve that as well. Please refer: https://github.com/jahstreet/spark-on-kubernetes-helm/tree/spark-3.0.0-upgrade . @jahstreet , thanks for quick reply. Please let me know how I can help/contribute ? In the meantime, I tried to deploy `https://github.com/jahstreet/spark-on-kubernetes-helm/tree/spark-3.0.0-upgrade` chart but the driver failed to launch. Is it because of images not updated yet ? ``` log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected at io.netty.util.concurrent.DefaultPromise.get(DefaultPromise.java:349) at org.apache.livy.rsc.driver.RSCDriver.initializeServer(RSCDriver.java:210) at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:343) at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected at org.apache.livy.shaded.kryo.reflectasm.ConstructorAccess.insertConstructor(ConstructorAccess.java:128) at org.apache.livy.shaded.kryo.reflectasm.ConstructorAccess.get(ConstructorAccess.java:98) at org.apache.livy.shaded.kryo.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1271) at org.apache.livy.shaded.kryo.kryo.Kryo.newInstantiator(Kryo.java:1127) at org.apache.livy.shaded.kryo.kryo.Kryo.newInstance(Kryo.java:1136) at org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.create(FieldSerializer.java:562) at org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.read(FieldSerializer.java:538) at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:813) at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:67) at org.apache.livy.rsc.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:70) at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
