Hi Meisam, Comments below. How many sessions are you creating? Abhi> We had tried creating around 5-8 sessions when we start seeing the below exception. Have you tried to throttle down session creation? Abhi> We do not have any issue if we create sessions one after the other. What is the value for `livy.server.session.max-creation` in your setup? Abhi> We're not setting any value. So it should be using default. I guess it is 100. Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory. Abhi> We did not see any out of memory errors during our tests. But just out of curiosity, we tried to create sessions in an infinite loop. After creating around 35 sessions, we’ve started seeing out of memory errors.
Hi Jerry, We've raised the ticket LIVY-453 for this. Thanks & Regards, Abhishek -----Original Message----- From: Meisam Fathi [mailto:[email protected]] Sent: Wednesday, March 21, 2018 5:41 AM To: [email protected] Subject: Re: Query on creating multiple livy sessions in parallel How many sessions are you creating? Have you tried to throttle down session creation? What is the value for `livy.server.session.max-creation` in your setup? Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory. Thanks, Meisam On Mon, Mar 19, 2018 at 8:15 PM Saisai Shao <[email protected]<mailto:[email protected]>> wrote: > This might be a BUG. If possible can you please create a JIRA to track > this issue. Thanks! > > Best, > Jerry > > 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < > [email protected]<mailto:[email protected]>>: > > > Hi, > > > > We're trying to create multiple livy sessions in parallel and then > > using them. But when we try to create the sessions continuously, > > we're seeing that few sessions are entering to dead state. We see > > the below exception > in > > the logs. > > > > 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly. > > 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC. > > io.netty.util.concurrent.BlockingOperationException: > > DefaultChannelPromise@7a828ea3(uncancellable) > > at > > io.netty.util.concurrent.DefaultPromise.checkDeadLock( > > DefaultPromise.java:390) > > at > > io.netty.channel.DefaultChannelPromise.checkDeadLock( > > DefaultChannelPromise.java:157) > > at io.netty.util.concurrent.DefaultPromise.await( > > DefaultPromise.java:251) > > at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:129) > > at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:28) > > at io.netty.util.concurrent.DefaultPromise.sync( > > DefaultPromise.java:218) > > at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:117) > > at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:28) > > at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307) > > at > com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225) > > at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:122) > > at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:116) > > at com.cloudera.livy.rsc.Utils$2. > > operationComplete(Utils.java:108) > > at > io.netty.util.concurrent.DefaultPromise.notifyListener0( > > DefaultPromise.java:680) > > at > io.netty.util.concurrent.DefaultPromise.notifyListeners( > > DefaultPromise.java:567) > > at > > io.netty.util.concurrent.DefaultPromise.trySuccess( > > DefaultPromise.java:406) > > at io.netty.channel.DefaultChannelPromise.trySuccess( > > DefaultChannelPromise.java:82) > > at io.netty.channel.AbstractChannel$CloseFuture. > > setClosed(AbstractChannel.java:956) > > at io.netty.channel.AbstractChannel$ > > AbstractUnsafe.doClose0(AbstractChannel.java:608) > > at > > io.netty.channel.AbstractChannel$AbstractUnsafe.close( > > AbstractChannel.java:586) > > at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > > at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.read(AbstractNioByteChannel.java:158) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKey( > > NioEventLoop.java:511) > > at io.netty.channel.nio.NioEventLoop. > > processSelectedKeysOptimized(NioEventLoop.java:468) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKeys( > > NioEventLoop.java:382) > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > > java:354) > > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > > run(SingleThreadEventExecutor.java:111) > > at java.lang.Thread.run(Thread.java:748) > > 18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context > > dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false. > > > > However, when we create the sessions one after the other (Create > > session > 1 > > after session 0 is in Idle state), it works fine. > > We wanted to know if there is any known restriction in livy for > > creating multiple sessions in parallel. > > > > Thanks & Regards, > > Abhishek > > > > >
