My big data cluster is being migrated from one cloud subscription to another one which is more secure. since then the issue "ChannelClosedException" happening however there is no change in machine configurations. I have same job/scripts loops multiple times,for one loop the job fails(ChannelClosedException) but for next loop it just works .
Wondering what would be the issue ?Network latency? Thanks, Divya On 20 March 2018 at 03:37, Kunal Khatua <[email protected]> wrote: > The exception in your snippet is basically saying that one of the > communication channels on which the client was communicating with the > server to fetch sever metadata closed. > Usually, this is observed if a system is under load (e.g. many concurrent > queries, etc). We'd need to trace back through a repro of the problem and > see where was the communication channel actually closed. This > can, however, be triggered even by the underlying network layers. Drill > uses the Netty RPC libraries for communication, so there could be a bug > there too. > > Do you have an easy repro for this that we could try? Do provide details > of the setup as well, since beefier machines might not have the same > problem. > > On 3/19/2018 3:57:33 AM, Divya Gehlot <[email protected]> wrote: > Hi , > The file size is quite small in KBs only . > If you could tell me few scenarios when it happens will help me debug it ? > > > > Thanks, > Divya > > On 15 March 2018 at 15:00, Kunal Khatua wrote: > > > There could be multiple reasons for why the ChannelClosedException is > > thrown. What kind of a load are you running on your Drillbits? > > > > There might be resources in the mailing list archives that have touched > > upon solutions for this, so you could look up there as well. > > > > On Wed, Mar 14, 2018 at 11:09 PM, Divya Gehlot > > wrote: > > > > > Hi, > > > At times my drill query throws channel closed exception. > > > then next one works fine . > > > > > > Here is the stacktrace of the error : > > > > > > > java.sql.SQLException: Error when getting server meta > > > > at > > > > org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl.getServerMeta( > > > DrillDatabaseMetaDataImpl.java:178) > > > > at > > > > org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl. > > > getIdentifierQuoteString(DrillDatabaseMetaDataImpl.java:407) > > > > at > > > > sqlline.DatabaseConnection.setCompletions( > DatabaseConnection.java:64) > > > > at sqlline.SqlLine.setCompletions(SqlLine.java:1670) > > > > at sqlline.Commands.connect(Commands.java:1085) > > > > at sqlline.Commands.connect(Commands.java:1015) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java:62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > at > > > > sqlline.ReflectiveCommandHandler.execute( > > ReflectiveCommandHandler.java: > > > 36) > > > > at sqlline.SqlLine.dispatch(SqlLine.java:742) > > > > at sqlline.SqlLine.initArgs(SqlLine.java:528) > > > > at sqlline.SqlLine.begin(SqlLine.java:596) > > > > at sqlline.SqlLine.start(SqlLine.java:375) > > > > at sqlline.SqlLine.main(SqlLine.java:268) > > > > Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel > > > > closed /xx.xx.xx.x:63750 xxxxxxxxx-dn1.xxx.xxxx.com/xx. > > > xx.xx.x:31010. > > > > at > > > > org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler. > > > operationComplete(RpcBus.java:164) > > > > at > > > > org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler. > > > operationComplete(RpcBus.java:144) > > > > at > > > > io.netty.util.concurrent.DefaultPromise.notifyListener0( > > > DefaultPromise.java:680) > > > > at > > > > io.netty.util.concurrent.DefaultPromise.notifyListeners0( > > > DefaultPromise.java:603) > > > > at > > > > io.netty.util.concurrent.DefaultPromise.notifyListeners( > > > DefaultPromise.java:563) > > > > at > > > > io.netty.util.concurrent.DefaultPromise.trySuccess( > > > DefaultPromise.java:406) > > > > at > > > > io.netty.channel.DefaultChannelPromise.trySuccess( > > > DefaultChannelPromise.java:82) > > > > at > > > > io.netty.channel.AbstractChannel$CloseFuture. > > setClosed(AbstractChannel. > > > java:943) > > > > at > > > > io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0( > > > AbstractChannel.java:592) > > > > at > > > > io.netty.channel.AbstractChannel$AbstractUnsafe.close( > > > AbstractChannel.java:584) > > > > at > > > > io.netty.channel.nio.AbstractNioByteChannel$ > NioByteUnsafe.closeOnRead( > > > AbstractNioByteChannel.java:71) > > > > at > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe. > > > handleReadException(AbstractNioByteChannel.java:89) > > > > at > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > > > AbstractNioByteChannel.java:162) > > > > at > > > > io.netty.channel.nio.NioEventLoop.processSelectedKey( > > > NioEventLoop.java:511) > > > > at > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > > > NioEventLoop.java:468) > > > > at > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys( > > > NioEventLoop.java:382) > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > > java:354) > > > > at > > > > io.netty.util.concurrent.SingleThreadEventExecutor$2. > > > run(SingleThreadEventExecutor.java:111) > > > > at java.lang.Thread.run(Thread.java:748) > > > > apache drill 1.10.0 > > > > > > > > > Appreciate the help ! > > > > > > Thanks, > > > Divya > > > > > >
