The issue is resolved. Resolution is little weird but it worked. It was due to scala version mismatch with projects in my package.
On Wed, 20 Jan 2021 at 18:07, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Vikas, > > Are you running this on your local laptop etc or using some IDE etc? > > What is your available memory for Spark? > > Start with minimum set like below > > def spark_session_local(appName): > return SparkSession.builder \ > .master('local[1]') \ > .appName(appName) \ > .enableHiveSupport() \ > .getOrCreate() > > > HTH > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Wed, 20 Jan 2021 at 12:32, Vikas Garg <sperry...@gmail.com> wrote: > >> Hi Sachit, >> >> I am running it in local. My IP mentioned is the private IP address and >> therefore it is useless for anyone. >> >> On Wed, 20 Jan 2021 at 17:37, Sachit Murarka <connectsac...@gmail.com> >> wrote: >> >>> Hi Vikas >>> >>> 1. Are you running in local mode? Master has local[*] >>> 2. Pls mask the ip or confidential info while sharing logs >>> >>> Thanks >>> Sachit >>> >>> On Wed, 20 Jan 2021, 17:35 Vikas Garg, <sperry...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am facing issue with spark executor. I am struggling with this issue >>>> since last many days and unable to resolve the issue. >>>> >>>> Below is the configuration I have given. >>>> >>>> val spark = SparkSession.builder() >>>> .appName("Spark Job") >>>> .master("local[*]") >>>> .config("spark.dynamicAllocation.enabled", true) >>>> .config("spark.shuffle.service.enabled", true) >>>> .config("spark.driver.maxResultSize", "8g") >>>> .config("spark.driver.memory", "8g") >>>> .config("spark.executor.memory", "8g") >>>> .config("spark.network.timeout", "3600s") >>>> .getOrCreate() >>>> >>>> 1/01/20 17:06:57 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.start( >>>> *RetryingBlockFetcher.java:121*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks( >>>> *NettyBlockTransferService.scala:143*) >>>> >>>> at >>>> org.apache.spark.network.BlockTransferService.fetchBlockSync( >>>> *BlockTransferService.scala:103*) >>>> >>>> at >>>> org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer( >>>> *BlockManager.scala:1010*) >>>> >>>> at >>>> org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8( >>>> *BlockManager.scala:954*) >>>> >>>> at scala.Option.orElse(*Option.scala:289*) >>>> >>>> at org.apache.spark.storage.BlockManager.getRemoteBlock( >>>> *BlockManager.scala:954*) >>>> >>>> at org.apache.spark.storage.BlockManager.getRemoteBytes( >>>> *BlockManager.scala:1092*) >>>> >>>> at >>>> org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1( >>>> *TaskResultGetter.scala:88*) >>>> >>>> at >>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) >>>> >>>> at org.apache.spark.util.Utils$.logUncaughtExceptions( >>>> *Utils.scala:1932*) >>>> >>>> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run( >>>> *TaskResultGetter.scala:63*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:06:57 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.start( >>>> *RetryingBlockFetcher.java:121*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks( >>>> *NettyBlockTransferService.scala:143*) >>>> >>>> at >>>> org.apache.spark.network.BlockTransferService.fetchBlockSync( >>>> *BlockTransferService.scala:103*) >>>> >>>> at >>>> org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer( >>>> *BlockManager.scala:1010*) >>>> >>>> at >>>> org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8( >>>> *BlockManager.scala:954*) >>>> >>>> at scala.Option.orElse(*Option.scala:289*) >>>> >>>> at org.apache.spark.storage.BlockManager.getRemoteBlock( >>>> *BlockManager.scala:954*) >>>> >>>> at org.apache.spark.storage.BlockManager.getRemoteBytes( >>>> *BlockManager.scala:1092*) >>>> >>>> at >>>> org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1( >>>> *TaskResultGetter.scala:88*) >>>> >>>> at >>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) >>>> >>>> at org.apache.spark.util.Utils$.logUncaughtExceptions( >>>> *Utils.scala:1932*) >>>> >>>> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run( >>>> *TaskResultGetter.scala:63*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:07:02 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks (after 1 retries) >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>> *RetryingBlockFetcher.java:169*) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>> *Executors.java:511*) >>>> >>>> at java.util.concurrent.FutureTask.run( >>>> *FutureTask.java:266*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:07:02 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks (after 1 retries) >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>> *RetryingBlockFetcher.java:169*) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>> *Executors.java:511*) >>>> >>>> at java.util.concurrent.FutureTask.run( >>>> *FutureTask.java:266*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:07:07 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks (after 2 retries) >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>> *RetryingBlockFetcher.java:169*) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>> *Executors.java:511*) >>>> >>>> at java.util.concurrent.FutureTask.run( >>>> *FutureTask.java:266*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:07:07 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks (after 2 retries) >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>> *RetryingBlockFetcher.java:169*) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>> *Executors.java:511*) >>>> >>>> at java.util.concurrent.FutureTask.run( >>>> *FutureTask.java:266*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketChannel.java:330*) >>>> >>>> at >>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>> *AbstractNioChannel.java:334*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>> *NioEventLoop.java:702*) >>>> >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>> *NioEventLoop.java:650*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>> *NioEventLoop.java:576*) >>>> >>>> at io.netty.channel.nio.NioEventLoop.run( >>>> *NioEventLoop.java:493*) >>>> >>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>> *SingleThreadEventExecutor.java:989*) >>>> >>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>> *ThreadExecutorMap.java:74*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> 21/01/20 17:07:12 ERROR RetryingBlockFetcher: Exception while beginning >>>> fetch of 1 outstanding blocks (after 3 retries) >>>> >>>> *java.io.IOException*: Failed to connect to >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:253*) >>>> >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>> *TransportClientFactory.java:195*) >>>> >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>> *NettyBlockTransferService.scala:122*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>> *RetryingBlockFetcher.java:141*) >>>> >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>> *RetryingBlockFetcher.java:169*) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>> *Executors.java:511*) >>>> >>>> at java.util.concurrent.FutureTask.run( >>>> *FutureTask.java:266*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> *ThreadPoolExecutor.java:1149*) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> *ThreadPoolExecutor.java:624*) >>>> >>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>> *FastThreadLocalRunnable.java:30*) >>>> >>>> at java.lang.Thread.run(*Thread.java:748*) >>>> >>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>> Permission denied: no further information: >>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348 >>>> >>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>> information >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native >>>> Method*) >>>> >>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>> *SocketChannelImpl.java:715*) >>>> >>>> at >>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>> *NioSocketC* >>>> >>>