On changing the default compression codec which is snappy to lzf the errors are gone !!
How can I fix this using snappy as the codec ? Is there any downside of using lzf as snappy is the default codec that ships with spark. Thanks !!! Abhi On Mon, Feb 22, 2016 at 7:42 PM, Abhishek Anand <abhis.anan...@gmail.com> wrote: > Hi , > > I am getting the following exception on running my spark streaming job. > > The same job has been running fine since long and when I added two new > machines to my cluster I see the job failing with the following exception. > > > > 16/02/22 19:23:01 ERROR Executor: Exception in task 2.0 in stage 4229.0 > (TID 22594) > java.io.IOException: java.lang.reflect.InvocationTargetException > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1257) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:59) > at org.apache.spark.scheduler.Task.run(Task.scala:70) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:408) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60) > at org.apache.spark.broadcast.TorrentBroadcast.org > $apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1254) > ... 11 more > Caused by: java.lang.IllegalArgumentException > at > org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152) > ... 20 more > > > > Thanks !!! > Abhi >
