Hi Lewis, For debugging purpose, can you try using HttpBroadCast to see if the error remains? You can enable HttpBroadCast by setting spark.broadcast.factory to org.apache.spark.broadcast.HttpBroadcastFactory in spark conf.
Thanks, Liquan On Wed, Oct 8, 2014 at 11:21 AM, Steve Lewis <lordjoe2...@gmail.com> wrote: > I am running on Windows 8 using Spark 1.1.0 in local mode with Hadoop 2.2 > - I repeatedly see > the following in my logs. > > I believe this happens in combineByKey > > > 14/10/08 09:36:30 INFO executor.Executor: Running task 3.0 in stage 0.0 > (TID 3) > 14/10/08 09:36:30 INFO broadcast.TorrentBroadcast: Started reading > broadcast variable 0 > 14/10/08 09:36:35 ERROR broadcast.TorrentBroadcast: Reading broadcast > variable 0 failed > 14/10/08 09:36:35 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 0 took 5.006378813 s > 14/10/08 09:36:35 INFO broadcast.TorrentBroadcast: Started reading > broadcast variable 0 > 14/10/08 09:36:35 ERROR executor.Executor: Exception in task 0.0 in stage > 0.0 (TID 0) > java.lang.NullPointerException > at java.nio.ByteBuffer.wrap(ByteBuffer.java:392) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) > at org.apache.spark.scheduler.Task.run(Task.scala:54) > > - > -- Liquan Pei Department of Physics University of Massachusetts Amherst
