Anyone have a chance to look at this? Am I just doing something silly somewhere?
If it makes any difference, I am using the elasticsearch-hadoop plugin for ESInputFormat. But as I say, I can parse the data (count, first() etc). I just can't save it as text file. On Tue, Apr 8, 2014 at 4:50 PM, Nick Pentreath <nick.pentre...@gmail.com>wrote: > Hi > > I'm using Spark 0.9.0. > > When calling saveAsTextFile on a custom hadoop inputformat (loaded with > newAPIHadoopRDD), I get the following error below. > > If I call count, I get the correct count of number of records, so the > inputformat is being read correctly... the issue only appears when trying > to use saveAsTextFile. > > If I call first() I get the correct output, also. So it doesn't appear to > be anything with the data or inputformat. > > Any idea what the actual problem is, since this stack trace is not obvious > (though it seems to be in ResultTask which ultimately causes this). > > Is this a known issue at all? > > > ====== > > 14/04/08 16:00:46 ERROR OneForOneStrategy: > java.lang.NullPointerException > at > com.typesafe.config.impl.SerializedConfigValue.writeOrigin(SerializedConfigValue.java:202) > at > com.typesafe.config.impl.ConfigImplUtil.writeOrigin(ConfigImplUtil.java:228) > at > com.typesafe.config.ConfigException.writeObject(ConfigException.java:58) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) > at scala.collection.immutable.$colon$colon.writeObject(List.scala:379) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) > at > org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:28) > at > org.apache.spark.scheduler.ResultTask$.serializeInfo(ResultTask.scala:48) > at > org.apache.spark.scheduler.ResultTask.writeExternal(ResultTask.scala:123) > at > java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1443) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1414) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) > at > org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:28) > at > org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:48) > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:778) > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:724) > at > org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:554) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > >
