That can't be – Externalizable is a specific sub-interface of Serializable that allows custom serialization formats.
Granted, I could accomplish the same thing by implementing the private readObject() and writeObject() methods in this class, and implement Serializable only. It just seems odd to me that I'd have to do so. Especially since the tuning guide suggests to use Externalizable: http://spark.incubator.apache.org/docs/latest/tuning.html -Matt Cheah From: Andrew Ash <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, December 2, 2013 10:45 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Serializable incompatible with Externalizable error At least from http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java<https://urldefense.proofpoint.com/v1/url?u=http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=gxvgJndY02bAG2cHbPl1cUTcd%2FLzFGz7wtfiAfRKPpk%3D%0A&m=1AzN%2FqpvE4RUjLF0SQcqdBp8GpAKxHbhF7JFlnvwkHI%3D%0A&s=ca3c6dbf7a4045601fbd4b5634863075aca6f5dda2606bb01b34eb47f49eb225> it looks like Externalizable is roughly an old-java version of Serializable. Does that class implement both interfaces? Can you take away the Externalizable interface if it's being used? On Mon, Dec 2, 2013 at 7:15 PM, Matt Cheah <[email protected]<mailto:[email protected]>> wrote: Hi everyone, I'm running into a case where I'm creating a Java RDD of an Externalizable class, and getting this stack trace: java.io.InvalidClassException (java.io.InvalidClassException: com.palantir.finance.datatable.server.spark.WritableDataRow; Serializable incompatible with Externalizable) java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634)java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)java.io.ObjectInputStream.readClass(ObjectInputStream.java:1483)<some other Java stuff>java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153) I'm running on a spark cluster generated by the EC2 Scripts. This doesn't happen if I'm running things with local[N]. Any ideas? Thanks, -Matt Cheah
