This discussion seems to indicate the possibility of a mismatch between one side being Serializable and the other being Externalizable: https://forums.oracle.com/thread/2147644
In general, the semantics of Serializable can be pretty strange as it doesn't really behave the same as usual interfaces. It may actually be possible that if a superclass implements Serializable and you implement Externalizable, things can screw up, for instance. On Tue, Dec 3, 2013 at 10:48 AM, Matt Cheah <[email protected]> wrote: > That can't be – Externalizable is a specific sub-interface of > Serializable that allows custom serialization formats. > > Granted, I could accomplish the same thing by implementing the private > readObject() and writeObject() methods in this class, and implement > Serializable only. It just seems odd to me that I'd have to do so. > Especially since the tuning guide suggests to use Externalizable: > http://spark.incubator.apache.org/docs/latest/tuning.html > > -Matt Cheah > > From: Andrew Ash <[email protected]> > Reply-To: "[email protected]" < > [email protected]> > Date: Monday, December 2, 2013 10:45 PM > To: "[email protected]" <[email protected]> > Subject: Re: Serializable incompatible with Externalizable error > > At least from > http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java<https://urldefense.proofpoint.com/v1/url?u=http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=gxvgJndY02bAG2cHbPl1cUTcd%2FLzFGz7wtfiAfRKPpk%3D%0A&m=1AzN%2FqpvE4RUjLF0SQcqdBp8GpAKxHbhF7JFlnvwkHI%3D%0A&s=ca3c6dbf7a4045601fbd4b5634863075aca6f5dda2606bb01b34eb47f49eb225>it > looks like Externalizable is roughly an old-java version of > Serializable. Does that class implement both interfaces? Can you take > away the Externalizable interface if it's being used? > > > On Mon, Dec 2, 2013 at 7:15 PM, Matt Cheah <[email protected]> wrote: > >> Hi everyone, >> >> I'm running into a case where I'm creating a Java RDD of an >> Externalizable class, and getting this stack trace: >> >> java.io.InvalidClassException (java.io.InvalidClassException: >> com.palantir.finance.datatable.server.spark.WritableDataRow; Serializable >> incompatible with Externalizable) >> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634) >> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) >> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) >> java.io.ObjectInputStream.readClass(ObjectInputStream.java:1483)<some >> other Java stuff> >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39) >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61) >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153) >> I'm running on a spark cluster generated by the EC2 Scripts. This doesn't >> happen if I'm running things with local[N]. Any ideas? >> Thanks, >> -Matt Cheah >> > >
