[
https://issues.apache.org/jira/browse/HADOOP-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-3565:
------------------------------
Attachment: hadoop-3565.patch
This happens because java.io.ObjectOutputStream can write back-references to
instances previously serialized in the stream, and after the MapReduce shuffle
the references can appear before the instances they point to under some
circumstances.
Here's a patch to fix this, which writes unshared objects, and also resets the
stream to clear shared classes. I've written a unit test too, which fails
without this fix.
> JavaSerialization can throw java.io.StreamCorruptedException
> ------------------------------------------------------------
>
> Key: HADOOP-3565
> URL: https://issues.apache.org/jira/browse/HADOOP-3565
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0, 0.18.0
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.1, 0.18.0
>
> Attachments: hadoop-3565.patch
>
>
> When using JavaSerialization I sometimes get:
> {noformat}
> Caused by: java.io.StreamCorruptedException: illegal handle value
> at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1285)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
> at
> org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
> ze(JavaSerialization.java:53)
> at
> org.apache.hadoop.io.serializer.JavaSerialization$JavaSerializationDeserializer.deseriali
> ze(JavaSerialization.java:36)
> at
> org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:735)
> at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:675)
> ... 4 more
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
