Matthias Boehm created SYSTEMML-2162:
----------------------------------------
Summary: Deserialization of compressed blocks w/ shared dictionary
fails in Spark
Key: SYSTEMML-2162
URL: https://issues.apache.org/jira/browse/SYSTEMML-2162
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
For very large datasets that exceed aggregate cluster memory, we automatically
apply compression. However, in case blocks with shared DDC1 dictionaries are
evicted and subsequently deserialized this deserialization fails with
mismatching number of bytes.
{code}
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
