[ https://issues.apache.org/jira/browse/SPARK-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-17491. -------------------------------- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull request 15043 [https://github.com/apache/spark/pull/15043] > MemoryStore.putIteratorAsBytes() may silently lose values when KryoSerializer > is used > ------------------------------------------------------------------------------------- > > Key: SPARK-17491 > URL: https://issues.apache.org/jira/browse/SPARK-17491 > Project: Spark > Issue Type: Bug > Components: Block Manager > Affects Versions: 2.0.0 > Reporter: Josh Rosen > Assignee: Josh Rosen > Priority: Blocker > Labels: correctness > Fix For: 2.0.1, 2.1.0 > > > MemoryStore.putIteratorAsBytes() may silently lose values when used with > KryoSerializer because it does not properly close the serialization stream > before attempting to deserialize the already-serialized values, which may > cause values buffered in Kryo's internal buffers to not be read. > This is the root cause behind a user-reported "wrong answer" bug in PySpark > caching reported by Ben Leslie on the Spark user mailing list in a thread > titled "pyspark persist MEMORY_ONLY vs MEMORY_AND_DISK") -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org