Liang-Chi Hsieh created SPARK-18986:
---------------------------------------
Summary: ExternalAppendOnlyMap shouldn't fail when forced to spill
before calling its iterator
Key: SPARK-18986
URL: https://issues.apache.org/jira/browse/SPARK-18986
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Liang-Chi Hsieh
{{ExternalAppendOnlyMap.forceSpill}} now uses an assert to check if an iterator
is not null in the map. However, the assertion is only true after the map is
asked for iterator. Before it, if another memory consumer asks more memory than
currently available, {{ExternalAppendOnlyMap.forceSpill}} is also be called
too. In this case, we will see failure like this:
{code}
[info] java.lang.AssertionError: assertion failed
[info] at scala.Predef$.assert(Predef.scala:156)
[info] at
org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
[info] at
org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
[info] at
org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly
MapSuite.scala:294)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
