[ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786971#comment-15786971 ]
Sameer Kumar commented on SPARK-18986: -------------------------------------- Shouldn't the priority be increased for this because I am facing this issue on almost every batch interval and the data doesn't get processed any further, which is a significant data loss for any application. > ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its > iterator > ------------------------------------------------------------------------------------- > > Key: SPARK-18986 > URL: https://issues.apache.org/jira/browse/SPARK-18986 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: Liang-Chi Hsieh > > {{ExternalAppendOnlyMap.forceSpill}} now uses an assert to check if an > iterator is not null in the map. However, the assertion is only true after > the map is asked for iterator. Before it, if another memory consumer asks > more memory than currently available, {{ExternalAppendOnlyMap.forceSpill}} is > also be called too. In this case, we will see failure like this: > {code} > [info] java.lang.AssertionError: assertion failed > [info] at scala.Predef$.assert(Predef.scala:156) > [info] at > org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196) > [info] at > org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111) > [info] at > org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly > MapSuite.scala:294) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org