Thomas Graves created SPARK-23964: ------------------------------------- Summary: why does Spillable wait for 32 elements? Key: SPARK-23964 URL: https://issues.apache.org/jira/browse/SPARK-23964 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.2.1 Environment: The spillable class has a check:
if (elementsRead % 32 == 0 && currentMemory >= myMemoryThreshold) { Before it looks to see if it should spill. [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/collection/Spillable.scala#L83] I'm wondering why it has the elementsRead %32 in it? If I have a small number of elements that are huge this can easily cause OOM before we actually spill. I saw a few conversations on this and one Jira related: https://issues.apache.org/jira/browse/SPARK-4456 . but I've never seen an answer to this. anyone have history on this? Reporter: Thomas Graves -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org