Andrew Or created SPARK-15357: --------------------------------- Summary: Cooperative spilling should check consumer memory mode Key: SPARK-15357 URL: https://issues.apache.org/jira/browse/SPARK-15357 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.6.0 Reporter: Andrew Or
In TaskMemoryManager.java: {code} for (MemoryConsumer c: consumers) { if (c != consumer && c.getUsed() > 0) { try { long released = c.spill(required - got, consumer); if (released > 0 && mode == tungstenMemoryMode) { logger.debug("Task {} released {} from {} for {}", taskAttemptId, Utils.bytesToString(released), c, consumer); got += memoryManager.acquireExecutionMemory(required - got, taskAttemptId, mode); if (got >= required) { break; } } } catch (IOException e) { ... } } } {code} Currently, when non-tungsten consumers acquire execution memory, they may force other tungsten consumers to spill and then NOT use the freed memory. A better way to do this is to incorporate the memory mode in the consumer itself and spill only those with matching memory modes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org