Andrew Or created SPARK-9693:
--------------------------------
Summary: Reserve a page in all sorters to avoid starving an
operator
Key: SPARK-9693
URL: https://issues.apache.org/jira/browse/SPARK-9693
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 1.5.0
Reporter: Andrew Or
Assignee: Andrew Or
Priority: Critical
Currently we can do up to 3 sorts within a task:
(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle
In environments with tight memory restrictions, the first operator may acquire
so much memory such that the subsequent ones in the same task are starved. A
simple fix is to reserve at least a page in advance in each of these places.
The reserved page size need not be the same as the normal page size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
