[
https://issues.apache.org/jira/browse/SPARK-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-9693:
-----------------------------
Description:
E.g. Currently we can do up to 3 sorts within a task:
(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle
In environments with tight memory restrictions, the first operator may acquire
so much memory such that the subsequent ones in the same task are starved. A
simple fix is to reserve at least a page in advance in each of these places.
The reserved page size need not be the same as the normal page size.
was:
Currently we can do up to 3 sorts within a task:
(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle
In environments with tight memory restrictions, the first operator may acquire
so much memory such that the subsequent ones in the same task are starved. A
simple fix is to reserve at least a page in advance in each of these places.
The reserved page size need not be the same as the normal page size.
> Reserve a page in all unsafe operators to avoid starving an operator
> --------------------------------------------------------------------
>
> Key: SPARK-9693
> URL: https://issues.apache.org/jira/browse/SPARK-9693
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Critical
>
> E.g. Currently we can do up to 3 sorts within a task:
> (1) During the aggregation
> (2) During a sort on the same key
> (3) During the shuffle
> In environments with tight memory restrictions, the first operator may
> acquire so much memory such that the subsequent ones in the same task are
> starved. A simple fix is to reserve at least a page in advance in each of
> these places. The reserved page size need not be the same as the normal page
> size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
