[
https://issues.apache.org/jira/browse/SPARK-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
uncleGen updated SPARK-2773:
----------------------------
Description: Right now, Spark uses the total usage of "shuffle" memory of
each thread to predict if need to spill. I think it is not very reasonable. For
example, there are two threads pulling "shuffle" data. The total memory used to
buffer data is 21G. The first time to trigger spilling it when one thread has
used 7G memory to buffer "shuffle" data, here I assume another one has used the
same size. Unfortunately, I still have remaining 7G to use. So, I think current
prediction mode is too conservative, and can not maximize the usage of
"shuffle" memory. In my solution, I use the growth rate of "shuffle" memory.
Again, the growth of each time is limited, maybe 10K * 1024(my assumption),
then the first time to trigger spilling is when the remaining "shuffle" memory
is less than threads * growth * 2, i.e. 2 * 10M * 2. I think it can maximize
the usage of "shuffle" memory. In my solution, there is also a conservative
assumption, i.e. all of threads is pulling shuffle data in one executor.
However it dose not have much effect, the grow is limited after all. Any
suggestion? (was: Right now, Spark uses the total usage of "shuffle" memory of
each thread to predict if need to spill. I think it is not very reasonable. For
example, there are two threads pulling "shuffle" data. The total memory used to
buffer data is 21G. The first time to trigger spilling it when one thread has
used 7G memory to buffer "shuffle" data, here I assume another one has used the
same size. Unfortunately, I still have remaining 7G to use. So, I think current
prediction mode is too conservation, and can not maximize the usage of
"shuffle" memory. In my solution, I use the growth rate of "shuffle" memory.
Again, the growth of each time is limited, maybe 10K * 1024(my assumption),
then the first time to trigger spilling is when the remaining "shuffle" memory
is less than threads * growth * 2, i.e. 2 * 10M * 2. I think it can maximize
the usage of "shuffle" memory. Any suggestion?)
> Shuffle：use growth rate to predict if need to spill
> ---------------------------------------------------
>
> Key: SPARK-2773
> URL: https://issues.apache.org/jira/browse/SPARK-2773
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 0.9.0, 1.0.0
> Reporter: uncleGen
> Priority: Minor
>
> Right now, Spark uses the total usage of "shuffle" memory of each thread to
> predict if need to spill. I think it is not very reasonable. For example,
> there are two threads pulling "shuffle" data. The total memory used to buffer
> data is 21G. The first time to trigger spilling it when one thread has used
> 7G memory to buffer "shuffle" data, here I assume another one has used the
> same size. Unfortunately, I still have remaining 7G to use. So, I think
> current prediction mode is too conservative, and can not maximize the usage
> of "shuffle" memory. In my solution, I use the growth rate of "shuffle"
> memory. Again, the growth of each time is limited, maybe 10K * 1024(my
> assumption), then the first time to trigger spilling is when the remaining
> "shuffle" memory is less than threads * growth * 2, i.e. 2 * 10M * 2. I think
> it can maximize the usage of "shuffle" memory. In my solution, there is also
> a conservative assumption, i.e. all of threads is pulling shuffle data in one
> executor. However it dose not have much effect, the grow is limited after
> all. Any suggestion?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
