[
https://issues.apache.org/jira/browse/SPARK-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoukang updated SPARK-23129:
-----------------------------
Description:
Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init
when DiskMapIterator instance created.This will cause memory use overhead when
ExternalAppendOnlyMap spill too many times.
We can avoid this by making deserializeStream init when it is used the first
time.
was:
Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init
when DiskMapIterator instance created.This will cause memory use overhead when
ExternalAppendOnlyMap spill too much times.
We can avoid this by making deserializeStream init when it is used the first
time.
> Lazy init DiskMapIterator#deserializeStream to reduce memory usage when
> ExternalAppendOnlyMap spill too many times
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-23129
> URL: https://issues.apache.org/jira/browse/SPARK-23129
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: zhoukang
> Assignee: zhoukang
> Priority: Major
> Fix For: 2.3.0
>
>
> Currently,the deserializeStream in ExternalAppendOnlyMap#DiskMapIterator init
> when DiskMapIterator instance created.This will cause memory use overhead
> when ExternalAppendOnlyMap spill too many times.
> We can avoid this by making deserializeStream init when it is used the first
> time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
