[
https://issues.apache.org/jira/browse/SPARK-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576235#comment-14576235
]
Yuri Makhno commented on SPARK-8147:
------------------------------------
[~srowen] exception in iterator will fail spark job and jvm will be able to
clean all resources that are allocated for this job. OutOfMemoryError is on
another level and spark executor can't recover from it
> Add ability to decorate RDD iterators
> -------------------------------------
>
> Key: SPARK-8147
> URL: https://issues.apache.org/jira/browse/SPARK-8147
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.3.1
> Reporter: Yuri Makhno
>
> In Spark all computations are done through iterators which are created by
> RDD.iterator method. It would be good if we can specify some
> RDDIteratorDecoratorFactory in SparkConf and be able to decorate all RDD
> iterators created in executor JVM.
> For us it would be extremely useful because we want to control executor's
> memory and prevent OutOfMemory on executor but instead fail job with
> NotEnoughMemory reason in case when we see that we don't have more memory to
> do this. Also we want to collect some computation statistics on executor.
> I can provide PR in case this improvement is approved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
