[
https://issues.apache.org/jira/browse/SPARK-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Shreedharan updated SPARK-3110:
------------------------------------
Issue Type: Sub-task (was: Bug)
Parent: SPARK-3129
> Add a "ha" mode in YARN mode to keep executors in between restarts
> ------------------------------------------------------------------
>
> Key: SPARK-3110
> URL: https://issues.apache.org/jira/browse/SPARK-3110
> Project: Spark
> Issue Type: Sub-task
> Reporter: Hari Shreedharan
>
> The idea is for long running processes like streaming, you'd want the AM to
> come back up and reuse the same executors, so it can get the blocks from the
> memory of the executors because many streaming systems like Flume cannot
> really replay the data once it has been taken out. Even for others which can,
> the time period before data "expires" can mean some data could be lost. This
> is the first step in a series of patches for this one. The next is to get the
> AM to find the executors. My current plan is to use HDFS to keep track of
> where the executors are running and then communicate to them via Akka, to get
> a block list.
> I plan to expose this via SparkSubmit as the last step once we have all of
> the other pieces in place.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
