[
https://issues.apache.org/jira/browse/SPARK-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-8882:
---------------------------------
Summary: A New Receiver Scheduling Mechanism to solve unbalanced receivers
(was: A New Receiver Scheduling Mechanism)
> A New Receiver Scheduling Mechanism to solve unbalanced receivers
> -----------------------------------------------------------------
>
> Key: SPARK-8882
> URL: https://issues.apache.org/jira/browse/SPARK-8882
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
> Fix For: 1.5.0
>
>
> There are some problems in the current mechanism:
> - If a task fails more than “spark.task.maxFailures” (default: 4) times, the
> job will fail. For a long-running Streaming applications, it’s possible that
> a Receiver task fails more than 4 times because of Executor lost.
> - When an executor is lost, the Receiver tasks on it will be rescheduled.
> However, because there may be many Spark jobs at the same time, it’s possible
> that TaskScheduler cannot schedule them to make Receivers be distributed
> evenly.
> To solve such limitations, we need to change the receiver scheduling
> mechanism. Here is the design doc:
> https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
