[ https://issues.apache.org/jira/browse/SPARK-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tathagata Das updated SPARK-8882: --------------------------------- Summary: A New Receiver Scheduling Mechanism to solve unbalanced receivers (was: A New Receiver Scheduling Mechanism) > A New Receiver Scheduling Mechanism to solve unbalanced receivers > ----------------------------------------------------------------- > > Key: SPARK-8882 > URL: https://issues.apache.org/jira/browse/SPARK-8882 > Project: Spark > Issue Type: New Feature > Components: Streaming > Reporter: Shixiong Zhu > Assignee: Shixiong Zhu > Fix For: 1.5.0 > > > There are some problems in the current mechanism: > - If a task fails more than “spark.task.maxFailures” (default: 4) times, the > job will fail. For a long-running Streaming applications, it’s possible that > a Receiver task fails more than 4 times because of Executor lost. > - When an executor is lost, the Receiver tasks on it will be rescheduled. > However, because there may be many Spark jobs at the same time, it’s possible > that TaskScheduler cannot schedule them to make Receivers be distributed > evenly. > To solve such limitations, we need to change the receiver scheduling > mechanism. Here is the design doc: > https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org