Tathagata Das created SPARK-3283:
------------------------------------
Summary: Receivers sometimes do not get spread out to multiple
nodes
Key: SPARK-3283
URL: https://issues.apache.org/jira/browse/SPARK-3283
Project: Spark
Issue Type: Bug
Components: Streaming
Reporter: Tathagata Das
Assignee: Tathagata Das
The probable reason this happens is because the JobGenerator and JobScheduler
start generating jobs with tasks. When the ReceiverTracker submits the task
containing receivers, the tasks get assigned according to empty slots, which
may be instantaneously available on one node, instead of all the nodes.
The original behavior was that the jobs started only after the receivers are
started, thus ensuring that all the slots are free and the receivers are spread
evenly across all the nodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
