Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/15588
Spark Streaming would do a very simple dummy job ensure that all slaves
have registered before scheduling the `Receiver`s; please see
https://github.com/apache/spark/blob/v2.0.0/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala#L436-L447.
@Astralidea, `spark.scheduler.minRegisteredResourcesRatio` is the minimum
ratio of registered resources to wait for before the dummy job begins.In our
private clusters, configuring that to be `0.9` or even `1.0` helps a lot to
balance our 100+ `Receiver`s. Maybe you could also give it a try.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
