Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/7276#issuecomment-123400104
I reorganized ReceiverTracker to use an event loop, including:
1. Add fields into ReceiverTrackingInfo to contain all necessary
information for ReceiverInfo. When sending ReceiverInfo to StreamingListener,
create ReceiverInfo from ReceiverTrackingInfo directly.
2. Move methods that use ReceiverTrackingInfo into ReceiverTrackerEndpoint
so that we can maintain all ReceiverTrackingInfos in an event loop.
3. Move ReceiverTrackingInfo to a separate file ReceiverTrackingInfo.scala.
4. Address some code style issues
> how does it behave when there are 100 executors, and 100 receivers need
to be scheduled in the beginning. ideally there should be 1 receiver per
executor
Assume we only have 2 executors and 2 receivers. The current scheduling
policy will always return the 2 executors. Then it's possible that 2 receivers
run in the same executor. But if returning only 1 executor, we may keep
restarting receivers if the location does not match.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
