Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/4467#issuecomment-73831487
The state behavior should be:
* If state = Initialized, then disallow all actions. (actor is not
initialized yet)
* If state = Started, then allow all actions {register, addblock, report,
deregister}.
* If state = Stopping, then disallow {register, addblock}, allow {report,
deregister}.
* If state = Stopped, then disallow all actions. (actor is destroyed)
For the receivers that are still starting and have not registered yet, we
have two options.
1. As you said, we have to wait for them to all be started.
2. We ignore their register messages and let them timeout. So they simply
fail their starting process and don't process any data.
I think both options guarantee no data would be lost. I was thinking using
option 2 in this pr. Because it should be more simple and, semantically we
should not allow receivers to register and then process data after stop is
called.
I just realized that the current implementation of `ReceiverSupervisor`
calls `receiver.onStart` before sending `RegisterReceiver` message. Thus it is
possible the receiver are already started and process data. I think it is
incorrect behavior. Correct one should be getting registered first and then
begin to start the receiver. Otherwise, the receiver may process data and store
data before it has registered successfully. It might cause some problems hard
to detect. We should change the order so it register with tracer first then
begin starting procedure.
The important reason I think we don't choose option 1 to wait for receivers
to all be started is, from the tracker's aspect, it has not idea what receivers
are started or not. It just asynchronously waits for them to register and
deregister. The receivers are visible to the tracker only when they are
registered with it. When it is going to stop, because it doesn't know if there
are receivers started but not registered yet, so it doesn't know how longer it
should wait for them. Thus it is safer to make sure that the receivers must
register before they start.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
