Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4155#issuecomment-71252591
I had this (unbased) notion that tasks knew whether they were speculative
or not, and thus the "non-speculative" ones would be able to avoid this extra
hop to the driver and just commit things. But it seems that's not the case (and
it sort of makes sense, in case the "speculative" task finishes first), so I
guess this approach is fine.
One thing that worries me a bit is that I've been told before that akka
actors' `onReceive` methods are single-threaded (meaning they'll never be
called concurrently, even for messages coming from different remote endpoints).
That can become a bottleneck on really large jobs. If that's really true, we
should probably look at decoupling the processing of the message from the
`onReceive` method so that multiple executors can be serviced concurrently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
