Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/214#issuecomment-38609414
I thought about this a bit more and I think it makes sense to do something
similar to what @CodingCat suggested: in CoarseGrainedSchedulerBackend, when we
call scheduler.resourceOffers(), we get back a list of tasks but where we don't
have the serialized task yet. At this point, the makeOffers() method passes
the tasks off to a thread pool that serializes the task and then launches it on
the appropriate executor. We could pass the task and executor actor to the
thread pool, so that it doesn't have to synchronize on the CGSB at all (and
this way we don't need to add a new message to CGSB). Does this make sense /
seem reasonable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
