Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/214#issuecomment-38614287
Yeah exactly -- so my proposal was something like, inn CBSG.makeOffers():
-still do scheduler.resourceOffers(), only now this returns unserialized
tasks
-update freeCores (as is currently done in launchTasks)
-add the list of tasks and associated actors to a thread pool that
(asynchronously) serializes and launches each task
Here, freeCores is still being updated by the main CBSG actor thread. The
only thing that's happening in a separate thread is the serialization and
sending a message to the executor actor, neither of which need to be
synchronized on the state in CGSB.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
