Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1680#issuecomment-50798724
@davies I hadn't heard about this Xen fork() performance issue before;
thanks for pointing it out!
Both before and after this patch, we still fork() the same number of
processes when launching a set of workers; the only difference here is that
those fork()s are issued serially rather than in parallel. Do you know whether
these fork()s benefit from a parallel speedup? The old code probably has a
higher latency to launch the very first set of tasks, since it has to serially
fork() the intermediate layer of processes that this patch removes.
Keep in mind that we're fork()ing a process with an extremely small heap,
so it may not actually be that expensive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
