Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2339#issuecomment-55481878
Hey @nchammas - I'm a bit worried about the behavior here when launching
large clusters. We sometimes launch several-hundred-node ec2 clusters. So say I
launch a 300 node cluster. This approach will, every 3 seconds, fork 300
sub-processes and try to open 300 ssh client sessions from a user's machine.
For every 1 minute of polling, that's 6,000 sub-processes created.
Is that a potential concern for the forking activity and/or the network
activity? Previously it looks like we would run `i.update()` every 5 seconds...
I suppose that created an HTTP connection to the AWS, but it wouldn't fork a
subprocess.
Do you have any idea how long it takes to fork the sub-process and SSH into
the machine?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
