Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2836#issuecomment-59552268
I like the idea behind this, but I'm worried about adding even more stuff
to the `spark-ec2` script, especially since I think this use case could be
addressed by a more general "post startup" hook or script.
I think that you can already install pip packages using `pssh`, e.g.
```bash
pip install numpy # Install on master
pssh -h /root/spark-ec2/slaves pip install numpy # Install on workers
```
If we run these commands inside of `spark-ec2`, then what happens if one of
them fails? What if I want to pass more configuration options to pip other
than what the `spark-ec2` wrapper supports?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
