Re: New Amazon AMIs for EC2 script
You should look into AWS EMR instead, with adding pip install steps to the launch process. They have a pretty nice Jupyter notebook script that setups up jupyter and lets you choose what packages you want to install - https://aws.amazon.com/blogs/big-data/running-jupyter-notebook-and-jupyterhub-on-amazon-emr/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/New-Amazon-AMIs-for-EC2-script-tp28419p28421.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: New Amazon AMIs for EC2 script
spark-ec2 has moved to GitHub and is no longer part of the Spark project. A related issue from the current issue tracker that you may want to follow/comment on is this one: https://github.com/amplab/spark-ec2/issues/74 As I said there, I think requiring custom AMIs is one of the major maintenance headaches of spark-ec2. I solved this problem in my own project, Flintrock <https://github.com/nchammas/flintrock>, by working with the default Amazon Linux AMIs and letting people more freely bring their own AMI. Nick On Thu, Feb 23, 2017 at 7:23 AM in4maniac wrote: > Hyy all, > > I have been using the EC2 script to launch R&D pyspark clusters for a while > now. As we use alot of packages such as numpy and scipy with openblas, > scikit-learn, bokeh, vowpal wabbit, pystan and etc... All this time, we > have > been building AMIs on top of the standard spark-AMIs at > https://github.com/amplab/spark-ec2/tree/branch-1.6/ami-list/us-east-1 > > Mainly, I have done the following: > - updated yum > - Changed the standard python to python 2.7 > - changed pip to 2.7 and installed alot of libararies on top of the > existing > AMIs and created my own AMIs to avoid having to boostrap. > > But the ec-2 standard AMIs are from *Early February , 2014* and now have > become extremely fragile. For example, when I update a certain library, > ipython would break, or pip would break and so forth. > > Can someone please direct me to a more upto date AMI that I can use with > more confidence. And I am also interested to know what things need to be in > the AMI, if I wanted to build an AMI from scratch (Last resort :( ) > > And isn't it time to have a ticket in the spark project to build a new > suite > of AMIs for the EC2 script? > https://issues.apache.org/jira/browse/SPARK-922 > > Many thanks > in4maniac > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/New-Amazon-AMIs-for-EC2-script-tp28419.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
New Amazon AMIs for EC2 script
Hyy all, I have been using the EC2 script to launch R&D pyspark clusters for a while now. As we use alot of packages such as numpy and scipy with openblas, scikit-learn, bokeh, vowpal wabbit, pystan and etc... All this time, we have been building AMIs on top of the standard spark-AMIs at https://github.com/amplab/spark-ec2/tree/branch-1.6/ami-list/us-east-1 Mainly, I have done the following: - updated yum - Changed the standard python to python 2.7 - changed pip to 2.7 and installed alot of libararies on top of the existing AMIs and created my own AMIs to avoid having to boostrap. But the ec-2 standard AMIs are from *Early February , 2014* and now have become extremely fragile. For example, when I update a certain library, ipython would break, or pip would break and so forth. Can someone please direct me to a more upto date AMI that I can use with more confidence. And I am also interested to know what things need to be in the AMI, if I wanted to build an AMI from scratch (Last resort :( ) And isn't it time to have a ticket in the spark project to build a new suite of AMIs for the EC2 script? https://issues.apache.org/jira/browse/SPARK-922 Many thanks in4maniac -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/New-Amazon-AMIs-for-EC2-script-tp28419.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org