Hi Ben,
This is great! I just spun up an EC2 cluster and tested basic pyspark +
ipython/numpy/scipy functionality, and all seems to be working so far. Will let
you know if any issues arise.
We do a lot with pyspark + scientific computing, and for EC2 usage I think this
is a terrific way to
Hi All,
Thanks to Jey's help, I have a release AMI candidate for
spark-1.0/anaconda-2.0 integration. It's currently limited to availability
in US-EAST: ami-3ecd0c56
Give it a try if you have some time. This should* just work* with spark
1.0:
./spark-ec2 -k my_key -i ~/.ssh/mykey.rsa -a
Hi All,
I'm a dev a Continuum and we are developing a fair amount of tooling around
Spark. A few days ago someone expressed interest in numpy+pyspark and
Anaconda came up as a reasonable solution.
I spent a number of hours yesterday trying to rework the base Spark AMI on
EC2 but sadly was
Hi Ben,
Has the PYSPARK_PYTHON environment variable been set in
spark/conf/spark-env.sh to the path of the new python binary?
FYI, there's a /root/copy-dirs script that can be handy when updating
files on an already-running cluster. You'll want to restart the spark
cluster for the changes to
