GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3737
[SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi
instead of packaging it
This patch upgrades `spark-ec2`'s Boto version to 2.34.0, since this is
blocking several features. Newer versions of Boto don't work properly when
they're loaded from a zipfile since they try to read a JSON file from a path
relative to the Boto library sources.
Therefore, this patch also changes spark-ec2 to automatically download Boto
from PyPi if it's not present in `SPARK_EC2_DIR/lib`, similar to what we do in
the `sbt/sbt` script. This shouldn't ben an issue for users since they already
need to have an internet connection to launch an EC2 cluster. By performing
the downloading in spark_ec2.py instead of the Bash script, this should also
work for Windows users.
I've tested this with Python 2.6, too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark update-boto
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3737.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3737
----
commit 587ae892a1effca5c0ef779f72de6db283916d91
Author: Josh Rosen <[email protected]>
Date: 2014-12-18T21:42:20Z
[SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi
instead of packaging it
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
