[
https://issues.apache.org/jira/browse/SPARK-7640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brad Willard updated SPARK-7640:
--------------------------------
Description:
If you create a spark cluster in a private vpc, the amazon yum repos return 403
permission denied because Amazon cannot discern the vms are in their
datacenter. This makes it incredibly annoying to install things like python 2.7
and different compression libs or consider updating anything.
Potential fixes:
Add fedora yum repos on the default ami to ones outside of amazon.
Change the ami to be based on a non amazon ami, like a standard red-hat one.
Switch everything to support ec2-user like most modern aws amis to make it
easier for the user to pick an ami
Petition amazon to open up their repos.
Failed Workaround:
I attempted to use a normal red-hat ami, however the current deploy scripts
assume the user and the install path are root. While the deploy script allows
you to override the user, they don't work if you set ec2-user basically
preventing you from using any current ami other than the default amazon one
which is unfortunate.
So normally this would work, but because amazon 403s you get this if you want
to use python 2.7
$ yum install -y python27.x86_64 python27-devel.x86_64 python27-pip.noarch
Loaded plugins: priorities, security, update-motd, upgrade-helper
http://packages.us-east-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.us-west-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.us-west-2.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.eu-west-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.ap-southeast-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.ap-northeast-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.sa-east-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
http://packages.ap-southeast-2.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
was:
If you create a spark cluster in a private vpc, the amazon yum repos return 403
permission denied because Amazon cannot discern the vms are in their
datacenter. This makes it incredibly annoying to install things like python 2.7
and different compression libs or consider updating anything.
Potential fixes:
Add fedora yum repos on the default ami to ones outside of amazon.
Change the ami to be based on a non amazon ami, like a standard red-hat one.
Switch everything to support ec2-user like most modern aws amis to make it
easier for the user to pick an ami
Petition amazon to open up their repos.
Failed Workaround:
I attempted to use a normal red-hat ami, however the current deploy scripts
assume the user and the install path are root. While the deploy script allows
you to override the user, they don't work if you set ec2-user basically
preventing you from using any current ami other than the default amazon one
which is unfortunate.
> Private VPC with default Spark AMI breaks yum
> ---------------------------------------------
>
> Key: SPARK-7640
> URL: https://issues.apache.org/jira/browse/SPARK-7640
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Affects Versions: 1.3.0, 1.3.1
> Reporter: Brad Willard
> Priority: Minor
>
> If you create a spark cluster in a private vpc, the amazon yum repos return
> 403 permission denied because Amazon cannot discern the vms are in their
> datacenter. This makes it incredibly annoying to install things like python
> 2.7 and different compression libs or consider updating anything.
> Potential fixes:
> Add fedora yum repos on the default ami to ones outside of amazon.
> Change the ami to be based on a non amazon ami, like a standard red-hat one.
> Switch everything to support ec2-user like most modern aws amis to make it
> easier for the user to pick an ami
> Petition amazon to open up their repos.
> Failed Workaround:
> I attempted to use a normal red-hat ami, however the current deploy scripts
> assume the user and the install path are root. While the deploy script allows
> you to override the user, they don't work if you set ec2-user basically
> preventing you from using any current ami other than the default amazon one
> which is unfortunate.
> So normally this would work, but because amazon 403s you get this if you want
> to use python 2.7
> $ yum install -y python27.x86_64 python27-devel.x86_64 python27-pip.noarch
> Loaded plugins: priorities, security, update-motd, upgrade-helper
> http://packages.us-east-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.us-west-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.us-west-2.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.eu-west-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.ap-southeast-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.ap-northeast-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.sa-east-1.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
> http://packages.ap-southeast-2.amazonaws.com/2015.03/main/20150301f40d/x86_64/repodata/repomd.xml:
> [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403
> Forbidden"
> Trying other mirror.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
