[
https://issues.apache.org/jira/browse/SPARK-7640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544066#comment-14544066
]
Brad Willard commented on SPARK-7640:
-------------------------------------
I manually implemented the first one just to get a standard 1.3.1 cluster
working in the private vpc so I don't have to wait until 1.4.0.
However neither of those address the problem that using a private vpc with the
default amazon based spark ami breaks yum because amazon restricts access to
their repos unless it's from a direct connection in their datacenter which the
vpc bypasses.
While the amazon ami might have the best support, I think it would actually be
better in general if the default ami was instead based upon an open source
red-hat one instead. It would also fix my problem because the other ami's repos
are public.
> Private VPC with default Spark AMI breaks yum
> ---------------------------------------------
>
> Key: SPARK-7640
> URL: https://issues.apache.org/jira/browse/SPARK-7640
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Affects Versions: 1.3.0, 1.3.1
> Reporter: Brad Willard
> Priority: Minor
>
> If you create a spark cluster in a private vpc, the amazon yum repos return
> 403 permission denied because Amazon cannot discern the vms are in their
> datacenter. This makes it incredibly annoying to install things like python
> 2.7 and different compression libs or consider updating anything.
> Potential fixes:
> Add fedora yum repos on the default ami to ones outside of amazon.
> Change the ami to be based on a non amazon ami, like a standard red-hat one.
> Switch everything to support ec2-user like most modern aws amis to make it
> easier for the user to pick an ami
> Petition amazon to open up their repos.
> Failed Workaround:
> I attempted to use a normal red-hat ami, however the current deploy scripts
> assume the user and the install path are root. While the deploy script allows
> you to override the user, they don't work if you set ec2-user basically
> preventing you from using any current ami other than the default amazon one
> which is unfortunate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
