Thanks for the response Akhil. My email may not have been clear, but my
question is about what should be inside the AMI image, not how to pass an
AMI id in to the spark_ec2 script.
Should certain packages be installed? Do certain directories need to exist?
etc...
On Fri, Jun 6, 2014 at 4:40 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> you can comment out this function and Create a new one which will return
> your ami-id and the rest of the script will run fine.
>
> def get_spark_ami(opts):
> instance_types = {
> "m1.small": "pvm",
> "m1.medium": "pvm",
> "m1.large": "pvm",
> "m1.xlarge": "pvm",
> "t1.micro": "pvm",
> "c1.medium": "pvm",
> "c1.xlarge": "pvm",
> "m2.xlarge": "pvm",
> "m2.2xlarge": "pvm",
> "m2.4xlarge": "pvm",
> "cc1.4xlarge": "hvm",
> "cc2.8xlarge": "hvm",
> "cg1.4xlarge": "hvm",
> "hs1.8xlarge": "hvm",
> "hi1.4xlarge": "hvm",
> "m3.xlarge": "hvm",
> "m3.2xlarge": "hvm",
> "cr1.8xlarge": "hvm",
> "i2.xlarge": "hvm",
> "i2.2xlarge": "hvm",
> "i2.4xlarge": "hvm",
> "i2.8xlarge": "hvm",
> "c3.large": "pvm",
> "c3.xlarge": "pvm",
> "c3.2xlarge": "pvm",
> "c3.4xlarge": "pvm",
> "c3.8xlarge": "pvm"
> }
> if opts.instance_type in instance_types:
> instance_type = instance_types[opts.instance_type]
> else:
> instance_type = "pvm"
> print >> stderr,\
> "Don't recognize %s, assuming type is pvm" % opts.instance_type
>
> ami_path = "%s/%s/%s" % (AMI_PREFIX, opts.region, instance_type)
> try:
> ami = urllib2.urlopen(ami_path).read().strip()
> print "Spark AMI: " + ami
> except:
> print >> stderr, "Could not resolve AMI at: " + ami_path
> sys.exit(1)
>
> return ami
>
> Thanks
> Best Regards
>
>
> On Fri, Jun 6, 2014 at 2:14 AM, Matt Work Coarr <mattcoarr.w...@gmail.com>
> wrote:
>
>> How would I go about creating a new AMI image that I can use with the
>> spark ec2 commands? I can't seem to find any documentation. I'm looking
>> for a list of steps that I'd need to perform to make an Amazon Linux image
>> ready to be used by the spark ec2 tools.
>>
>> I've been reading through the spark 1.0.0 documentation, looking at the
>> script itself (spark_ec2.py), and looking at the github project
>> mesos/spark-ec2.
>>
>> From what I can tell, the spark_ec2.py script looks up the id of the AMI
>> based on the region and machine type (hvm or pvm) using static content
>> derived from the github repo mesos/spark-ec2.
>>
>> The spark ec2 script loads the AMI id from this base url:
>> https://raw.github.com/mesos/spark-ec2/v2/ami-list
>> (Which presumably comes from https://github.com/mesos/spark-ec2 )
>>
>> For instance, I'm working with us-east-1 and pvm, I'd end up with AMI id:
>> ami-5bb18832
>>
>> Is there a list of instructions for how this AMI was created? Assuming
>> I'm starting with my own Amazon Linux image, what would I need to do to
>> make it usable where I could pass that AMI id to spark_ec2.py rather than
>> using the default spark-provided AMI?
>>
>> Thanks,
>> Matt
>>
>
>
