Hi Matt,
You will be needing the following on the AMI:
1. Java Installed
2. Root login enabled
3. /mnt should be available (Since all the storage goes here)
Rest of the things spark-ec2 script will set up for you. Let me know if you
need anymore clarification on this.
Thanks
Best Regards
On Fri, Jun 6, 2014 at 6:31 PM, Matt Work Coarr <mattcoarr.w...@gmail.com>
wrote:
> Thanks for the response Akhil. My email may not have been clear, but my
> question is about what should be inside the AMI image, not how to pass an
> AMI id in to the spark_ec2 script.
>
> Should certain packages be installed? Do certain directories need to
> exist? etc...
>
>
> On Fri, Jun 6, 2014 at 4:40 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> you can comment out this function and Create a new one which will return
>> your ami-id and the rest of the script will run fine.
>>
>> def get_spark_ami(opts):
>> instance_types = {
>> "m1.small": "pvm",
>> "m1.medium": "pvm",
>> "m1.large": "pvm",
>> "m1.xlarge": "pvm",
>> "t1.micro": "pvm",
>> "c1.medium": "pvm",
>> "c1.xlarge": "pvm",
>> "m2.xlarge": "pvm",
>> "m2.2xlarge": "pvm",
>> "m2.4xlarge": "pvm",
>> "cc1.4xlarge": "hvm",
>> "cc2.8xlarge": "hvm",
>> "cg1.4xlarge": "hvm",
>> "hs1.8xlarge": "hvm",
>> "hi1.4xlarge": "hvm",
>> "m3.xlarge": "hvm",
>> "m3.2xlarge": "hvm",
>> "cr1.8xlarge": "hvm",
>> "i2.xlarge": "hvm",
>> "i2.2xlarge": "hvm",
>> "i2.4xlarge": "hvm",
>> "i2.8xlarge": "hvm",
>> "c3.large": "pvm",
>> "c3.xlarge": "pvm",
>> "c3.2xlarge": "pvm",
>> "c3.4xlarge": "pvm",
>> "c3.8xlarge": "pvm"
>> }
>> if opts.instance_type in instance_types:
>> instance_type = instance_types[opts.instance_type]
>> else:
>> instance_type = "pvm"
>> print >> stderr,\
>> "Don't recognize %s, assuming type is pvm" % opts.instance_type
>>
>> ami_path = "%s/%s/%s" % (AMI_PREFIX, opts.region, instance_type)
>> try:
>> ami = urllib2.urlopen(ami_path).read().strip()
>> print "Spark AMI: " + ami
>> except:
>> print >> stderr, "Could not resolve AMI at: " + ami_path
>> sys.exit(1)
>>
>> return ami
>>
>> Thanks
>> Best Regards
>>
>>
>> On Fri, Jun 6, 2014 at 2:14 AM, Matt Work Coarr <mattcoarr.w...@gmail.com
>> > wrote:
>>
>>> How would I go about creating a new AMI image that I can use with the
>>> spark ec2 commands? I can't seem to find any documentation. I'm looking
>>> for a list of steps that I'd need to perform to make an Amazon Linux image
>>> ready to be used by the spark ec2 tools.
>>>
>>> I've been reading through the spark 1.0.0 documentation, looking at the
>>> script itself (spark_ec2.py), and looking at the github project
>>> mesos/spark-ec2.
>>>
>>> From what I can tell, the spark_ec2.py script looks up the id of the AMI
>>> based on the region and machine type (hvm or pvm) using static content
>>> derived from the github repo mesos/spark-ec2.
>>>
>>> The spark ec2 script loads the AMI id from this base url:
>>> https://raw.github.com/mesos/spark-ec2/v2/ami-list
>>> (Which presumably comes from https://github.com/mesos/spark-ec2 )
>>>
>>> For instance, I'm working with us-east-1 and pvm, I'd end up with AMI id:
>>> ami-5bb18832
>>>
>>> Is there a list of instructions for how this AMI was created? Assuming
>>> I'm starting with my own Amazon Linux image, what would I need to do to
>>> make it usable where I could pass that AMI id to spark_ec2.py rather than
>>> using the default spark-provided AMI?
>>>
>>> Thanks,
>>> Matt
>>>
>>
>>
>
