[
https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183390#comment-14183390
]
Nicholas Chammas commented on SPARK-3821:
-----------------------------------------
Going for something like EMR's CLI is potentially very useful, though perhaps a
bit outside the scope of the original {{spark-ec2}} (and there's nothing wrong
with that!).
What I'm doing will keep {{spark-ec2}} mostly as-is on the surface, but tackle
the launch times and parallelism as you described.
I'm currently only generating AMIs with Hadoop 2 and Spark 1.1.0, or a base AMI
with everything except Hadoop and Spark. I haven't yet figured out the details
of how to handle the full version matrix. Right now I'm leaning towards having
a "base" AMI that any version of Spark can be installed on relatively quickly
and AMIs for specific versions of Spark starting from 1.1.0.
> Develop an automated way of creating Spark images (AMI, Docker, and others)
> ---------------------------------------------------------------------------
>
> Key: SPARK-3821
> URL: https://issues.apache.org/jira/browse/SPARK-3821
> Project: Spark
> Issue Type: Improvement
> Components: Build, EC2
> Reporter: Nicholas Chammas
> Assignee: Nicholas Chammas
>
> Right now the creation of Spark AMIs or Docker containers is done manually.
> With tools like [Packer|http://www.packer.io/], we should be able to automate
> this work, and do so in such a way that multiple types of machine images can
> be created from a single template.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
