[
https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352489#comment-14352489
]
Nicholas Chammas commented on SPARK-6220:
-----------------------------------------
cc [~joshrosen] and [~shivaram] for feedback.
The immediate motivation for this is the work I'm doing on automating
spark-perf runs.
As part of an automated spark-perf run, I'd like to:
* set {{instance_initiated_shutdown_behavior=terminate}} for the non-spot
instances launched by spark-ec2 (i.e. the master), so that the cluster can
self-terminate without needing outside input
* set {{instance_profile_arn}} for the master so that spark-perf results can be
uploaded to S3 without having to handle AWS user credentials, via use of IAM
profiles
Since my use case is specialized, I didn't think it was worth adding top-level
options for these EC2 features. So I generalized the idea to support any EC2
option supported by boto.
> Allow extended EC2 options to be passed through spark-ec2
> ---------------------------------------------------------
>
> Key: SPARK-6220
> URL: https://issues.apache.org/jira/browse/SPARK-6220
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Nicholas Chammas
> Priority: Minor
>
> There are many EC2 options exposed by the boto library that spark-ec2 uses.
> Over time, many of these EC2 options have been bubbled up here and there to
> become spark-ec2 options.
> Examples:
> * spot prices
> * placement groups
> * VPC, subnet, and security group assignments
> It's likely that more and more EC2 options will trickle up like this to
> become spark-ec2 options.
> While major options are well suited to this type of promotion, we should
> probably allow users to pass through EC2 options they want to use through
> spark-ec2 in some generic way.
> Let's add two options:
> * {{--ec2-instance-option}} ->
> [{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
> * {{--ec2-spot-instance-option}} ->
> [{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]
> Each option can be specified multiple times and is simply passed directly to
> the underlying boto call.
> For example:
> {code}
> spark-ec2 --ec2-instance-option
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
