[
https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Chammas updated SPARK-6220:
------------------------------------
Description:
There are many EC2 options exposed by the boto library that spark-ec2 uses.
Over time, many of these EC2 options have been bubbled up here and there to
become spark-ec2 options.
Examples:
* spot prices
* placement groups
* VPC, subnet, and security group assignments
It's likely that more and more EC2 options will trickle up like this to become
spark-ec2 options.
While major options are well suited to this type of promotion, we should
probably allow users to pass through EC2 options they want to use through
spark-ec2 in some generic way.
Let's add two options:
* {{--ec2-instance-option}} ->
[{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
* {{--ec2-spot-instance-option}} ->
[{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]
Each option can be specified multiple times and is simply passed directly to
the underlying boto call.
For example:
{code}
spark-ec2 \
...
--ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
--ec2-instance-option "ebs_optimized=True"
{code}
I'm not sure about the exact syntax of the extended options, but something like
this will do the trick as long as it can be made to pass the options correctly
to boto in most cases.
I followed the example of {{ssh}}, which supports multiple extended options
similarly.
{code}
ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
{code}
was:
There are many EC2 options exposed by the boto library that spark-ec2 uses.
Over time, many of these EC2 options have been bubbled up here and there to
become spark-ec2 options.
Examples:
* spot prices
* placement groups
* VPC, subnet, and security group assignments
It's likely that more and more EC2 options will trickle up like this to become
spark-ec2 options.
While major options are well suited to this type of promotion, we should
probably allow users to pass through EC2 options they want to use through
spark-ec2 in some generic way.
Let's add two options:
* {{--ec2-instance-option}} ->
[{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
* {{--ec2-spot-instance-option}} ->
[{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]
Each option can be specified multiple times and is simply passed directly to
the underlying boto call.
For example:
{code}
spark-ec2 \
...
--ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
--ec2-instance-option "ebs_optimized=True"
{code}
I'm not sure about the exact syntax of the extended options, but something like
this will do the trick.
I followed the example of {{ssh}}, which supports multiple extended options
similarly.
{code}
ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
{code}
> Allow extended EC2 options to be passed through spark-ec2
> ---------------------------------------------------------
>
> Key: SPARK-6220
> URL: https://issues.apache.org/jira/browse/SPARK-6220
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Nicholas Chammas
> Priority: Minor
>
> There are many EC2 options exposed by the boto library that spark-ec2 uses.
> Over time, many of these EC2 options have been bubbled up here and there to
> become spark-ec2 options.
> Examples:
> * spot prices
> * placement groups
> * VPC, subnet, and security group assignments
> It's likely that more and more EC2 options will trickle up like this to
> become spark-ec2 options.
> While major options are well suited to this type of promotion, we should
> probably allow users to pass through EC2 options they want to use through
> spark-ec2 in some generic way.
> Let's add two options:
> * {{--ec2-instance-option}} ->
> [{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
> * {{--ec2-spot-instance-option}} ->
> [{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]
> Each option can be specified multiple times and is simply passed directly to
> the underlying boto call.
> For example:
> {code}
> spark-ec2 \
> ...
> --ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
> --ec2-instance-option "ebs_optimized=True"
> {code}
> I'm not sure about the exact syntax of the extended options, but something
> like this will do the trick as long as it can be made to pass the options
> correctly to boto in most cases.
> I followed the example of {{ssh}}, which supports multiple extended options
> similarly.
> {code}
> ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
