[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138590#comment-15138590
]
Sean Owen commented on SPARK-5629:
--
I'm bulk closing some EC2 issues as the ones that are still live can be
'ported' to the new amplab repo
> Add spark-ec2 action to return info about an existing cluster
> -
>
> Key: SPARK-5629
> URL: https://issues.apache.org/jira/browse/SPARK-5629
> Project: Spark
> Issue Type: Improvement
> Components: EC2
>Reporter: Nicholas Chammas
>Priority: Minor
>
> You can launch multiple clusters using spark-ec2. At some point, you might
> just want to get some information about an existing cluster.
> Use cases include:
> * Wanting to check something about your cluster in the EC2 web console.
> * Wanting to feed information about your cluster to another tool (e.g. as
> described in [SPARK-5627]).
> So, in addition to the [existing
> actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
> * {{launch}}
> * {{destroy}}
> * {{login}}
> * {{stop}}
> * {{start}}
> * {{get-master}}
> * {{reboot-slaves}}
> We add a new action, {{describe}}, which describes an existing cluster if
> given a cluster name, and all clusters if not.
> Some examples:
> {code}
> # describes all clusters launched by spark-ec2
> spark-ec2 describe
> {code}
> {code}
> # describes cluster-1
> spark-ec2 describe cluster-1
> {code}
> In combination with the proposal in [SPARK-5627]:
> {code}
> # describes cluster-3 in a machine-readable way (e.g. JSON)
> spark-ec2 describe cluster-3 --machine-readable
> {code}
> Parallels in similar tools include:
> * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
> * [{{starcluster
> listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
> from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124613#comment-15124613
]
Shivaram Venkataraman commented on SPARK-5629:
--
I manually cleaned up some of the issues and pointed them to open issues on
spark-ec2. I think for some of the issues we should just ping the issue and see
if its still a relevant issue. Finally I think some of the S3 reading issues
aren't spark-ec2 issues but more an issue with jets3t etc.
> Add spark-ec2 action to return info about an existing cluster
> -
>
> Key: SPARK-5629
> URL: https://issues.apache.org/jira/browse/SPARK-5629
> Project: Spark
> Issue Type: Improvement
> Components: EC2
>Reporter: Nicholas Chammas
>Priority: Minor
>
> You can launch multiple clusters using spark-ec2. At some point, you might
> just want to get some information about an existing cluster.
> Use cases include:
> * Wanting to check something about your cluster in the EC2 web console.
> * Wanting to feed information about your cluster to another tool (e.g. as
> described in [SPARK-5627]).
> So, in addition to the [existing
> actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
> * {{launch}}
> * {{destroy}}
> * {{login}}
> * {{stop}}
> * {{start}}
> * {{get-master}}
> * {{reboot-slaves}}
> We add a new action, {{describe}}, which describes an existing cluster if
> given a cluster name, and all clusters if not.
> Some examples:
> {code}
> # describes all clusters launched by spark-ec2
> spark-ec2 describe
> {code}
> {code}
> # describes cluster-1
> spark-ec2 describe cluster-1
> {code}
> In combination with the proposal in [SPARK-5627]:
> {code}
> # describes cluster-3 in a machine-readable way (e.g. JSON)
> spark-ec2 describe cluster-3 --machine-readable
> {code}
> Parallels in similar tools include:
> * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
> * [{{starcluster
> listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
> from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118972#comment-15118972
]
Sean Owen commented on SPARK-5629:
--
OK, I'm not sure how to implement that. There are lots of EC2 issues that
haven't been touched in a year. I tend to think those are ones nobody is
concerned with any longer. Maybe the reverse -- when someone cares enough to
open the spark-ec2 issues for business, go back and search for "EC2" issues and
port those that matter?
> Add spark-ec2 action to return info about an existing cluster
> -
>
> Key: SPARK-5629
> URL: https://issues.apache.org/jira/browse/SPARK-5629
> Project: Spark
> Issue Type: Improvement
> Components: EC2
>Reporter: Nicholas Chammas
>Priority: Minor
>
> You can launch multiple clusters using spark-ec2. At some point, you might
> just want to get some information about an existing cluster.
> Use cases include:
> * Wanting to check something about your cluster in the EC2 web console.
> * Wanting to feed information about your cluster to another tool (e.g. as
> described in [SPARK-5627]).
> So, in addition to the [existing
> actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
> * {{launch}}
> * {{destroy}}
> * {{login}}
> * {{stop}}
> * {{start}}
> * {{get-master}}
> * {{reboot-slaves}}
> We add a new action, {{describe}}, which describes an existing cluster if
> given a cluster name, and all clusters if not.
> Some examples:
> {code}
> # describes all clusters launched by spark-ec2
> spark-ec2 describe
> {code}
> {code}
> # describes cluster-1
> spark-ec2 describe cluster-1
> {code}
> In combination with the proposal in [SPARK-5627]:
> {code}
> # describes cluster-3 in a machine-readable way (e.g. JSON)
> spark-ec2 describe cluster-3 --machine-readable
> {code}
> Parallels in similar tools include:
> * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
> * [{{starcluster
> listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
> from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110853#comment-15110853
]
Sean Owen commented on SPARK-5629:
--
Are all of the EC2 tickets becoming essentially "wont fix" as the support moves
out of Spark itself?
> Add spark-ec2 action to return info about an existing cluster
> -
>
> Key: SPARK-5629
> URL: https://issues.apache.org/jira/browse/SPARK-5629
> Project: Spark
> Issue Type: Improvement
> Components: EC2
>Reporter: Nicholas Chammas
>Priority: Minor
>
> You can launch multiple clusters using spark-ec2. At some point, you might
> just want to get some information about an existing cluster.
> Use cases include:
> * Wanting to check something about your cluster in the EC2 web console.
> * Wanting to feed information about your cluster to another tool (e.g. as
> described in [SPARK-5627]).
> So, in addition to the [existing
> actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
> * {{launch}}
> * {{destroy}}
> * {{login}}
> * {{stop}}
> * {{start}}
> * {{get-master}}
> * {{reboot-slaves}}
> We add a new action, {{describe}}, which describes an existing cluster if
> given a cluster name, and all clusters if not.
> Some examples:
> {code}
> # describes all clusters launched by spark-ec2
> spark-ec2 describe
> {code}
> {code}
> # describes cluster-1
> spark-ec2 describe cluster-1
> {code}
> In combination with the proposal in [SPARK-5627]:
> {code}
> # describes cluster-3 in a machine-readable way (e.g. JSON)
> spark-ec2 describe cluster-3 --machine-readable
> {code}
> Parallels in similar tools include:
> * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
> * [{{starcluster
> listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
> from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111056#comment-15111056
]
Shivaram Venkataraman commented on SPARK-5629:
--
Yes - though I think its beneficial to see if the ticket is still valid and if
it is, we can open a corresponding issue at github.com/amplab/spark-ec2. Then
we can leave a marker here saying where this issue is being followed up at.
> Add spark-ec2 action to return info about an existing cluster
> -
>
> Key: SPARK-5629
> URL: https://issues.apache.org/jira/browse/SPARK-5629
> Project: Spark
> Issue Type: Improvement
> Components: EC2
>Reporter: Nicholas Chammas
>Priority: Minor
>
> You can launch multiple clusters using spark-ec2. At some point, you might
> just want to get some information about an existing cluster.
> Use cases include:
> * Wanting to check something about your cluster in the EC2 web console.
> * Wanting to feed information about your cluster to another tool (e.g. as
> described in [SPARK-5627]).
> So, in addition to the [existing
> actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
> * {{launch}}
> * {{destroy}}
> * {{login}}
> * {{stop}}
> * {{start}}
> * {{get-master}}
> * {{reboot-slaves}}
> We add a new action, {{describe}}, which describes an existing cluster if
> given a cluster name, and all clusters if not.
> Some examples:
> {code}
> # describes all clusters launched by spark-ec2
> spark-ec2 describe
> {code}
> {code}
> # describes cluster-1
> spark-ec2 describe cluster-1
> {code}
> In combination with the proposal in [SPARK-5627]:
> {code}
> # describes cluster-3 in a machine-readable way (e.g. JSON)
> spark-ec2 describe cluster-3 --machine-readable
> {code}
> Parallels in similar tools include:
> * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
> * [{{starcluster
> listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
> from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329771#comment-14329771
]
Nicholas Chammas commented on SPARK-5629:
-
[~florianverhein] - Hmm... Thinking about this for a bit, I'd be against
introducing such flexibility right off the bat. Every additional option or
non-standard flow will impose a maintenance burden.
By that argument, actually, we should also just stick to JSON even though its
less readable. As in, {{describe}} outputs in pretty-printed JSON and that's it.
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325372#comment-14325372
]
Nicholas Chammas commented on SPARK-5629:
-
YAML is not part of the Python standard library, unfortunately.
Agree on somehow marking this as experimental.
I think YAML is like JSON in that adding or removing fields shouldn't break any
parsers. There is no pre-defined schema. It should just affect you if you are
trying to access a field that was removed, for example, just like it would with
JSON.
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325354#comment-14325354
]
Shivaram Venkataraman commented on SPARK-5629:
--
This sounds fine to me and I really like YAML -- Does Python have native
support for printing out YAML ?
One thing we should do is probably marking this as experimental as we might not
be able maintain backwards compatibility etc. (On that note are YAML parsers
backwards compatible ? i.e. if we add a new field in the next release will it
break things ?)
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325346#comment-14325346
]
Nicholas Chammas commented on SPARK-5629:
-
For example, you run:
{code}
$ spark-ec2 describe my-spark-cluster
{code}
And you get back something like this:
{code}
my-spark-cluster:
launched: 2015-02-18 14:03:22 UTC
status: running
nodes:
master: ec2-54-69-105-224.us-west-2.compute.amazonaws.com
slaves:
- ec2-54-69-1215-97.us-west-2.compute.amazonaws.com
- ec2-54-69-186-101.us-west-2.compute.amazonaws.com
- ec2-54-69-186-109.us-west-2.compute.amazonaws.com
{code}
Actually, since this is both valid YAML and very human-readable, we probably
don't need the {{--machine-readable}} option mentioned in the ticket body.
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324888#comment-14324888
]
Nicholas Chammas commented on SPARK-5629:
-
cc [~joshrosen] / [~shivaram]
I see that we already have a {{get-master}} action which will probably serve
most use cases where spark-ec2 is being used as part of some automated pipeline
(e.g. spark-perf testing). Typically, you just want the master address so you
can ssh in and do stuff.
Still, I'm looking for your initial reaction to this proposal.
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[
https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324907#comment-14324907
]
Shivaram Venkataraman commented on SPARK-5629:
--
Is there an example output for `describe` you have in mind ? And I am not sure
it'll be easy to list all the clusters as spark-ec2 looks up clusters by the
security group / cluster-id ?
Add spark-ec2 action to return info about an existing cluster
-
Key: SPARK-5629
URL: https://issues.apache.org/jira/browse/SPARK-5629
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Nicholas Chammas
Priority: Minor
You can launch multiple clusters using spark-ec2. At some point, you might
just want to get some information about an existing cluster.
Use cases include:
* Wanting to check something about your cluster in the EC2 web console.
* Wanting to feed information about your cluster to another tool (e.g. as
described in [SPARK-5627]).
So, in addition to the [existing
actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]:
* {{launch}}
* {{destroy}}
* {{login}}
* {{stop}}
* {{start}}
* {{get-master}}
* {{reboot-slaves}}
We add a new action, {{describe}}, which describes an existing cluster if
given a cluster name, and all clusters if not.
Some examples:
{code}
# describes all clusters launched by spark-ec2
spark-ec2 describe
{code}
{code}
# describes cluster-1
spark-ec2 describe cluster-1
{code}
In combination with the proposal in [SPARK-5627]:
{code}
# describes cluster-3 in a machine-readable way (e.g. JSON)
spark-ec2 describe cluster-3 --machine-readable
{code}
Parallels in similar tools include:
* [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju
* [{{starcluster
listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node]
from MIT StarCluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
