[
https://issues.apache.org/jira/browse/SPARK-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Chammas resolved SPARK-5879.
-------------------------------------
Resolution: Duplicate
Resolving as duplicate of the pre-existing SPARK-5629. Feel free to reopen if
there is a difference.
> spary_ec2.py should expose/return master and slave lists (e.g. write to file)
> -----------------------------------------------------------------------------
>
> Key: SPARK-5879
> URL: https://issues.apache.org/jira/browse/SPARK-5879
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Florian Verhein
>
> After running spark_ec2.py, it is often useful/necessary to know the master's
> ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline.
> For example, consider a wrapper that launches a cluster, then waits for
> completion of some application running on it (e.g. polling via ssh), before
> destroying the cluster.
> Some options:
> - write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically
> a subset of the ec2_variables.sh that is temporarily created as part of
> deploy_files variable substitution)
> - launch-variables.json (same info but as json)
> Both would be useful depending on the wrapper language.
> I think we should incorporate the cluster name for the case that multiple
> clusters are launched. E.g. <cluster_name>_variables.sh/.json
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
