[
https://issues.apache.org/jira/browse/SPARK-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326699#comment-14326699
]
Apache Spark commented on SPARK-5879:
-------------------------------------
User 'mengxr' has created a pull request for this issue:
https://github.com/apache/spark/pull/4680
> spary_ec2.py should expose/return master and slave lists (e.g. write to file)
> -----------------------------------------------------------------------------
>
> Key: SPARK-5879
> URL: https://issues.apache.org/jira/browse/SPARK-5879
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Florian Verhein
>
> After running spark_ec2.py, it is often useful/necessary to know the master's
> ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline.
> For example, consider a wrapper that launches a cluster, then waits for
> completion of some application running on it (e.g. polling via ssh), before
> destroying the cluster.
> Some options:
> - write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically
> a subset of the ec2_variables.sh that is temporarily created as part of
> deploy_files variable substitution)
> - launch-variables.json (same info but as json)
> Both would be useful depending on the wrapper language.
> I think we should incorporate the cluster name for the case that multiple
> clusters are launched. E.g. <cluster_name>_variables.sh/.json
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
