Florian Verhein created SPARK-5879:
--------------------------------------
Summary: spary_ec2.py should expose/return master and slave lists
(e.g. write to file)
Key: SPARK-5879
URL: https://issues.apache.org/jira/browse/SPARK-5879
Project: Spark
Issue Type: Improvement
Components: EC2
Reporter: Florian Verhein
After running spark_ec2.py, it is often useful/necessary to know the master's
ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline.
For example, consider a wrapper that launches a cluster, then waits for
completion of some application running on it (e.g. polling via ssh), before
destroying the cluster.
Some options:
- write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically a
subset of the ec2_variables.sh that is temporarily created as part of
deploy_files variable substitution)
- launch-variables.json (same info but as json)
Both would be useful depending on the wrapper language.
I think we should incorporate the cluster name for the case that multiple
clusters are launched. E.g. <cluster_name>_variables.sh/.json
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
