Github user acvogel commented on a diff in the pull request:
https://github.com/apache/spark/pull/5109#discussion_r27149279
--- Diff: ec2/spark_ec2.py ---
@@ -567,16 +592,28 @@ def launch_cluster(conn, opts, cluster_name):
for i in my_req_ids:
if i in id_to_req and id_to_req[i].state == "active":
active_instance_ids.append(id_to_req[i].instance_id)
- if len(active_instance_ids) == opts.slaves:
- print "All %d slaves granted" % opts.slaves
+ if len(active_instance_ids) == len(my_req_ids):
+ if opts.master_spot_price is not None:
+ if opts.spot_price is not None:
+ print "Master and all %d slaves granted" %
opts.slaves
+ else:
+ print "Master granted"
+ else:
+ print "All %d slaves granted" % opts.slaves
reservations =
conn.get_all_reservations(active_instance_ids)
+ master_nodes = []
slave_nodes = []
for r in reservations:
- slave_nodes += r.instances
+ if any([group.name == cluster_name + "-master" for
group in r.groups]):
+ master_nodes += r.instances
+ elif any([group.name == cluster_name + "-slaves"
for group in r.groups]):
+ slave_nodes += r.instances
+ else:
+ print >> stderr, ("WARNING: Unknown security
group " % group.name)
--- End diff --
`reservations` is populated by `reservations =
conn.get_all_reservations(active_instance_ids)`, where `active_instance_ids` is
restricted to those instances which correspond to a request in `my_req_ids`
(the union of `master_reqs` and `slave_reqs`). Thus, `reservations` should
never contain anything besides the master and slave reservations. I added the
`else` statement in the case that future modifications change the security
group names. This method of getting the relevant reservations mirrors that used
in the current master version.
An alternative approach along the lines of your suggestion is to split
`active_instance_ids` into `active_master_instance_ids` and
`active_slave_instance_ids` arrays. I'd be happy to submit a patch for this
approach if you think it would be cleaner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
