Allan Douglas R. de Oliveira created SPARK-3332:
---------------------------------------------------
Summary: Tags shouldn't be the main strategy for machine
membership on clusters
Key: SPARK-3332
URL: https://issues.apache.org/jira/browse/SPARK-3332
Project: Spark
Issue Type: Bug
Components: EC2
Reporter: Allan Douglas R. de Oliveira
The implementation for SPARK-2333 changed the machine membership mechanism from
security groups to tags.
This is a fundamentally flawed strategy as there aren't guarantees at all the
machines will have a tag (even with a retry mechanism).
For instance, if the script is killed after launching the instances but before
setting the tags the machines will be "invisible" to a destroy command, leaving
a unmanageable cluster behind.
The initial proposal is to go back to the previous behavior for all cases but
when the new flag (--security-group-prefix) is used.
Also it's worthwhile to mention that SPARK-3180 introduced the
--additional-security-group flag which is a reasonable solution to SPARK-2333
(but isn't a full replacement to all use cases of --security-group-prefix).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
