[
https://issues.apache.org/jira/browse/SPARK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116950#comment-14116950
]
Allan Douglas R. de Oliveira commented on SPARK-3332:
-----------------------------------------------------
[~pwendell], yes this is a good reword and I agree to revert it for now. You
mentioned the two potential solutions but I think a good compromise is the one
implemented by the PR which keeps the flag allowing reuse of the same security
group but also allowing to match the machines by the security group in the
other cases.
> Tagging is not atomic with launching instances on EC2
> -----------------------------------------------------
>
> Key: SPARK-3332
> URL: https://issues.apache.org/jira/browse/SPARK-3332
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Reporter: Allan Douglas R. de Oliveira
>
> The implementation for SPARK-2333 changed the machine membership mechanism
> from security groups to tags.
> This is a fundamentally flawed strategy as there aren't guarantees at all the
> machines will have a tag (even with a retry mechanism).
> For instance, if the script is killed after launching the instances but
> before setting the tags the machines will be "invisible" to a destroy
> command, leaving a unmanageable cluster behind.
> The initial proposal is to go back to the previous behavior for all cases but
> when the new flag (--security-group-prefix) is used.
> Also it's worthwhile to mention that SPARK-3180 introduced the
> --additional-security-group flag which is a reasonable solution to SPARK-2333
> (but isn't a full replacement to all use cases of --security-group-prefix).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
