GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2225
[SPARK-3332] Revert spark-ec2 patch that identifies clusters using tags
This reverts #1899 and #2163, two patches that modified `spark-ec2` so that
clusters are identified using tags instead of security groups. The original
motivation for this patch was to allow multiple clusters to run in the same
security group.
Unfortunately, tagging is not atomic with launching instances on EC2, so
with this approach we have the possibility of `spark-ec2` launching instances
and crashing before they can be tagged, effectively orphaning those instances.
The orphaned instances won't belong to any cluster, so the `spark-ec2` script
will be unable to clean them up.
Since this feature may still be worth supporting, there are several
alternative approaches that we might consider, including detecting orphaned
instances and logging warnings, or maybe using another mechanism to group
instances into clusters. For the 1.1.0 release, though, I propose that we just
revert this patch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark revert-ec2-cluster-naming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2225.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2225
----
commit c2ca2d483f5b051eaf0a801ad50900feadfaa325
Author: Josh Rosen <[email protected]>
Date: 2014-08-31T21:46:03Z
Revert "Spark-3213 Fixes issue with spark-ec2 not detecting slaves created
with "Launch More like this""
This reverts commit 3cb4e1718f40a18e3d19a33fd627960687bbcb6c.
commit 0c18e86ef5d0effc1ca41cfec30ef0f744d81e66
Author: Josh Rosen <[email protected]>
Date: 2014-08-31T21:46:20Z
Revert "SPARK-2333 - spark_ec2 script should allow option for existing
security group"
This reverts commit c3952b092a2f7fea4798f4cb7abac300b9dc9c29.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
