The issues discussed in this email are now resolved.
Instance creation on Horizon and Wikitech has been re-enabled. After
some discussion we've decided to enable security group editing on
Horizon but leave it disabled on Wikitech -- the Horizon interface is
generally nicer, more feature-rich, and more reliable. Please go to
Horizon for any future security group needs.
There were two bugs that triggered this incident. One of them[1]
prevented enforcement of firewall rules in certain cases, and the
other[2] enforced rules but updated them very haphazardly. Both issues
are now well understood, with patches in place and proper long-term
solutions underway.
We have not yet written a full incident report, but when we do it will
most likely be here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20160805-LabsSecurityGroups
Sorry for the inconvenience!
-Andrew
[1] https://phabricator.wikimedia.org/T142388
[2] https://phabricator.wikimedia.org/T142165
On 8/5/16 3:21 PM, Chase Pettet wrote:
Currently running instances within Labs are fine.
This week we upgraded to Openstack Liberty[1][2]. Thursday (8/4) we
had reports of issues involving new instances[3]. We have now
determined there is errant behavior with Liberty managing source
groups. We use this to allow instances within the same project to
communicate with each other. Attempts to resolve this behavior for
the Tool Labs project resulted in a short issue today[4]. Requests
via the web proxy were failing to connect. Tools and bots within Tool
Labs were still running.
Currently:
* Newly created instances are not being integrated into their security
groups appropriately
* We have disabled the self-serve options for instance creation
temporarily
* Modifying security groups can result in existing instances
experiencing issues
* We have disabled the self-serve options for security group
management temporarily as well
We'll update the task[3] as we have more information. An incident
report will be filed as well. As always, we can be found at labs-l
or on IRC in #wikimedia-labs.
Thanks,
Chase Pettet (on behalf of the Labs team)
[1] https://www.openstack.org/software/liberty/
[2] https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html
[3] https://phabricator.wikimedia.org/T142165
[4] https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html
_______________________________________________
Labs-announce mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-announce
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
