Thanks for your feedback too Ihar, comments inline. ----- Original Message ----- > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 24/10/14 11:56, Miguel Angel Ajo Pelayo wrote: > > > > > > ----- Original Message ----- > >> Hi Miguel, > >> > >> while we'd need to hear from the stable team, I think it's not > >> such a bad idea to make this tool available to users of pre-juno > >> openstack releases. > > It's a great idea actually. It's great when code emerged from real > life downstream support cases eventually flow up to upstream for all > operator's benefit (and not just those who pay huge money for > commercial service). > > >> As far as upstream repos are concerned, I don't know if this tool > >> violates the criteria for stable branches. Even if it would be a > >> rather large change for stable/icehouse, it is pretty much > >> orthogonal to the existing code, so it could be ok. However, > >> please note that stable/havana has now reached its EOL, so there > >> will be no more stable release for it. > > > > Sure, I was mentioning havana as affected, but I understand it's > > already under U/S EOL, D/S distributions would always be free to > > backport, specially on an orthogonal change like this. > > > > About stable/icehouse, I'd like to hear from the stable > > maintainers. > > I'm for inclusion of the tool in the main neutron package. Though it's > possible to publish it on pypi as a separate package, I would better > apply formal review process to it, plus reduce packaging efforts for > distributions (and myself). The tool may be later expanded for other > useful operator hooks, so I'm for inclusion of the tool in master and > backporting it back to all supported branches. > > Though official stable maintainership rules state that 'New features' > are no-go for stable branch [1], I think they should not apply in this > case since the tool does not touch production code in any way and just > provides a way to heal security groups on operator demand. Also, rules > are to break them. ;) Quoting the same document, "Proposed backports > breaking any of above guidelines can be discussed as exception > requests on openstack-stable-maint list where stable-maint team will > try to reach consensus." > > Operators should be more happy if we ship such a tool as part of > neutron release and not as another third-party tool from pypi of > potentially unsafe origin. > > BTW I wonder whether the tool can be useful for Juno+ setups too. > Though we mostly mitigated the problem by RPC interface rework and > ipset, some operators may still hit some limitation that could be > workarounded by optimizing their rules. Also, I think the idea of > having a tool with miscellaneous operator hooks in the master tree is > quite interesting. I would recommend to still go with pushing it to > master and then backporting to stable branches. That would also help > to get more review attention from cores than stable branch requests > usually receive. ;)
I believe the tool could also be expanded to report and , equally generate scripts to cleanup orphaned resources, those happen to be when you remove an instance and the port is not deleted, or you delete a tenant, but the resources are kept, etc. I know there are efforts to do proper cleanup when tenants are deleted, but still, I see production databases plagued of orphaned resources. > > [1]: https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes > > > > >> > >> The orthogonal nature of this tool however also make the case for > >> making it widely available on pypi. I think it should be ok to > >> describe the scalability issue in the official OpenStack Icehouse > >> docs and point out to this tool for mitigation. > > > > Yes, of course, I consider that as a second option, my point here > > is that direct upstream review time would result in better quality > > code here, and could certainly spot any hidden bugs, and increase > > testing quality. > > > > It also reduces packaging time all across distributions making it > > available via the standard neutron repository. > > > > > > Thanks for the feedback!, > > > >> > >> Salvatore > >> > >> On 23 October 2014 14:03, Miguel Angel Ajo Pelayo < > >> [email protected] > wrote: > >> > >> > >> > >> > >> Recently, we have identified clients with problems due to the bad > >> scalability of security groups in Havana and Icehouse, that was > >> addressed during juno here [1] [2] > >> > >> This situation is identified by blinking agents (going UP/DOWN), > >> high AMQP load, nigh neutron-server load, and timeout from > >> openvswitch agents when trying to contact neutron-server > >> "security_group_rules_for_devices". > >> > >> Doing a [1] backport involves many dependent patches related to > >> the general RPC refactor in neutron (which modifies all > >> plugins), and subsequent ones fixing a few bugs. Sounds risky to > >> me. [2] Introduces new features and it's dependent on features > >> which aren't available on all systems. > >> > >> To remediate this on production systems, I wrote a quick tool to > >> help on reporting security groups and mitigating the problem by > >> writing almost-equivalent rules [3]. > >> > >> We believe this tool would be better available to the wider > >> community, and under better review and testing, and, since it > >> doesn't modify any behavior or actual code in neutron, I'd like > >> to propose it for inclusion into, at least, Icehouse stable > >> branch where it's more relevant. > >> > >> I know the usual way is to go master->Juno->Icehouse, but at this > >> moment the tool is only interesting for Icehouse (and Havana), > >> although I believe it could be extended to cleanup orphaned > >> resources, or any other cleanup tasks, in that case it could make > >> sense to be available for K->J->I. > >> > >> As a reference, I'm leaving links to outputs from the tool > >> [4][5] > >> > >> Looking forward to get some feedback, Miguel Ángel. > >> > >> > >> [1] https://review.openstack.org/#/c/111876/ security group rpc > >> refactor [2] https://review.openstack.org/#/c/111877/ ipset > >> support [3] https://github.com/mangelajo/neutrontool [4] > >> http://paste.openstack.org/show/123519/ [5] > >> http://paste.openstack.org/show/123525/ > >> > >> _______________________________________________ OpenStack-dev > >> mailing list [email protected] > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> > >> > >> > _______________________________________________ > >> OpenStack-dev mailing list [email protected] > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > >> > > _______________________________________________ OpenStack-dev > > mailing list [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > > iQEcBAEBCgAGBQJUSjIHAAoJEC5aWaUY1u57dUsH/0hs0OUJQYJJ5Ll/DbT8BgnD > sQvYMVVzVQDKU58qLYT0j03+TfdPOHcPrQ9q0/Oi2kzzpAsFS9/ZY/apIX9ztvbQ > BkCQ8tubQEy8v7gpaUgxP6tREY36g0lLWpO9D2YhNXMhzlugzbgME5s6NEa3yrNM > sXQJH9F3KpsppNtBdlUNImvcNB4xOD2IY6nQMbPTBX9CXY2y7pLDLYZchVXlexuB > S5lRKuKN3E6N9uGOUGeAqfGT0ghJZyUFNNsSuNG9H4338jXQFayGHjKiJsde8tK1 > IyOYMNT9sULMpDlrVjh0+u9YZyV5bKQTjM/ldbjoRM/he1scTxe2XWe1xO4lmfY= > =0r7e > -----END PGP SIGNATURE----- > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
