Hi Kris, We are moving to neutron in a month and are hitting this in our pre prod environment too.
Sam > On 2 Sep 2015, at 6:32 am, Kris G. Lindgren <[email protected]> wrote: > > Hello, > > We ran into this again today. > > I created bug: https://bugs.launchpad.net/neutron/+bug/1491131 for this. > With the log files for ~10 seconds before the issue happened to the first > couple ipset delete failures. > > > > > On 8/20/15, 6:37 AM, "Miguel Angel Ajo" <[email protected]> wrote: > >> Hi Kris, >> >> I'm adding Shi Han Zhang to the thread, >> >> I'm was involved in some refactors during kilo and Han Zhang in some >> extra fixes during Liberty [1] [2] [3], >> >> Could you get us some logs of such failures to see what was >> happening around the failure time?, as a minimum we should >> post the log error traces to a bug in https://bugs.launchpad.net/neutron >> >> We will be glad to use such information to make the ipset more >> fault tolerant, and try to identify the cause of the >> possible race conditions. >> >> >> [1] https://review.openstack.org/#/c/187483/ >> [2] https://review.openstack.org/190991 >> [3] https://review.openstack.org/#/c/187433/ >> >> >> >> Kris G. Lindgren wrote: >>> >>> We have been using ipsets since juno. Twice now since our kilo >>> upgrade we have had issues with ipsets blowing up on a compute node. >>> >>> The first time, was iptables was referencing an ipset that was either >>> no longer there or was not added, and was trying to apply the iptables >>> config every second and dumping the full iptables-resotore output into >>> the log when it failed at TRACE level. >>> Second time, was that ipsets was failing to remove an element that was >>> no longer there. >>> >>> For #1 I solved by restarting the neutron-openvswitch-agent. For #2 >>> we just added the entry that ipsets was trying to remove. It seems >>> like we are having some race conditions under kilo that were not >>> present under juno (or we managed to run it for 6+ months without it >>> biting us). >>> >>> Is anyone else seeing the same problems? I am noticing some commits >>> reverting/re-adding around ipsets in kilo and liberty so trying to >>> confirm if I need to open a new bug on this. >>> ____________________________________________ >>> >>> Kris Lindgren >>> Senior Linux Systems Engineer >>> GoDaddy, LLC. >>> >>> _______________________________________________ >>> OpenStack-operators mailing list >>> [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> >> Kris G. Lindgren wrote: >>> We have been using ipsets since juno. Twice now since our kilo upgrade we >>> have had issues with ipsets blowing up on a compute node. >>> >>> The first time, was iptables was referencing an ipset that was either no >>> longer there or was not added, and was trying to apply the iptables config >>> every second and dumping the full iptables-resotore output into the log >>> when it failed at TRACE level. >>> Second time, was that ipsets was failing to remove an element that was no >>> longer there. >>> >>> For #1 I solved by restarting the neutron-openvswitch-agent. For #2 we >>> just added the entry that ipsets was trying to remove. It seems like we >>> are having some race conditions under kilo that were not present under juno >>> (or we managed to run it for 6+ months without it biting us). >>> >>> Is anyone else seeing the same problems? I am noticing some commits >>> reverting/re-adding around ipsets in kilo and liberty so trying to confirm >>> if I need to open a new bug on this. >>> ____________________________________________ >>> >>> Kris Lindgren >>> Senior Linux Systems Engineer >>> GoDaddy, LLC. >>> >>> _______________________________________________ >>> OpenStack-operators mailing list >>> [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
