Hello, We ran into this again today.
I created bug: https://bugs.launchpad.net/neutron/+bug/1491131 for this. With the log files for ~10 seconds before the issue happened to the first couple ipset delete failures. On 8/20/15, 6:37 AM, "Miguel Angel Ajo" <[email protected]> wrote: >Hi Kris, > > I'm adding Shi Han Zhang to the thread, > > I'm was involved in some refactors during kilo and Han Zhang in some >extra fixes during Liberty [1] [2] [3], > > Could you get us some logs of such failures to see what was >happening around the failure time?, as a minimum we should >post the log error traces to a bug in https://bugs.launchpad.net/neutron > > We will be glad to use such information to make the ipset more >fault tolerant, and try to identify the cause of the >possible race conditions. > > >[1] https://review.openstack.org/#/c/187483/ >[2] https://review.openstack.org/190991 >[3] https://review.openstack.org/#/c/187433/ > > > >Kris G. Lindgren wrote: >> >> We have been using ipsets since juno. Twice now since our kilo >> upgrade we have had issues with ipsets blowing up on a compute node. >> >> The first time, was iptables was referencing an ipset that was either >> no longer there or was not added, and was trying to apply the iptables >> config every second and dumping the full iptables-resotore output into >> the log when it failed at TRACE level. >> Second time, was that ipsets was failing to remove an element that was >> no longer there. >> >> For #1 I solved by restarting the neutron-openvswitch-agent. For #2 >> we just added the entry that ipsets was trying to remove. It seems >> like we are having some race conditions under kilo that were not >> present under juno (or we managed to run it for 6+ months without it >> biting us). >> >> Is anyone else seeing the same problems? I am noticing some commits >> reverting/re-adding around ipsets in kilo and liberty so trying to >> confirm if I need to open a new bug on this. >> ____________________________________________ >> >> Kris Lindgren >> Senior Linux Systems Engineer >> GoDaddy, LLC. >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > >Kris G. Lindgren wrote: >> We have been using ipsets since juno. Twice now since our kilo upgrade we >> have had issues with ipsets blowing up on a compute node. >> >> The first time, was iptables was referencing an ipset that was either no >> longer there or was not added, and was trying to apply the iptables config >> every second and dumping the full iptables-resotore output into the log when >> it failed at TRACE level. >> Second time, was that ipsets was failing to remove an element that was no >> longer there. >> >> For #1 I solved by restarting the neutron-openvswitch-agent. For #2 we just >> added the entry that ipsets was trying to remove. It seems like we are >> having some race conditions under kilo that were not present under juno (or >> we managed to run it for 6+ months without it biting us). >> >> Is anyone else seeing the same problems? I am noticing some commits >> reverting/re-adding around ipsets in kilo and liberty so trying to confirm >> if I need to open a new bug on this. >> ____________________________________________ >> >> Kris Lindgren >> Senior Linux Systems Engineer >> GoDaddy, LLC. >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
