Also the weird part with this conntrack deletion I perform a conntrack –L to view the table I see no entry for any of the entries its trying to delete. Those entries are all removed anyways when Vms are cleaned up from the look of it. So it looks like all those conntrack deletions were pretty much no-ops Ajay
From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>> Date: Tuesday, September 12, 2017 at 9:30 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>> Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup Hi Kevin Sure will log a bug Also does the config change involve having both these lines in the neutron.conf file? [agent] root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf If I have only the second line I see the exception below on neutron openvswitch agent bring up: 2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', '--ppid', '103', '-o', 'pid='] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89 2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 42, in agent_main_wrapper ovs_agent.main(bridge_classes) File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2184, in main agent.daemon_loop() File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2100, in daemon_loop self.ovsdb_monitor_respawn_interval) as pm: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager pm.start() File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 57, in start while not self.is_active(): File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 100, in is_active self.pid, self.cmd_without_namespace) File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 159, in pid run_as_root=self.run_as_root) File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 297, in get_root_helper_child_pid pid = find_child_pids(pid)[0] File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 179, in find_child_pids log_fail_as_error=False) File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 128, in execute _stdout, _stderr = obj.communicate(_process_input) File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate return self._communicate(input) File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate stdout, stderr = self._communicate_with_select(input) File "/usr/lib64/python2.7/subprocess.py", line 1504, in _communicate_with_select rlist, wlist, xlist = select.select(read_set, write_set, []) File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in select return hub.switch() File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch return self.greenlet.switch() Timeout: 5 seconds 2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=95 Ajay From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Monday, September 11, 2017 at 1:12 PM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>> Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup Can you start a bug on launchpad and upload the conntrack attachment to the bug? Switching to the rootwrap daemon should also help significantly. On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu) <akala...@cisco.com<mailto:akala...@cisco.com>> wrote: Hi Kevin The information you asked for For 1 compute node with 45 Vms here is the number of connection tracking entries getting deleted cat conntrack.file | wc -l 38528 The file with output is 14MB so ill email it to Ian and he can share it if needed Security group rules DirectionEther TypeIP ProtocolPort RangeRemote IP PrefixRemote Security GroupActions EgressIPv4AnyAny0.0.0.0/0<http://0.0.0.0/0> IngressIPv6AnyAny-default EgressIPv6AnyAny::/0- IngressIPv4AnyAny- Please let me know if u need the dump of conntrack entries if so I can email it to email address of your choice Ajay From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Monday, September 11, 2017 at 10:02 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup Hi Kevin Thanks for your response it was about 50 vms Ajay On Sep 11, 2017, at 9:49 AM, Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>> wrote: The biggest improvement will be switching to native netlink calls: https://review.openstack.org/#/c/470912/ How many VMs were on a single compute node? On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu) <akala...@cisco.com<mailto:akala...@cisco.com>> wrote: Hi I am performing a scale test and I see that after creating 500 Vms with ping traffic between them it took almost 1 hr for the connection tracking To clean up and ovs agent was busy doing this and unable to service any new port bind requests on some computes for almost an hr It took that long for conntrack clean up to complete I see the following bug https://bugs.launchpad.net/neutron/+bug/1513765 And I also have the fix below https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e Still see really long times for conntrack cleanup What is the solution to this problem in scale scenarios? Ajay __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
