[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Reviewed: https://review.opendev.org/c/openstack/neutron/+/871659 Committed: https://opendev.org/openstack/neutron/commit/c609084b59c68f003153a58c2063f99b52f169e0 Submitter: "Zuul (22348)" Branch:master commit c609084b59c68f003153a58c2063f99b52f169e0 Author: Brian Haley Date: Tue Jan 24 15:16:47 2023 -0500 Add doc note on nf_conntrack module requirement The OVS firewall driver requires nf_conntrack module(s) to be loaded to function properly. While they are typically loaded automatically, add a note to the admin guide about the requirement to make it explicit. Closes-bug: #1834213 Change-Id: I55871eff1e37d4155b8d2b5ae8c182d160c4af9f ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack Neutron Open vSwitch Charm: Fix Released Status in neutron: Fix Released Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
** Changed in: neutron Assignee: (unassigned) => Brian Haley (brian-haley) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack Neutron Open vSwitch Charm: Fix Released Status in neutron: In Progress Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Fix proposed to branch: master Review: https://review.opendev.org/c/openstack/neutron/+/871659 ** Changed in: neutron Status: New => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack Neutron Open vSwitch Charm: Fix Released Status in neutron: In Progress Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
I've filed a follow-up bug of neutron-openvswitch on kernel upgrade: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1851764 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Released Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
** Changed in: charm-neutron-openvswitch Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Released Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
** Changed in: charm-neutron-openvswitch Milestone: None => 19.10 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Work around Load: sudo modprobe nf_conntrack_ipv4 Confirm: lsmod | grep nf_conntrack_ipv4 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
on the new HWE kernel from ubuntu 18.04, which is linux 5.0, the modules are not present anymore nf_conntrack_ipv4 and nf_conntrack_ipv6 i think it was merged into nf_conntrack but i'm not sure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Hi James. I don't think that Neutron should load this module. We are not managing any modules in Neutron AFAICT. It is on deployment tools/distro to ensure that proper modules are loaded. Maybe we should add note about this module in https://github.com/openstack/neutron/blob/688bbdd5cd10a13b010902525617fd43d8a415b6/doc/source/admin/config-ovsfwdriver.rst - What do You think about it? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Adding a neutron bug-task to get an upstream opinion on whether neutron should be loading these modules as the n-ovs-agent starts up. ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in neutron: New Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Reviewed: https://review.opendev.org/678956 Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=b76a59299794700fae1878af513c90ca5182a9f6 Submitter: Zuul Branch:master commit b76a59299794700fae1878af513c90ca5182a9f6 Author: tpsilva Date: Tue Aug 27 17:41:24 2019 -0300 Explicitly load nf_conntrack_ipv4 module When neutron-openvswitch-agent is using the openvswitch firewall, it needs the nf_conntrack_ipv4 module to be loaded. Usually, this module gets loaded by some other external tool, but in case this does not happen, neither the charm nor neutron will load it, so all traffic to the instances in this host will fail. This patch fixes that by explicitly loading the module. Change-Id: Ia788e870c124de7da17961c02259cfe80938e5d2 Closes-bug: #1834213 ** Changed in: charm-neutron-openvswitch Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Fix Committed Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Fix up for review: https://review.opendev.org/#/c/678956/ ** Changed in: charm-neutron-openvswitch Assignee: (unassigned) => Tiago Pasqualini da Silva (tiago.pasqualini) ** Changed in: charm-neutron-openvswitch Status: Confirmed => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: In Progress Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Just tested on devstack deployed on Xenial. The module gets loaded at some point during neutron configuration on the deployment script. It seems like a neutron bug to me. It relies on conntrack for the firewall to work, but never actually loads the module. In most cases something else will end up loading it, but in the event that no one else loads it, it will fail. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Spent some time debugging this and I found some interesting bits. I was able to reproduce this by deploying a xenial-queens environment with VXLAN and the OVS firewall. Investigating this, here is what I found: 1) This module is first loaded on the compute nodes when libvirt-bin is installed. This package's postinst script creates the default libvirt network, so when libvirt service is enabled it will create some rules on iptables for this network, which will load the nf_conntrack_ipv4 module. 2) At some point during the configuration of the environment, this default network is destroyed (I'm still investigating who is doing this, but makes sense since nova/neutron won't use the default libvirt network), so those iptables rules won't be added anymore on libvirt service startup, so what was previously loading the module, won't do it. 3) Neutron relies on conntrack for the OVS firewall to work. It's on the documentation: https://docs.openstack.org/newton/networking-guide /config-ovsfwdriver.html 4) As pointed on the bug description, OVS complains whenever the module is not loaded, so we can assume that it's not its responsibility to load it. In my opinion this is something that neutron-ovs-agent should be loading, since the OVS firewall requires conntrack to work and OVS complains that it is not loaded. It would be interesting to see how (if) this works on different openstack deployments. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
** Tags added: sts -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
** Changed in: charm-neutron-openvswitch Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
oddly, this did not happen on all hosts with this version kernel, it was pseudo random and about ~30-40%. There must be another variable at play. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Incomplete Status in linux package in Ubuntu: Incomplete Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Kernel version uname -r 4.4.0-150-generic apt list --installed | fgrep image cloud-image-utils/xenial-updates,now 0.27-0ubuntu25.1 all [installed,automatic] genisoimage/xenial,now 9:1.1.11-3ubuntu1 amd64 [installed] linux-image-4.4.0-137-generic/xenial-updates,xenial-security,now 4.4.0-137.163 amd64 [installed,automatic] linux-image-4.4.0-148-generic/xenial-updates,xenial-security,now 4.4.0-148.174 amd64 [installed,automatic] linux-image-4.4.0-150-generic/xenial-updates,xenial-security,now 4.4.0-150.176 amd64 [installed,automatic] linux-image-extra-4.4.0-137-generic/xenial-updates,xenial-security,now 4.4.0-137.163 amd64 [installed,automatic] linux-image-generic/now 4.4.0.150.158 amd64 [installed,upgradable to: 4.4.0.154.162] linux-signed-image-4.4.0-137-generic/xenial-updates,xenial-security,now 4.4.0-137.163 amd64 [installed,automatic] ubuntu-cloudimage-keyring/xenial,now 2013.11.11 all [installed] openvswitch version apt list --installed | fgrep vswitch neutron-openvswitch-agent/now 2:12.0.5-0ubuntu1~cloud0 all [installed,upgradable to: 2:12.0.6-0ubuntu2~cloud0] openvswitch-common/xenial-updates,now 2.9.2-0ubuntu0.18.04.3~cloud0 amd64 [installed] openvswitch-switch/xenial-updates,now 2.9.2-0ubuntu0.18.04.3~cloud0 amd64 [installed] python-openvswitch/xenial-updates,now 2.9.2-0ubuntu0.18.04.3~cloud0 all [installed] let me know if you need anything else. Thanks, Steven -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Incomplete Status in linux package in Ubuntu: Incomplete Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Ignore prior comment: $ lsmod | grep conntrack nf_conntrack_ipv6 20480 1 nf_conntrack_ipv4 16384 1 nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 nf_defrag_ipv6 36864 2 nf_conntrack_ipv6,openvswitch nf_conntrack 131072 6 nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_nat_ipv6,nf_nat_ipv4,openvswitch libcrc32c 16384 5 nf_conntrack,nf_nat,openvswitch,xfs,raid456 as soon as a loaded the openvswitch kernel module the nf_conntrack_* modules where loaded as well. ** Changed in: charm-neutron-openvswitch Status: New => Incomplete ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Incomplete Status in linux package in Ubuntu: New Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1834213] Re: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances
Raising a kernel bug task. Note my testing was on Bionic not Xenial. Drew - can you confirm which kernel version and packages you are using. ** Changed in: charm-neutron-openvswitch Importance: Undecided => Low -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1834213 Title: After kernel upgrade, nf_conntrack_ipv4 module unloaded, no IP traffic to instances Status in OpenStack neutron-openvswitch charm: Incomplete Status in linux package in Ubuntu: New Bug description: With an environment running Xenial-Queens, and having just upgraded the linux-image-generic kernel for MDS patching, a few of our hypervisor hosts that were rebooted (3 out of 100) ended up dropping IP (tcp/udp) ingress traffic. It turns out that nf_conntrack module was loaded, but nf_conntrack_ipv4 was not loading, and the traffic was being dropped by this rule: table=72, n_packets=214989, priority=50,ct_state=+inv+trk actions=resubmit(,93) The ct_state "inv" means invalid conntrack state, which the manpage describes as: The state is invalid, meaning that the connection tracker couldn’t identify the connection. This flag is a catch- all for problems in the connection or the connection tracker, such as: • L3/L4 protocol handler is not loaded/unavailable. With the Linux kernel datapath, this may mean that the nf_conntrack_ipv4 or nf_conntrack_ipv6 modules are not loaded. • L3/L4 protocol handler determines that the packet is malformed. • Packets are unexpected length for protocol. It appears that there may be an issue when patching the OS of a hypervisor not running instances may fail to update initrd to load nf_conntrack_ipv4 (and/or _ipv6). I couldn't find anywhere in the charm code that this would be loaded unless the charm's "harden" option is used on nova-compute charm (see charmhelpers contrib/host templates). It is unset in our environment, so we are not using any special module probing. Did nf_conntrack_ipv4 get split out from nf_conntrack in recent kernel upgrades or is it possible that the charm should define a modprobe file if we have the OVS firewall driver configured? To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1834213/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
