[Bug 1786752] Re: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release)
We have upgraded to kernel v4.20.15 and its going to be more than a week, we are not seeing any more call traces in this kernel logs. We will watch for few more days and i will report back here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1786752 Title: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1786752/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1786752] Re: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release)
# lspci 00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01) 00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01) 00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01) 00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01) 00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01) 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) 00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01) 00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01) 00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01) 00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01) 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05) 00:11.1 SMBus: Intel Corporation C610/X99 series chipset MS SMBus 0 (rev 05) 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05) 00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05) 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) 00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 (rev d5) 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05) 01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02) 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 06:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 09:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05) 7f:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) 7f:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) 7f:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01) 7f:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) 7f:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) 7f:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01) 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) 7f:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01) 7f:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R
[Bug 1786752] Re: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release)
We have upgraded the latest kernel and even occasionally we facing this call traces and CPU tainted, # uname -rn 4.15.0-45-generic # uname -a Linux 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux # dmesg -T | grep "Fri Mar 1.*CPU" [Fri Mar 1 10:28:42 2019] WARNING: CPU: 33 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:28:42 2019] CPU: 33 PID: 0 Comm: swapper/33 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 10:29:56 2019] WARNING: CPU: 12 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:29:56 2019] CPU: 12 PID: 0 Comm: swapper/12 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 10:53:01 2019] WARNING: CPU: 13 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:53:01 2019] WARNING: CPU: 1 PID: 18428 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:53:01 2019] WARNING: CPU: 5 PID: 18424 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:53:01 2019] CPU: 1 PID: 18428 Comm: haproxy Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 10:53:01 2019] CPU: 5 PID: 18424 Comm: haproxy Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 10:53:01 2019] CPU: 13 PID: 0 Comm: swapper/13 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 10:53:50 2019] WARNING: CPU: 5 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:53:50 2019] CPU: 5 PID: 0 Comm: swapper/5 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 12:29:32 2019] WARNING: CPU: 22 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 12:29:32 2019] CPU: 22 PID: 0 Comm: swapper/22 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:01:20 2019] WARNING: CPU: 31 PID: 20932 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:01:20 2019] CPU: 31 PID: 20932 Comm: thread.rb:70 Tainted: G W OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:01:23 2019] WARNING: CPU: 14 PID: 18429 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:01:23 2019] CPU: 14 PID: 18429 Comm: haproxy Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:01:23 2019] WARNING: CPU: 31 PID: 7532 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:01:23 2019] CPU: 31 PID: 7532 Comm: telegraf Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:16:28 2019] WARNING: CPU: 32 PID: 7842 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:16:28 2019] CPU: 32 PID: 7842 Comm: haproxy Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:48:17 2019] WARNING: CPU: 13 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:48:17 2019] CPU: 13 PID: 0 Comm: swapper/13 Tainted: GW OE4.15.0-45-generic #48-Ubuntu [Fri Mar 1 13:48:20 2019] WARNING: CPU: 5 PID: 41 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 13:48:20 2019] CPU: 5 PID: 41 Comm: ksoftirqd/5 Tainted: GW OE4.15.0-45-generic #48-Ubuntu # dmesg -T | grep "Fri Mar 1" [Fri Mar 1 10:28:42 2019] WARNING: CPU: 33 PID: 0 at /build/linux-uQJ2um/linux-4.15.0/include/net/dst.h:256 nf_xfrm_me_harder+0x127/0x140 [nf_nat] [Fri Mar 1 10:28:42 2019] Modules linked in: sctp xt_recent xfrm4_tunnel tunnel4 nfnetlink_queue nfnetlink_log ipcomp xfrm_ipcomp esp4 ah4 af_key tcp_diag udp_diag inet_diag sch_htb xt_TPROXY xt_multiport veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter bridge aufs overlay macvlan 8021q garp mrp stp llc bonding wireguard(OE) ip6_udp_tunnel udp_tunnel algif_skcipher af_alg xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_DSCP xt_dscp xt_mark dm_crypt iptable_mangle xt_limit xt_tcpudp xt_addrtype intel_rapl nf_conntrack_ipv4 nf_defrag_ipv4 sb_edac x86_pkg_temp_thermal intel_powerclamp xt_conntrack coretemp ipt_REJECT nf_reject_ipv4 kvm_intel kvm shpchp ftdi_sio usbserial irqbypass lpc_ich joydev input_leds [Fri Mar 1 10:28:42 2019] intel_cstate intel_rapl_perf mei_me mei ipmi_si ipmi_devintf ipmi_msghandler ioatdma ip6table_filter ip6_tables mac_hid acpi_power_meter acpi_pad nf_conntrack_netbios_ns nf_connt
[Bug 1786752] Re: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release)
Just to follow up , on behalf of Juergen. Do we have any progress with this bug ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1786752 Title: Reference counter issue in 4.15 (nf_xfrm_me_harder / dst_release) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1786752/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1809795] [NEW] getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP.
Public bug reported: We are using below version of Ubuntu/kernel, ~# lsb_release -rd Description:Ubuntu 18.04.1 LTS Release:18.04 What you expected to happen: getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) - should return always destination IP(connected IP). What happened instead: getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP. Using getsockname() instead looks like solving the issue. For just an example: Out of 6569124 requests , 4 requests were wrong 0.60891 % (this is just an rough estimate to give you idea on frequency) Some old reference: https://stackoverflow.com/questions/11417187/getsockopt-so-original-dst-occasionally-returns-client-address We searched in launchpad for any bug filed for this but couldn't see any. Hence filing this behavior here. Side Note: (about how we identified this behavior) We were using haproxy-version 1.8.14-1 which is using this kernel function getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) to get the destination IP details/connected address details. But instead we are getting very occasional source IP address instead of destination(Connected IP). ** Affects: dpkg (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1809795 Title: getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1809795/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
