Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

Hey guys,

I finally got some time to do a bisect and we narrowed the problem down to:

b838d5e1c5b6e57b10ec8af2268824041e3ea911 is the first bad commit
commit b838d5e1c5b6e57b10ec8af2268824041e3ea911
Author: Wei Wang 
Date:   Sat Jun 17 10:42:32 2017 -0700

ipv4: mark DST_NOGC and remove the operation of dst_free()

With the previous preparation patches, we are ready to get rid of the
dst gc operation in ipv4 code and release dst based on refcnt only.
So this patch adds DST_NOGC flag for all IPv4 dst and remove the calls
to dst_free().
At this point, all dst created in ipv4 code do not use the dst gc
anymore and will be destroyed at the point when refcnt drops to 0.

Signed-off-by: Wei Wang 
Acked-by: Martin KaFai Lau 
Signed-off-by: David S. Miller 

:04 04 9b7e7fb641de6531fc7887473ca47ef7cb6a11da 
831a73b71d3df1755f3e24c0d3c86d7a93fd55e2 M  net


I also saw there was a new thread some days ago reporting a similar problem. So
I put you guys (Wolfgang, Wei) into Cc.

Tobi

On Thu, Jun 14, 2018 at 10:38:01AM +0200, Kristian Evensen wrote:
> Hello,
> 
> On Tue, Jun 12, 2018 at 10:29 AM, Kristian Evensen
>  wrote:
> > Thanks for spending time on this. I will see what I can manage in
> > terms of a bisect. Our last good kernel was 4.9, so at least it
> > narrows the scope down a bit compared to 4.4 or 4.1.
> 
> I hope we might have got somewhere. While looking more into ipsec and
> 4.14, we noticed large performance regressions (-~20%) on some
> low-powered devices we are also using. We quickly identified the
> removal of the flow cache as the "culprit", and the performance
> regression is discussed in the netdev-thread for the removal of the
> cache ("xfrm: remove flow cache"). For the time being and in order to
> restore the performance, we have reverted the patch series removing
> the flow cache. When running our tests (on the APU) after the revert,
> we no longer see the crash. Before the revert, the APU would always
> crash within some hours. After the revert, our tests have been running
> for 24 hours+. Our test is quite basic, we establish 1, 2, 3 ...,  50
> tunnels and then run iperf on all tunnels in parallel. The tunnels are
> teared down between each iteration.
> 
> We are still running the test and will keep doing so, but I thought I
> should share this finding in case it can help in fixing the error. I
> will report back in case we find out something more, and please let me
> know if you have any suggestions for things I can test. I don't for
> example know if it is safe to revert one and one commit of the flow
> cache, to try to pin the crash even more down.
> 
> BR,
> Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Kristian Evensen]

Hello,

On Tue, Jun 12, 2018 at 10:29 AM, Kristian Evensen
 wrote:
> Thanks for spending time on this. I will see what I can manage in
> terms of a bisect. Our last good kernel was 4.9, so at least it
> narrows the scope down a bit compared to 4.4 or 4.1.

I hope we might have got somewhere. While looking more into ipsec and
4.14, we noticed large performance regressions (-~20%) on some
low-powered devices we are also using. We quickly identified the
removal of the flow cache as the "culprit", and the performance
regression is discussed in the netdev-thread for the removal of the
cache ("xfrm: remove flow cache"). For the time being and in order to
restore the performance, we have reverted the patch series removing
the flow cache. When running our tests (on the APU) after the revert,
we no longer see the crash. Before the revert, the APU would always
crash within some hours. After the revert, our tests have been running
for 24 hours+. Our test is quite basic, we establish 1, 2, 3 ...,  50
tunnels and then run iperf on all tunnels in parallel. The tunnels are
teared down between each iteration.

We are still running the test and will keep doing so, but I thought I
should share this finding in case it can help in fixing the error. I
will report back in case we find out something more, and please let me
know if you have any suggestions for things I can test. I don't for
example know if it is safe to revert one and one commit of the flow
cache, to try to pin the crash even more down.

BR,
Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Fri, Jun 08, 2018 at 10:41:37AM +0200, Kristian Evensen wrote:
> Hi,
> 
> On Wed, Jun 6, 2018 at 6:03 PM, Tobias Hommel  
> wrote:
> > Sorry no progress until now, I currently do not get time to have a deeper 
> > look
> > into that. We're back to 4.1.6 right now.
> 
> Thanks for letting me know. In the project I am currently involved in,
> we unfortunately don't have the option of reverting the kernel, so we
> are finding ways to live with the error. We have been looking into the
> error a bit more, and have made the following observations:
> 
> * First of all, as discussed earlier in the thread, the error is
> triggered by dst_orig being NULL. Our current work-around is just to
> return from xfrm_lookup if dst_orig is NULL and this seems to work
> fine, the error doesn't happen that often (in our use-cases at least).
> * The machine we use for testing (and where we first saw the error) is
> used as initiator.
The machine where I encountered the bug is a "roadwarrior gateway", so it only
serves as a responder.

> * When we compare the logs from Strongswan with the ones from the
> kernel, it seems that the error is typically triggered when a tunnels
> is teared down/about to come up. We need quite a lot of tunnels for
> the error to trigger, usually around 30+. I guess this might point to
> some race or some condition not being met when packets are
> sent/received.
> * We see the error much more frequently when hardware encryption is enabled.
> * Yesterday, we upgraded the kernel from 4.14.34 to 4.14.48, and the
> error happens much less frequently. I see that 4.14.48 includes
> several IPsec fixes (for example the previously mentioned ("xfrm: Fix
> a race in the xdst pcpu cache.")).
> 
> BR,
> Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Kristian Evensen]

Hi,

On Tue, Jun 12, 2018 at 10:03 AM, Steffen Klassert
 wrote:
> I spent quite some time again yesterday in trying to find a
> case where dst_orig can be NULL in xfrm_lookup(). I don't see
> how this can happen, so I fear we need a bisection on this.

Thanks for spending time on this. I will see what I can manage in
terms of a bisect. Our last good kernel was 4.9, so at least it
narrows the scope down a bit compared to 4.4 or 4.1.

BR,
Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Fri, Jun 08, 2018 at 10:41:37AM +0200, Kristian Evensen wrote:
> Hi,
> 
> On Wed, Jun 6, 2018 at 6:03 PM, Tobias Hommel  
> wrote:
> > Sorry no progress until now, I currently do not get time to have a deeper 
> > look
> > into that. We're back to 4.1.6 right now.
> 
> Thanks for letting me know. In the project I am currently involved in,
> we unfortunately don't have the option of reverting the kernel, so we
> are finding ways to live with the error. We have been looking into the
> error a bit more, and have made the following observations:
> 
> * First of all, as discussed earlier in the thread, the error is
> triggered by dst_orig being NULL.

I spent quite some time again yesterday in trying to find a
case where dst_orig can be NULL in xfrm_lookup(). I don't see
how this can happen, so I fear we need a bisection on this.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Kristian Evensen]

Hi,

On Wed, Jun 6, 2018 at 6:03 PM, Tobias Hommel  wrote:
> Sorry no progress until now, I currently do not get time to have a deeper look
> into that. We're back to 4.1.6 right now.

Thanks for letting me know. In the project I am currently involved in,
we unfortunately don't have the option of reverting the kernel, so we
are finding ways to live with the error. We have been looking into the
error a bit more, and have made the following observations:

* First of all, as discussed earlier in the thread, the error is
triggered by dst_orig being NULL. Our current work-around is just to
return from xfrm_lookup if dst_orig is NULL and this seems to work
fine, the error doesn't happen that often (in our use-cases at least).
* The machine we use for testing (and where we first saw the error) is
used as initiator.
* When we compare the logs from Strongswan with the ones from the
kernel, it seems that the error is typically triggered when a tunnels
is teared down/about to come up. We need quite a lot of tunnels for
the error to trigger, usually around 30+. I guess this might point to
some race or some condition not being met when packets are
sent/received.
* We see the error much more frequently when hardware encryption is enabled.
* Yesterday, we upgraded the kernel from 4.14.34 to 4.14.48, and the
error happens much less frequently. I see that 4.14.48 includes
several IPsec fixes (for example the previously mentioned ("xfrm: Fix
a race in the xdst pcpu cache.")).

BR,
Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

Hi,

On Wed, Jun 06, 2018 at 12:41:53PM +0200, Kristian Evensen wrote:
> Hi,
> 
> I am experiencing the same issue on a PC Engines APU2 running kernel
> 4.14.34, both with and without hardware encryption. With hw.
> encryption, the crash occurs within 2-4 hours. Without hw. encryption,
> it takes 7-8 hours. My setup is nothing crazy, between 7 and 20
> tunnels with heavy RX/TX.
> 
> On Fri, Feb 2, 2018 at 9:09 AM, Steffen Klassert
>  wrote:
> > Thanks for offering help, but I fear we have to wait until
> > Tobias has bisected it.
> 
> Has any progress been made here? I would like to try a bisect myself,

Sorry no progress until now, I currently do not get time to have a deeper look
into that. We're back to 4.1.6 right now.

> but my board is running OpenWRT, which makes bisecting hard. The only
> observation I have so far, is that I did not see the issue with kernel
> 4.9.
> 
> BR,
> Kristian

Tobi

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Kristian Evensen]

On Wed, Jun 6, 2018 at 12:41 PM, Kristian Evensen
 wrote:
> Hi,
>
> I am experiencing the same issue on a PC Engines APU2 running kernel
> 4.14.34, both with and without hardware encryption. With hw.
> encryption, the crash occurs within 2-4 hours. Without hw. encryption,
> it takes 7-8 hours. My setup is nothing crazy, between 7 and 20
> tunnels with heavy RX/TX.

I am very sorry, I forgot to c the oops. Here it is:

[26675.130763] BUG: unable to handle kernel NULL pointer dereference
at 0020
[26675.139335] IP: xfrm_lookup+0x1d/0x7b0
[26675.143318] PGD 11726e067 P4D 11726e067 PUD 11697e067 PMD 0
[26675.149542] Oops:  [#1] SMP NOPTI
[26675.153451] Modules linked in: qcserial ppp_async option cdc_mbim
usb_wwan rt2800pci rt2800mmio rt2800lib rndis_host qmi_wwan
ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6
iptable_nat ipt_REJECT ipt_MASQUERADE huawei_cdc_ncm cdc_ncm cdc_ether
ax88179_178a asix xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic
xt_state xt_recent xt_quota xt_policy xt_pkttype xt_owner xt_nfacct
xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper
xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit
xt_connlabel xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT
xt_NFQUEUE xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet
ts_fsm ts_bm slhc rt2x00pci rt2x00mmio rt2x00lib r8169 nfnetlink_queue
nfnetlink_acct nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip
nf_nat_redirect
[26675.227953]  nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc
nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_ftp nf_nat_amanda
nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp
nf_conntrack_snmp nf_conntrack_sip nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323
nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda
lib80211_crypt_wep lib80211_crypt_tkip lib80211_crypt_ccmp lib80211
iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip6table_raw
ip_tables crc_itu_t crc_ccitt compat_xtables cdc_wdm br_netfilter
ath10k_pci ath10k_core sch_cake ath9k ath9k_common ath9k_hw ath
mac80211 cfg80211 compat act_connmark nf_conntrack act_skbedit
act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw
sch_tbf sch_htb sch_hfsc sch_ingress
[26675.300777]  evdev i2c_dev cryptodev xt_set ip_set_list_set
ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet
ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac
ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport
ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT
nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle
ip6table_filter ip6_tables x_tables igb i2c_algo_bit e1000 ifb ipcomp6
xfrm6_tunnel xfrm6_mode_tunnel xfrm6_mode_transport xfrm6_mode_beet
esp6 ah6 ipcomp xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport
xfrm4_mode_beet esp4 ah4 tunnel6 tunnel4 tun af_key xfrm_user
xfrm_ipcomp xfrm_algo eeprom_93cx6 sha256_ssse3 sha1_ssse3
sha1_generic michael_mic md5 echainiv des_generic deflate zlib_deflate
[26675.373443]  cmac authenc crypto_acompress xhci_plat_hcd uhci_hcd
leds_apu2 ehci_platform gpio_button_hotplug ptp pps_core mii libphy
[26675.385812] CPU: 3 PID: 24 Comm: ksoftirqd/3 Not tainted 4.14.34 #0
[26675.392308] Hardware name: PC Engines apu2/apu2, BIOS 88a4f96 03/07/2016
[26675.399200] task: 88011a0fc680 task.stack: c90e
[26675.405368] RIP: 0010:xfrm_lookup+0x1d/0x7b0
[26675.409736] RSP: 0018:c90e3b68 EFLAGS: 00010292
[26675.415066] RAX:  RBX:  RCX: 
[26675.422407] RDX: c90e3be8 RSI:  RDI: 81c6b480
[26675.429732] RBP: c90e3bd8 R08: 0002 R09: 0001
[26675.437107] R10: 0032 R11:  R12: 880112954800
[26675.444524] R13: 0002 R14: c90e3be8 R15: 81c6b480
[26675.451876] FS:  () GS:88011ed8()
knlGS:
[26675.460101] CS:  0010 DS:  ES:  CR0: 80050033
[26675.465971] CR2: 0020 CR3: 000116928000 CR4: 000406e0
[26675.473282] Call Trace:
[26675.475801]  ? __xfrm_policy_check+0x1cc/0x520
[26675.480419]  __xfrm_route_forward+0x96/0xe0
[26675.484698]  ip_forward+0x10f/0x3a0
[26675.488233]  ? ip_route_input_noref+0x14/0x20
[26675.492713]  ip_rcv_finish+0x26d/0x2c0
[26675.496525]  ip_rcv+0x253/0x2c0
[26675.499767]  ? ip_local_deliver_finish+0x180/0x180
[26675.504752]  __netif_receive_skb_core+0x7b6/0x920
[26675.509577]  __netif_receive_skb+0x54/0x60
[26675.513754]  ? __netif_receive_skb+0x54/0x60
[26675.518140]  process_backlog+0x91/0x130
[26675.522194]  net_rx_action+0xf3/0x250
[26675.525933]  __do_softirq+0xb5/0x1d1
[26675.529600]  run_ksoftirqd+0x1b/0x40
[26675.533275]  smpboot_thread_fn+0x152/0x170
[26675.537538]  kthread+0x112/0x120
[26675.540873]  ? sort_range+0x20/0x20

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Kristian Evensen]

Hi,

I am experiencing the same issue on a PC Engines APU2 running kernel
4.14.34, both with and without hardware encryption. With hw.
encryption, the crash occurs within 2-4 hours. Without hw. encryption,
it takes 7-8 hours. My setup is nothing crazy, between 7 and 20
tunnels with heavy RX/TX.

On Fri, Feb 2, 2018 at 9:09 AM, Steffen Klassert
 wrote:
> Thanks for offering help, but I fear we have to wait until
> Tobias has bisected it.

Has any progress been made here? I would like to try a bisect myself,
but my board is running OpenWRT, which makes bisecting hard. The only
observation I have so far, is that I did not see the issue with kernel
4.9.

BR,
Kristian

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Wed, Jan 31, 2018 at 09:26:51PM +0100, Markus Berner wrote:
> > I'm running into a NULL pointer dereference after updating from Linux
> 4.1.6 to
> > 4.14.11 (see kernel log below).
> 
> We are running into the same problem on our production machine, running
> CoreOS 1576.5.0 Stable with the 4.14.11 kernel on a KVM Cloud VM. It is not
> as easy to reproduce though in our case – we observed a total of 5 crashes
> in the last 2 weeks - all except one on the production machine.
> 
> > I still can't reproduce it with my tests. This is probably some race
> > triggered due to your aggressive roadwarrior setup which I don't have.
> 
> We have a similar setup to Tobias
> - 2 Network Interfaces (KVM/virtio): Public and local VLAN
> - Strongswan VPN in Tunnel mode between local VLAN and on-premise network,
> running in a Docker container
> - Quite a few iptables NAT and forwarding rules regarding other local Docker
> containers
> 
> Some Observations:
> - The workaround of locking the IRQs of the Rx/Tx queues of all network
> interfaces to CPU0 Tobias described a while back did not prevent the crashes
> in our case
> - The bug does not seem to correlate with load in our case, but load in
> general is quite low.
> 
> I am happy to help if I can, but unfortunately our possibilities are a bit
> limited; both due to lack of kernel dev know-how as well as trying out
> changes to configuration on the production machine. I subscribed to LKML
> only now to respond, so I hope the reply works (and to the correct message).

Thanks for offering help, but I fear we have to wait until
Tobias has bisected it.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Markus Berner]

> I'm running into a NULL pointer dereference after updating from Linux 
4.1.6 to

> 4.14.11 (see kernel log below).

We are running into the same problem on our production machine, running 
CoreOS 1576.5.0 Stable with the 4.14.11 kernel on a KVM Cloud VM. It is 
not as easy to reproduce though in our case – we observed a total of 5 
crashes in the last 2 weeks - all except one on the production machine.


> I still can't reproduce it with my tests. This is probably some race
> triggered due to your aggressive roadwarrior setup which I don't have.

We have a similar setup to Tobias
- 2 Network Interfaces (KVM/virtio): Public and local VLAN
- Strongswan VPN in Tunnel mode between local VLAN and on-premise 
network, running in a Docker container
- Quite a few iptables NAT and forwarding rules regarding other local 
Docker containers


Some Observations:
- The workaround of locking the IRQs of the Rx/Tx queues of all network 
interfaces to CPU0 Tobias described a while back did not prevent the 
crashes in our case
- The bug does not seem to correlate with load in our case, but load in 
general is quite low.


I am happy to help if I can, but unfortunately our possibilities are a 
bit limited; both due to lack of kernel dev know-how as well as trying 
out changes to configuration on the production machine. I subscribed to 
LKML only now to respond, so I hope the reply works (and to the correct 
message).


Markus

Example Stack Trace below:

[740051.374799] BUG: unable to handle kernel NULL pointer dereference at 
0020

[740051.379386] IP: xfrm_lookup+0x32/0x8a0
[740051.379941] PGD 8004648d6067 P4D 8004648d6067 PUD 461405067 
PMD 0

[740051.380697] Oops:  [#1] SMP PTI
[740051.381060] Modules linked in: iptable_mangle drbg authenc echainiv 
esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel cbc binfmt_misc veth netconsole 
configfs softdog xt_nat nf_log_ipv4 nf_log_common xt_LOG xt_limit 
xt_policy xt_comment xt_multiport ipt_MASQUERADE nf_nat_masquerade_ipv4 
nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat 
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter 
xt_conntrack nf_nat nf_conntrack libcrc32c crc32c_generic br_netfilter 
bridge stp llc overlay sb_edac edac_core nls_ascii nls_cp437 kvm_intel 
vfat fat kvm mousedev psmouse i2c_piix4 irqbypass evdev virtio_balloon 
i2c_core pvpanic button sch_fq_codel hid_generic usbhid hid ext4 crc16 
mbcache jbd2 fscrypto dm_verity dm_bufio virtio_blk virtio_net uhci_hcd 
ehci_pci ata_piix ehci_hcd crc32c_intel
[740051.389167]  libata virtio_pci usbcore virtio_ring scsi_mod virtio 
usb_common dm_mirror dm_region_hash dm_log dm_mod dax

[740051.391444] CPU: 2 PID: 13516 Comm: java Not tainted 4.14.11-coreos #1
[740051.392792] Hardware name: QEMU CloudSigma, BIOS Bochs 01/01/2011
[740051.394120] task: 903022738000 task.stack: a791c768
[740051.395399] RIP: 0010:xfrm_lookup+0x32/0x8a0
[740051.396456] RSP: 0018:9030bfc838e8 EFLAGS: 00010246
[740051.397656] RAX:  RBX: 9030bfc83960 RCX: 

[740051.399526] RDX: 9030bfc83960 RSI:  RDI: 

[740051.401471] RBP:  R08: 0002 R09: 
46d7a6d9
[740051.403260] R10:  R11: 62f99322 R12: 
0002
[740051.405049] R13: 980e3080 R14: 903095c8a0a0 R15: 
9812dc20
[740051.406767] FS:  7f7d68cf6700() GS:9030bfc8() 
knlGS:

[740051.408900] CS:  0010 DS:  ES:  CR0: 80050033
[740051.410159] CR2: 0020 CR3: 0005fd444005 CR4: 
001606e0

[740051.411805] Call Trace:
[740051.412534]  
[740051.413190]  __xfrm_route_forward+0x61/0x100
[740051.414198]  ip_forward+0x39e/0x470
[740051.415148]  ? ip_rcv_finish+0xa5/0x3f0
[740051.416225]  br_netfilter_enable+0x10c/0x3e0 [br_netfilter]
[740051.417491]  nf_hook_slow+0x39/0xb0
[740051.418530]  ip_rcv+0x303/0x3a0
[740051.419647]  ? inet_del_offload+0x40/0x40
[740051.420303]  __netif_receive_skb_core+0x2c9/0xb60
[740051.420998]  ? x2apic_send_IPI+0x46/0x50
[740051.421648]  ? check_preempt_curr+0x56/0x90
[740051.422309]  ? ttwu_do_wakeup+0x19/0x150
[740051.422920]  ? netif_receive_skb_internal+0x42/0xf0
[740051.423710]  netif_receive_skb_internal+0x42/0xf0
[740051.424717]  br_port_flags_change+0x1d4/0x260 [bridge]
[740051.425884]  ? br_fdb_update+0xc3/0x2c0 [bridge]
[740051.427019]  br_handle_frame_finish+0x1e2/0x510 [bridge]
[740051.428339]  ? lock_timer_base+0x67/0x80
[740051.429036]  ? ipt_do_table+0x35f/0x610
[740051.429906]  ? br_port_flags_change+0x260/0x260 [bridge]
[740051.430683]  br_nf_hook_thresh+0xde/0x12a0 [br_netfilter]
[740051.431465]  ? br_port_flags_change+0x260/0x260 [bridge]
[740051.432217]  br_nf_hook_thresh+0xa8c/0x12a0 [br_netfilter]
[740051.433032]  ? br_port_flags_change+0x260/0x260 [bridge]
[740051.433966]  ? nf_nat_ipv4_in+0x28/0x80 [nf_nat_ipv4]
[740051.434848]  br_nf_hook_thresh+0xe2a/0x12a0 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Wed, Jan 24, 2018 at 10:59:21AM +0100, Steffen Klassert wrote:
> On Fri, Jan 19, 2018 at 03:45:46PM +0100, Tobias Hommel wrote:
> > 
> > I tried to strip down the system configuration and was able to reproduce the
> > problem with a minimal configuration:
> > * ipsets are not used anymore
> > * no firewall markings are used any longer
> > * iptables are "completely empty", i.e. all policies set to ACCEPT and 
> > there is
> >   no rule in any table
> > * no additional routing policies (ip rule) except the default ones
> > * only main routing table is used
> > * using a "minimal" kernel config:
> >  * run `make defconfig`
> >  * add basic things (ESP, IGB driver, some crypto algorithms)
> >  * add options required to boot up the system (TPM crypt, some device mapper
> >options, overlayfs)
> > 
> > I attached the minimal config (minimal.config) and the defconfig for 
> > reference
> > (minimal.defconfig).
> > 
> > The setup is really simple now, the gateway is forwarding HTTP connections
> > between eth1(IPSec tunnels) and eth0 without any firewall, NAT, whatsoever.
> 
> Thanks a lot for your debugging effort!
> 
> > 
> > The only thing I can think of are the rather aggressive roadwarrior clients.
> > There are 750 roadwarriors that are controlled by a script which starts and
> > stops the IPSec connection.
> 
> I still can't reproduce it with my tests. This is probably some race
> triggered due to your aggressive roadwarrior setup which I don't have.
> 
> > I tried 4.15-rc8 and have the same problem here (see attached
> > kernel-4.15-rc8.log). SMP affinity for IRQs has changed in 4.15 and 
> > something's
> 
> There is one patch that could influence this which is not in v4.15-rc8:
> 
> commit 76a4201191814a0061cb5c861fafb9ecaa764846
> ("xfrm: Fix a race in the xdst pcpu cache.")
> 
> It is included in v4.15-rc9.
I already tested that one some weeks ago, when it appeared on the mailing list,
with 4.14. Without any luck.

> 
> If this does not fix your problem, I'm out of ideas. In this case
> I have to ask to do a bisection to find the offending commit.
> 
I'll do a bisect session then. It'll take some time though as the hardware is
currently occupied with other tests. I'll keep you up-to-date about the
results.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Fri, Jan 19, 2018 at 03:45:46PM +0100, Tobias Hommel wrote:
> 
> I tried to strip down the system configuration and was able to reproduce the
> problem with a minimal configuration:
> * ipsets are not used anymore
> * no firewall markings are used any longer
> * iptables are "completely empty", i.e. all policies set to ACCEPT and there 
> is
>   no rule in any table
> * no additional routing policies (ip rule) except the default ones
> * only main routing table is used
> * using a "minimal" kernel config:
>  * run `make defconfig`
>  * add basic things (ESP, IGB driver, some crypto algorithms)
>  * add options required to boot up the system (TPM crypt, some device mapper
>options, overlayfs)
> 
> I attached the minimal config (minimal.config) and the defconfig for reference
> (minimal.defconfig).
> 
> The setup is really simple now, the gateway is forwarding HTTP connections
> between eth1(IPSec tunnels) and eth0 without any firewall, NAT, whatsoever.

Thanks a lot for your debugging effort!

> 
> The only thing I can think of are the rather aggressive roadwarrior clients.
> There are 750 roadwarriors that are controlled by a script which starts and
> stops the IPSec connection.

I still can't reproduce it with my tests. This is probably some race
triggered due to your aggressive roadwarrior setup which I don't have.

> I tried 4.15-rc8 and have the same problem here (see attached
> kernel-4.15-rc8.log). SMP affinity for IRQs has changed in 4.15 and 
> something's

There is one patch that could influence this which is not in v4.15-rc8:

commit 76a4201191814a0061cb5c861fafb9ecaa764846
("xfrm: Fix a race in the xdst pcpu cache.")

It is included in v4.15-rc9.

If this does not fix your problem, I'm out of ideas. In this case
I have to ask to do a bisection to find the offending commit.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Wed, Jan 10, 2018 at 08:30:38AM +0100, Steffen Klassert wrote:
> On Tue, Jan 09, 2018 at 03:49:21PM +0100, Tobias Hommel wrote:
> > 
> > I copied the config from my 4.14.12 sources to a fresh 4.13.16 source tree, 
> > ran
> > `make olddefconfig` and built a new kernel.
> > The kernel config is attached as kernel-4.13.16.config.
> > The panic*.log files are kernel logs from different crashes of this 4.13.16
> > kernel, but all from the same scenario as before.
> > I also enabled CONFIG_DEBUG_INFO, so if any disassemblies are required, I'd 
> > be
> > happy to provide them.
> > 
> > So, the system still crashes, but the traces are completely different from
> > those with 4.14.12. This time there are also WARNINGs and "refcnt: -1" 
> > messages
> > sometimes before the actual panic, so not sure if there is maybe some other
> > problem. Still, the crashes all seem to be related to ip routing somehow.
> 
> Strange, you must do something that other people don't do.
> Do you have some uncommon netfiler rules, namespaces, etc?
No, no namespaces yet.
However, the box uses marks and routing based on marks. Firewall marks are a
bit strange sometimes, so I'll try to clean up everything and see if it is
possible to reproduce the bug without marks.

> 
> Please try to build your kernels with
> 
> CONFIG_ORC_UNWINDER (v4.14 and above)
> 
> and
> 
> CONFIG_KASAN
> 
> This can give some better debug informations (depends on the compiler
> version).
I'll also try that. I'm currently using GCC 5.4.0.

> 
> There are some things we can do now:
> 
> - Try v4.15-rc7, just to be sure that we don't search for
>   something that is already fixed.
And that one, too. All this will probably take some time though. ;-)
I'll keep you informed.

> 
> - Find a working kernel version and try to bisect.
> 
> - Minimalize the configuration with that the bug happens,
>   so that I can try to reproduce it here.
> 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Wed, Jan 10, 2018 at 08:42:19AM +0100, Tobias Hommel wrote:
> On Tue, Jan 09, 2018 at 03:49:21PM +0100, Tobias Hommel wrote:
> > On Tue, Jan 09, 2018 at 10:26:24AM +0100, Steffen Klassert wrote:
> > > On Tue, Jan 09, 2018 at 10:06:51AM +0100, Tobias Hommel wrote:
> > > > > 
> > > > > You have CONFIG_INET_ESP_OFFLOAD enabled, this is new maybe it
> > > > > still has some problems. You should not hit an offload codepath
> > > > > because all your SAs are configured with UDP encapsulation which
> > > > > is still not supported with offload.
> I ran some new tests with 4.14.12. This time I removed encap=yes from the
> strongswan config so I have plain ESP tunnels, without UDP encapsulation. Just
> to be sure. It still crashes, the attached panic.noencap.log is pretty much
> the same as the logs before.

The bug happens before any xfrm processing, so not sure where it comes
from. It might be that we operate one an already feed object, I hope
that KASAN can tell us more.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Tue, Jan 09, 2018 at 03:49:21PM +0100, Tobias Hommel wrote:
> On Tue, Jan 09, 2018 at 10:26:24AM +0100, Steffen Klassert wrote:
> > On Tue, Jan 09, 2018 at 10:06:51AM +0100, Tobias Hommel wrote:
> > > > 
> > > > You have CONFIG_INET_ESP_OFFLOAD enabled, this is new maybe it
> > > > still has some problems. You should not hit an offload codepath
> > > > because all your SAs are configured with UDP encapsulation which
> > > > is still not supported with offload.
I ran some new tests with 4.14.12. This time I removed encap=yes from the
strongswan config so I have plain ESP tunnels, without UDP encapsulation. Just
to be sure. It still crashes, the attached panic.noencap.log is pretty much
the same as the logs before.

> > > > 
> > > > Please try to disable GRO on both interfaces and see what happens:
> > > > 
> > > > ethtool -K eth0 gro off
> > > > ethtool -K eth1 gro off
> > > I actually already tried that with only eth1 off, to verify I turned 
> > > offloading
> > > off for both interfaces. The same problem: see attached panic.gro_off.log
> > > 
> > > > 
> > > > Then disable CONFIG_INET_ESP_OFFLOAD and try again.
> > > Rebuild with CONFIG_INET_ESP_OFFLOAD disabled, same problem: see attached
> > > panic.esp_offload_disabled.log
> > 
> > So ESP offload is not the problem. Next thing that comes to my mind
> > is the flowcache removal, this was introduced with v4.14.
> > 
> > > 
> > > > 
> > > > This should show us if this feature is responsible for the bug.
> > > > 
> > > 
> > > I will try narrowing down the problem by trying out some older kernels 
> > > for now.
> > 
> > Thanks!
> > 
> > Let me know about the results.
> 
> I copied the config from my 4.14.12 sources to a fresh 4.13.16 source tree, 
> ran
> `make olddefconfig` and built a new kernel.
> The kernel config is attached as kernel-4.13.16.config.
> The panic*.log files are kernel logs from different crashes of this 4.13.16
> kernel, but all from the same scenario as before.
> I also enabled CONFIG_DEBUG_INFO, so if any disassemblies are required, I'd be
> happy to provide them.
> 
> So, the system still crashes, but the traces are completely different from
> those with 4.14.12. This time there are also WARNINGs and "refcnt: -1" 
> messages
> sometimes before the actual panic, so not sure if there is maybe some other
> problem. Still, the crashes all seem to be related to ip routing somehow.
[ 2298.720212] BUG: unable to handle kernel NULL pointer dereference at 
0020
[ 2298.728193] IP: xfrm_lookup+0x2a/0x7d0
[ 2298.731986] PGD 0 P4D 0 
[ 2298.734535] Oops:  [#1] SMP PTI
[ 2298.738035] Modules linked in:
[ 2298.741121] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.14.12 #3
[ 2298.747362] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 1.01 
07/11/2016
[ 2298.755136] task: a0dafb08dc00 task.stack: a211c004
[ 2298.761091] RIP: 0010:xfrm_lookup+0x2a/0x7d0
[ 2298.765403] RSP: 0018:a211c0043ad0 EFLAGS: 00010246
[ 2298.770656] RAX:  RBX: 87074080 RCX: 
[ 2298.777851] RDX: a211c0043b48 RSI:  RDI: 87074080
[ 2298.785025] RBP: 87074080 R08: 0002 R09: 
[ 2298.792184] R10: 0020 R11: 0020 R12: a211c0043b48
[ 2298.799351] R13:  R14: 0002 R15: a0dafb240078
[ 2298.806511] FS:  () GS:a0daffc0() 
knlGS:
[ 2298.814647] CS:  0010 DS:  ES:  CR0: 80050033
[ 2298.820428] CR2: 0020 CR3: 000177dcc000 CR4: 001006f0
[ 2298.827587] Call Trace:
[ 2298.830045]  __xfrm_route_forward+0xa4/0x110
[ 2298.834340]  ip_forward+0x3da/0x450
[ 2298.837851]  ? ip_rcv_finish+0x61/0x390
[ 2298.841708]  ip_rcv+0x2b5/0x380
[ 2298.844871]  ? inet_del_offload+0x30/0x30
[ 2298.848910]  __netif_receive_skb_core+0x751/0xb00
[ 2298.853640]  ? netif_receive_skb_internal+0x47/0xf0
[ 2298.858573]  ? inet_gro_receive+0x1fa/0x2a0
[ 2298.862785]  netif_receive_skb_internal+0x47/0xf0
[ 2298.867523]  dev_gro_receive+0x270/0x440
[ 2298.871487]  napi_gro_receive+0x28/0x90
[ 2298.875350]  igb_poll+0x600/0xe80
[ 2298.878695]  net_rx_action+0x1fc/0x310
[ 2298.882478]  __do_softirq+0xd5/0x1cf
[ 2298.886064]  run_ksoftirqd+0x14/0x30
[ 2298.889670]  smpboot_thread_fn+0xf9/0x150
[ 2298.893707]  kthread+0xf2/0x130
[ 2298.896869]  ? sort_range+0x20/0x20
[ 2298.900387]  ? kthread_park+0x60/0x60
[ 2298.904080]  ret_from_fork+0x1f/0x30
[ 2298.907684] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 89 
d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 <48> 8b 
46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84 
[ 2298.92] RIP: xfrm_lookup+0x2a/0x7d0 RSP: a211c0043ad0
[ 2298.932447] CR2: 0020
[ 2298.935792] ---[ end trace 4045e2796d0dd0c8 ]---
[ 2298.940446] Kernel panic - not syncing: Fatal exception in interrupt
[ 2298.946937] Kernel Offset: 0x500 from 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Tue, Jan 09, 2018 at 03:49:21PM +0100, Tobias Hommel wrote:
> 
> I copied the config from my 4.14.12 sources to a fresh 4.13.16 source tree, 
> ran
> `make olddefconfig` and built a new kernel.
> The kernel config is attached as kernel-4.13.16.config.
> The panic*.log files are kernel logs from different crashes of this 4.13.16
> kernel, but all from the same scenario as before.
> I also enabled CONFIG_DEBUG_INFO, so if any disassemblies are required, I'd be
> happy to provide them.
> 
> So, the system still crashes, but the traces are completely different from
> those with 4.14.12. This time there are also WARNINGs and "refcnt: -1" 
> messages
> sometimes before the actual panic, so not sure if there is maybe some other
> problem. Still, the crashes all seem to be related to ip routing somehow.

Strange, you must do something that other people don't do.
Do you have some uncommon netfiler rules, namespaces, etc?

Please try to build your kernels with

CONFIG_ORC_UNWINDER (v4.14 and above)

and

CONFIG_KASAN

This can give some better debug informations (depends on the compiler
version).

There are some things we can do now:

- Try v4.15-rc7, just to be sure that we don't search for
  something that is already fixed.

- Find a working kernel version and try to bisect.

- Minimalize the configuration with that the bug happens,
  so that I can try to reproduce it here.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Tue, Jan 09, 2018 at 10:06:51AM +0100, Tobias Hommel wrote:
> > 
> > You have CONFIG_INET_ESP_OFFLOAD enabled, this is new maybe it
> > still has some problems. You should not hit an offload codepath
> > because all your SAs are configured with UDP encapsulation which
> > is still not supported with offload.
> > 
> > Please try to disable GRO on both interfaces and see what happens:
> > 
> > ethtool -K eth0 gro off
> > ethtool -K eth1 gro off
> I actually already tried that with only eth1 off, to verify I turned 
> offloading
> off for both interfaces. The same problem: see attached panic.gro_off.log
> 
> > 
> > Then disable CONFIG_INET_ESP_OFFLOAD and try again.
> Rebuild with CONFIG_INET_ESP_OFFLOAD disabled, same problem: see attached
> panic.esp_offload_disabled.log

So ESP offload is not the problem. Next thing that comes to my mind
is the flowcache removal, this was introduced with v4.14.

> 
> > 
> > This should show us if this feature is responsible for the bug.
> > 
> 
> I will try narrowing down the problem by trying out some older kernels for 
> now.

Thanks!

Let me know about the results.

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Tue, Jan 09, 2018 at 09:19:39AM +0100, Steffen Klassert wrote:
> On Mon, Jan 08, 2018 at 02:53:48PM +0100, Tobias Hommel wrote:
> 
> ...
> 
> > [  439.095554] BUG: unable to handle kernel NULL pointer dereference at 
> > 0020
> > [  439.103664] IP: xfrm_lookup+0x2a/0x7d0
> > [  439.107551] PGD 0 P4D 0 
> > [  439.110144] Oops:  [#1] SMP PTI
> > [  439.113653] Modules linked in:
> > [  439.116774] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.14.12 #1
> > [  439.122900] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 
> > 1.01 07/11/2016
> > [  439.130769] task: 8cf33b0ea280 task.stack: 9492c009
> > [  439.136726] RIP: 0010:xfrm_lookup+0x2a/0x7d0
> > [  439.141005] RSP: 0018:8cf33fd83bd0 EFLAGS: 00010246
> > [  439.146315] RAX:  RBX: 87074080 RCX: 
> > 
> > [  439.153537] RDX: 8cf33fd83c48 RSI:  RDI: 
> > 87074080
> > [  439.160780] RBP: 87074080 R08: 0002 R09: 
> > 
> > [  439.167958] R10: 0020 R11: 0020 R12: 
> > 8cf33fd83c48
> > [  439.175115] R13:  R14: 0002 R15: 
> > 8cf33b240078
> > [  439.182337] FS:  () GS:8cf33fd8() 
> > knlGS:
> > [  439.190456] CS:  0010 DS:  ES:  CR0: 80050033
> > [  439.196227] CR2: 0020 CR3: 00013200a000 CR4: 
> > 001006e0
> > [  439.203386] Call Trace:
> > [  439.205869]  
> > [  439.207886]  __xfrm_route_forward+0xa4/0x110
> > [  439.212195]  ip_forward+0x3da/0x450
> > [  439.215696]  ? ip_rcv_finish+0x61/0x390
> > [  439.219542]  ip_rcv+0x2b5/0x380
> > [  439.222716]  ? inet_del_offload+0x30/0x30
> > [  439.226736]  __netif_receive_skb_core+0x751/0xb00
> > [  439.231469]  ? netif_receive_skb_internal+0x47/0xf0
> > [  439.236391]  netif_receive_skb_internal+0x47/0xf0
> > [  439.241150]  napi_gro_flush+0x50/0x70
> > [  439.244831]  napi_complete_done+0x90/0xd0
> > [  439.248872]  igb_poll+0x8fd/0xe80
> > [  439.252190]  net_rx_action+0x1fc/0x310
> > [  439.255978]  __do_softirq+0xd5/0x1cf
> > [  439.259584]  irq_exit+0xa3/0xb0
> > [  439.262763]  do_IRQ+0x45/0xc0
> > [  439.265772]  common_interrupt+0x95/0x95
> > [  439.269609]  
> > [  439.271733] RIP: 0010:cpuidle_enter_state+0x120/0x200
> > [  439.276810] RSP: 0018:9492c0093eb8 EFLAGS: 0282 ORIG_RAX: 
> > ff5d
> > [  439.284436] RAX: 8cf33fd9ea80 RBX: 0002 RCX: 
> > 00663c21ea0f
> > [  439.291604] RDX:  RSI: 36ca RDI: 
> > 
> > [  439.298772] RBP: 8cf33fda71e8 R08: 0003 R09: 
> > 0018
> > [  439.305930] R10:  R11: 057c R12: 
> > 00663c21ea0f
> > [  439.313089] R13: 00663c1c6c33 R14: 0002 R15: 
> > 
> > [  439.320259]  ? cpuidle_enter_state+0x11c/0x200
> > [  439.324740]  do_idle+0xd6/0x170
> > [  439.327885]  cpu_startup_entry+0x67/0x70
> > [  439.331837]  start_secondary+0x167/0x190
> > [  439.335788]  secondary_startup_64+0xa5/0xb0
> > [  439.340001] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 
> > 89 d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 
> > <48> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84 
> > [  439.358988] RIP: xfrm_lookup+0x2a/0x7d0 RSP: 8cf33fd83bd0
> > [  439.364759] CR2: 0020
> > [  439.368105] ---[ end trace c6b298b556ea7769 ]---
> > [  439.372752] Kernel panic - not syncing: Fatal exception in interrupt
> > [  439.379255] Kernel Offset: 0x500 from 0x8100 (relocation 
> > range: 0x8000-0xbfff)
> > [  439.390029] Rebooting in 10 seconds..
> 
> ...
> 
> > 4230 :
> > 4230:   41 57   push   %r15
> > 4232:   41 56   push   %r14
> > 4234:   45 89 c6mov%r8d,%r14d
> > 4237:   41 55   push   %r13
> > 4239:   41 54   push   %r12
> > 423b:   49 89 f5mov%rsi,%r13
> > 423e:   55  push   %rbp
> > 423f:   53  push   %rbx
> > 4240:   49 89 d4mov%rdx,%r12
> > 4243:   48 89 fbmov%rdi,%rbx
> > 4246:   48 83 ec 40 sub$0x40,%rsp
> > 424a:   65 48 8b 04 25 28 00mov%gs:0x28,%rax
> > 4251:   00 00 
> > 4253:   48 89 44 24 38  mov%rax,0x38(%rsp)
> > 4258:   31 c0   xor%eax,%eax
> > 425a:   48 8b 46 20 mov0x20(%rsi),%rax
> 
> 
> The above is the failing instruction, RSI holds the second argument
> of the called function which is a NULL pointer. The second argument
> of xfrm_lookup() is dst_orig, so it is as I thought. Now let's find
> out why. I don't see anything obvious, so we need to narrow it down.
> 
> > CONFIG_INET_ESP=y
> 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Mon, Jan 08, 2018 at 02:53:48PM +0100, Tobias Hommel wrote:

...

> [  439.095554] BUG: unable to handle kernel NULL pointer dereference at 
> 0020
> [  439.103664] IP: xfrm_lookup+0x2a/0x7d0
> [  439.107551] PGD 0 P4D 0 
> [  439.110144] Oops:  [#1] SMP PTI
> [  439.113653] Modules linked in:
> [  439.116774] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.14.12 #1
> [  439.122900] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 1.01 
> 07/11/2016
> [  439.130769] task: 8cf33b0ea280 task.stack: 9492c009
> [  439.136726] RIP: 0010:xfrm_lookup+0x2a/0x7d0
> [  439.141005] RSP: 0018:8cf33fd83bd0 EFLAGS: 00010246
> [  439.146315] RAX:  RBX: 87074080 RCX: 
> 
> [  439.153537] RDX: 8cf33fd83c48 RSI:  RDI: 
> 87074080
> [  439.160780] RBP: 87074080 R08: 0002 R09: 
> 
> [  439.167958] R10: 0020 R11: 0020 R12: 
> 8cf33fd83c48
> [  439.175115] R13:  R14: 0002 R15: 
> 8cf33b240078
> [  439.182337] FS:  () GS:8cf33fd8() 
> knlGS:
> [  439.190456] CS:  0010 DS:  ES:  CR0: 80050033
> [  439.196227] CR2: 0020 CR3: 00013200a000 CR4: 
> 001006e0
> [  439.203386] Call Trace:
> [  439.205869]  
> [  439.207886]  __xfrm_route_forward+0xa4/0x110
> [  439.212195]  ip_forward+0x3da/0x450
> [  439.215696]  ? ip_rcv_finish+0x61/0x390
> [  439.219542]  ip_rcv+0x2b5/0x380
> [  439.222716]  ? inet_del_offload+0x30/0x30
> [  439.226736]  __netif_receive_skb_core+0x751/0xb00
> [  439.231469]  ? netif_receive_skb_internal+0x47/0xf0
> [  439.236391]  netif_receive_skb_internal+0x47/0xf0
> [  439.241150]  napi_gro_flush+0x50/0x70
> [  439.244831]  napi_complete_done+0x90/0xd0
> [  439.248872]  igb_poll+0x8fd/0xe80
> [  439.252190]  net_rx_action+0x1fc/0x310
> [  439.255978]  __do_softirq+0xd5/0x1cf
> [  439.259584]  irq_exit+0xa3/0xb0
> [  439.262763]  do_IRQ+0x45/0xc0
> [  439.265772]  common_interrupt+0x95/0x95
> [  439.269609]  
> [  439.271733] RIP: 0010:cpuidle_enter_state+0x120/0x200
> [  439.276810] RSP: 0018:9492c0093eb8 EFLAGS: 0282 ORIG_RAX: 
> ff5d
> [  439.284436] RAX: 8cf33fd9ea80 RBX: 0002 RCX: 
> 00663c21ea0f
> [  439.291604] RDX:  RSI: 36ca RDI: 
> 
> [  439.298772] RBP: 8cf33fda71e8 R08: 0003 R09: 
> 0018
> [  439.305930] R10:  R11: 057c R12: 
> 00663c21ea0f
> [  439.313089] R13: 00663c1c6c33 R14: 0002 R15: 
> 
> [  439.320259]  ? cpuidle_enter_state+0x11c/0x200
> [  439.324740]  do_idle+0xd6/0x170
> [  439.327885]  cpu_startup_entry+0x67/0x70
> [  439.331837]  start_secondary+0x167/0x190
> [  439.335788]  secondary_startup_64+0xa5/0xb0
> [  439.340001] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 89 
> d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 <48> 
> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84 
> [  439.358988] RIP: xfrm_lookup+0x2a/0x7d0 RSP: 8cf33fd83bd0
> [  439.364759] CR2: 0020
> [  439.368105] ---[ end trace c6b298b556ea7769 ]---
> [  439.372752] Kernel panic - not syncing: Fatal exception in interrupt
> [  439.379255] Kernel Offset: 0x500 from 0x8100 (relocation 
> range: 0x8000-0xbfff)
> [  439.390029] Rebooting in 10 seconds..

...

> 4230 :
> 4230: 41 57   push   %r15
> 4232: 41 56   push   %r14
> 4234: 45 89 c6mov%r8d,%r14d
> 4237: 41 55   push   %r13
> 4239: 41 54   push   %r12
> 423b: 49 89 f5mov%rsi,%r13
> 423e: 55  push   %rbp
> 423f: 53  push   %rbx
> 4240: 49 89 d4mov%rdx,%r12
> 4243: 48 89 fbmov%rdi,%rbx
> 4246: 48 83 ec 40 sub$0x40,%rsp
> 424a: 65 48 8b 04 25 28 00mov%gs:0x28,%rax
> 4251: 00 00 
> 4253: 48 89 44 24 38  mov%rax,0x38(%rsp)
> 4258: 31 c0   xor%eax,%eax
> 425a: 48 8b 46 20 mov0x20(%rsi),%rax


The above is the failing instruction, RSI holds the second argument
of the called function which is a NULL pointer. The second argument
of xfrm_lookup() is dst_orig, so it is as I thought. Now let's find
out why. I don't see anything obvious, so we need to narrow it down.

> CONFIG_INET_ESP=y
> CONFIG_INET_ESP_OFFLOAD=y

You have CONFIG_INET_ESP_OFFLOAD enabled, this is new maybe it
still has some problems. You should not hit an offload codepath
because all your SAs are configured with UDP encapsulation which
is still not supported with 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Sat, Jan 06, 2018 at 12:27:11AM +0300, Ozgur wrote:
> 
> 
> 06.01.2018, 00:20, "Tobias Hommel" :
> > Hi,
> 
> Hi Tobias,
> 
> > I'm running into a NULL pointer dereference after updating from Linux 4.1.6 
> > to
> > 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> > either.
> > Anyone has an idea what is happening here?
> >
> > The affected machine has 2 active ethernet interfaces (igb driver) and acts 
> > as
> > a VPN gateway running strongswan. There are several hundreds of IPSec
> > roadwarriors connecting to eth1. eth0 connects to an infrastructure running 
> > an
> > HTTP server.
> > During my tests these roadwarriors connect to the gateway, sometimes 
> > download a
> > large file from the HTTP server, disconnect and after a random delay repeat
> > these steps.
> >
> > Some observations I made:
> > * SMP Affinity for IRQs of the NICs Rx/Tx queues 
> > (/proc/irq/$IRQ/smp_affinity)
> >   * all affinities set to default ff is broken
> >   * setting affinity for all queues of both interfaces to the same CPU 
> > seems to
> > work fine (running stable for more than 1 day now)
> >   * setting affinity of eth0 queues to CPU 1 and affinity of eth1 queues to 
> > CPU
> > 2 is broken and seems to always trigger the bug on CPU 1
> > * the top 6 entries of the call trace are the same every time the system
> >   crashes, the other entries differ sometimes
> >
> > The bug is 100% reproducible on the Intel Atom machine from the log below 
> > and
> > also on a HP ProLiant Gen6 (also igb driver).
> > I can, of course, provide further information (CPU, NIC, kernel config, more
> > traces, etc.) if required.
> > If helpful I could also run tests on HP ProLiant Gen9 which has different 
> > NICs
> > (tg3).
> >
> > [ 7998.489094] BUG: unable to handle kernel NULL pointer dereference at 
> > 0020
> > [ 7998.496993] IP: xfrm_lookup+0x2a/0x7e0
> > [ 7998.500759] PGD 0 P4D 0
> > [ 7998.503316] Oops:  [#1] SMP PTI
> > [ 7998.506835] Modules linked in:
> > [ 7998.509929] CPU: 2 PID: 22 Comm: ksoftirqd/2 Not tainted 4.14.11 #3
> > [ 7998.516244] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 
> > 1.01 07/11/2016
> > [ 7998.524039] task: 8826bb118000 task.stack: 947ac00f
> > [ 7998.530004] RIP: 0010:xfrm_lookup+0x2a/0x7e0
> > [ 7998.534298] RSP: 0018:947ac00f3b60 EFLAGS: 00010246
> > [ 7998.539550] RAX:  RBX: 93074040 RCX: 
> > 
> > [ 7998.546709] RDX: 947ac00f3bd8 RSI:  RDI: 
> > 93074040
> > [ 7998.553868] RBP: 93074040 R08: 0002 R09: 
> > 0001
> > [ 7998.561026] R10: 0032 R11:  R12: 
> > 947ac00f3bd8
> > [ 7998.568212] R13:  R14: 0002 R15: 
> > 8826b69a8078
> > [ 7998.575395] FS: () GS:8826bfc8() 
> > knlGS:
> > [ 7998.583550] CS: 0010 DS:  ES:  CR0: 80050033
> > [ 7998.589324] CR2: 0020 CR3: 0001781da000 CR4: 
> > 001006e0
> > [ 7998.596482] Call Trace:
> > [ 7998.598959] __xfrm_route_forward+0xa4/0x110
> > [ 7998.603263] ip_forward+0x3e0/0x450
> > [ 7998.606778] ? ip_rcv_finish+0x61/0x3a0
> > [ 7998.610645] ip_rcv+0x2c4/0x390
> > [ 7998.613818] ? inet_del_offload+0x30/0x30
> > [ 7998.617857] __netif_receive_skb_core+0x751/0xb00
> > [ 7998.622562] ? skb_send_sock+0x40/0x40
> > [ 7998.626356] ? netif_receive_skb_internal+0x47/0xf0
> > [ 7998.631252] netif_receive_skb_internal+0x47/0xf0
> > [ 7998.635987] napi_gro_receive+0x70/0x90
> > [ 7998.639835] gro_cell_poll+0x53/0x90
> > [ 7998.643439] net_rx_action+0x1fc/0x310
> > [ 7998.647210] ? rebalance_domains+0x101/0x2b0
> > [ 7998.651500] __do_softirq+0xd5/0x1cf
> > [ 7998.655105] run_ksoftirqd+0x14/0x30
> > [ 7998.658712] smpboot_thread_fn+0xf9/0x150
> > [ 7998.662723] kthread+0xef/0x130
> > [ 7998.665893] ? sort_range+0x20/0x20
> > [ 7998.669404] ? kthread_park+0x60/0x60
> > [ 7998.673098] ret_from_fork+0x1f/0x30
> > [ 7998.676674] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 
> > 89 d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 
> > <48> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84
> > [ 7998.695681] RIP: xfrm_lookup+0x2a/0x7e0 RSP: 947ac00f3b60
> > [ 7998.701479] CR2: 0020
> > [ 7998.704799] ---[ end trace 0544b1946919baad ]---
> > [ 7998.709442] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 7998.715918] Kernel Offset: 0x1100 from 0x8100 
> > (relocation range: 0x8000-0xbfff)
> 
> 
> this error doesn't look like the last version kernel, I think this problem 
> NIC driver.

Can you please explain why you think that this is a driver problem?

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Steffen Klassert]

On Fri, Jan 05, 2018 at 10:13:23PM +0100, Tobias Hommel wrote:
> Hi,
> 
> I'm running into a NULL pointer dereference after updating from Linux 4.1.6 to
> 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> either.
> Anyone has an idea what is happening here?
> 
> The affected machine has 2 active ethernet interfaces (igb driver) and acts as
> a VPN gateway running strongswan. There are several hundreds of IPSec
> roadwarriors connecting to eth1. eth0 connects to an infrastructure running an
> HTTP server.
> During my tests these roadwarriors connect to the gateway, sometimes download 
> a
> large file from the HTTP server, disconnect and after a random delay repeat
> these steps.
> 
> Some observations I made:
> * SMP Affinity for IRQs of the NICs Rx/Tx queues (/proc/irq/$IRQ/smp_affinity)
>   * all affinities set to default ff is broken
>   * setting affinity for all queues of both interfaces to the same CPU seems 
> to
> work fine (running stable for more than 1 day now)
>   * setting affinity of eth0 queues to CPU 1 and affinity of eth1 queues to 
> CPU
> 2 is broken and seems to always trigger the bug on CPU 1
> * the top 6 entries of the call trace are the same every time the system
>   crashes, the other entries differ sometimes
> 
> The bug is 100% reproducible on the Intel Atom machine from the log below and
> also on a HP ProLiant Gen6 (also igb driver).
> I can, of course, provide further information (CPU, NIC, kernel config, more
> traces, etc.) if required.
> If helpful I could also run tests on HP ProLiant Gen9 which has different NICs
> (tg3).
> 
> [ 7998.489094] BUG: unable to handle kernel NULL pointer dereference at 
> 0020
> [ 7998.496993] IP: xfrm_lookup+0x2a/0x7e0

xfrm_lookup+0x2a is at the very beginning of xfrm_lookup(), here we
find:

u16 family = dst_orig->ops->family;

ops has an offset of 32 bytes (20 hex) in dst_orig, so looks like
dst_orig is NULL.

In the forwarding case, we get dst_orig from the skb and dst_orig
can't be NULL here unless the skb itself is already fishy.

Can you provide the following informations:

- Your kernel config

- The output of 'ip x p' and 'ip x s'

- An object dump of xfrm_policy.o if possible 'objdump -d -S 
net/xfrm/xfrm_policy.o'
  (The path to xfrm_policy.o depends on how you build your kernels)

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Fri, Jan 05, 2018 at 09:55:23PM +, Tobias Hommel wrote:
> On Sat, Jan 06, 2018 at 12:27:11AM +0300, Ozgur wrote:
> > 
> > 
> > 06.01.2018, 00:20, "Tobias Hommel" :
> > > Hi,
> > 
> > Hi Tobias,
> > 
> > > I'm running into a NULL pointer dereference after updating from Linux 
> > > 4.1.6 to
> > > 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not 
> > > work
> > > either.
> > > Anyone has an idea what is happening here?
> > >
> > > The affected machine has 2 active ethernet interfaces (igb driver) and 
> > > acts as
> > > a VPN gateway running strongswan. There are several hundreds of IPSec
> > > roadwarriors connecting to eth1. eth0 connects to an infrastructure 
> > > running an
> > > HTTP server.
> > > During my tests these roadwarriors connect to the gateway, sometimes 
> > > download a
> > > large file from the HTTP server, disconnect and after a random delay 
> > > repeat
> > > these steps.
> > >
> > > Some observations I made:
> > > * SMP Affinity for IRQs of the NICs Rx/Tx queues 
> > > (/proc/irq/$IRQ/smp_affinity)
> > >   * all affinities set to default ff is broken
> > >   * setting affinity for all queues of both interfaces to the same CPU 
> > > seems to
> > > work fine (running stable for more than 1 day now)
> > >   * setting affinity of eth0 queues to CPU 1 and affinity of eth1 queues 
> > > to CPU
> > > 2 is broken and seems to always trigger the bug on CPU 1
> > > * the top 6 entries of the call trace are the same every time the system
> > >   crashes, the other entries differ sometimes
> > >
> > > The bug is 100% reproducible on the Intel Atom machine from the log below 
> > > and
> > > also on a HP ProLiant Gen6 (also igb driver).
> > > I can, of course, provide further information (CPU, NIC, kernel config, 
> > > more
> > > traces, etc.) if required.
> > > If helpful I could also run tests on HP ProLiant Gen9 which has different 
> > > NICs
> > > (tg3).
> > >
> > > [ 7998.489094] BUG: unable to handle kernel NULL pointer dereference at 
> > > 0020
> > > [ 7998.496993] IP: xfrm_lookup+0x2a/0x7e0
> > > [ 7998.500759] PGD 0 P4D 0
> > > [ 7998.503316] Oops:  [#1] SMP PTI
> > > [ 7998.506835] Modules linked in:
> > > [ 7998.509929] CPU: 2 PID: 22 Comm: ksoftirqd/2 Not tainted 4.14.11 #3
> > > [ 7998.516244] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 
> > > 1.01 07/11/2016
> > > [ 7998.524039] task: 8826bb118000 task.stack: 947ac00f
> > > [ 7998.530004] RIP: 0010:xfrm_lookup+0x2a/0x7e0
> > > [ 7998.534298] RSP: 0018:947ac00f3b60 EFLAGS: 00010246
> > > [ 7998.539550] RAX:  RBX: 93074040 RCX: 
> > > 
> > > [ 7998.546709] RDX: 947ac00f3bd8 RSI:  RDI: 
> > > 93074040
> > > [ 7998.553868] RBP: 93074040 R08: 0002 R09: 
> > > 0001
> > > [ 7998.561026] R10: 0032 R11:  R12: 
> > > 947ac00f3bd8
> > > [ 7998.568212] R13:  R14: 0002 R15: 
> > > 8826b69a8078
> > > [ 7998.575395] FS: () GS:8826bfc8() 
> > > knlGS:
> > > [ 7998.583550] CS: 0010 DS:  ES:  CR0: 80050033
> > > [ 7998.589324] CR2: 0020 CR3: 0001781da000 CR4: 
> > > 001006e0
> > > [ 7998.596482] Call Trace:
> > > [ 7998.598959] __xfrm_route_forward+0xa4/0x110
> > > [ 7998.603263] ip_forward+0x3e0/0x450
> > > [ 7998.606778] ? ip_rcv_finish+0x61/0x3a0
> > > [ 7998.610645] ip_rcv+0x2c4/0x390
> > > [ 7998.613818] ? inet_del_offload+0x30/0x30
> > > [ 7998.617857] __netif_receive_skb_core+0x751/0xb00
> > > [ 7998.622562] ? skb_send_sock+0x40/0x40
> > > [ 7998.626356] ? netif_receive_skb_internal+0x47/0xf0
> > > [ 7998.631252] netif_receive_skb_internal+0x47/0xf0
> > > [ 7998.635987] napi_gro_receive+0x70/0x90
> > > [ 7998.639835] gro_cell_poll+0x53/0x90
> > > [ 7998.643439] net_rx_action+0x1fc/0x310
> > > [ 7998.647210] ? rebalance_domains+0x101/0x2b0
> > > [ 7998.651500] __do_softirq+0xd5/0x1cf
> > > [ 7998.655105] run_ksoftirqd+0x14/0x30
> > > [ 7998.658712] smpboot_thread_fn+0xf9/0x150
> > > [ 7998.662723] kthread+0xef/0x130
> > > [ 7998.665893] ? sort_range+0x20/0x20
> > > [ 7998.669404] ? kthread_park+0x60/0x60
> > > [ 7998.673098] ret_from_fork+0x1f/0x30
> > > [ 7998.676674] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 
> > > 49 89 d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 
> > > 31 c0 <48> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84
> > > [ 7998.695681] RIP: xfrm_lookup+0x2a/0x7e0 RSP: 947ac00f3b60
> > > [ 7998.701479] CR2: 0020
> > > [ 7998.704799] ---[ end trace 0544b1946919baad ]---
> > > [ 7998.709442] Kernel panic - not syncing: Fatal exception in interrupt
> > > [ 7998.715918] Kernel Offset: 0x1100 from 0x8100 
> > > (relocation range: 0x8000-0xbfff)
> > 
> > 
> > this error 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Fri, Jan 05, 2018 at 09:51:16PM +, Holger Hoffstätte wrote:
> On Fri, 05 Jan 2018 22:13:23 +0100, Tobias Hommel wrote:
> 
> > Hi,
> > 
> > I'm running into a NULL pointer dereference after updating from Linux 4.1.6 
> > to
> > 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> > either.
> > Anyone has an idea what is happening here?
> 
> Try 4.14.12 because of:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.14.y=2d01ac8cc12b973668bf898b03bf9ffb12d83b83

Using tunnel mode here, not transport mode. Anyway, I tried it, the same
problem:
[  275.655170] BUG: unable to handle kernel NULL pointer dereference at 
0020
[  275.663230] IP: xfrm_lookup+0x2a/0x7d0
[  275.666986] PGD 0 P4D 0 
[  275.669579] Oops:  [#1] SMP PTI
[  275.673097] Modules linked in:
[  275.676182] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.14.12 #1
[  275.682215] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 1.01 
07/11/2016
[  275.690013] task: 9b43fb0ed080 task.stack: b0af0009
[  275.695960] RIP: 0010:xfrm_lookup+0x2a/0x7d0
[  275.700256] RSP: 0018:9b43ffd83bd0 EFLAGS: 00010246
[  275.705528] RAX:  RBX: 8e074080 RCX: 
[  275.712710] RDX: 9b43ffd83c48 RSI:  RDI: 8e074080
[  275.719895] RBP: 8e074080 R08: 0002 R09: 
[  275.727071] R10: 0020 R11: 0020 R12: 9b43ffd83c48
[  275.734248] R13:  R14: 0002 R15: 9b43fb240078
[  275.741415] FS:  () GS:9b43ffd8() 
knlGS:
[  275.749527] CS:  0010 DS:  ES:  CR0: 80050033
[  275.755307] CR2: 0020 CR3: 00013e00a000 CR4: 001006e0
[  275.762474] Call Trace:
[  275.764939]   
[  275.766986]  __xfrm_route_forward+0xa4/0x110
[  275.771282]  ip_forward+0x3da/0x450
[  275.774803]  ? ip_rcv_finish+0x61/0x390
[  275.778666]  ip_rcv+0x2b5/0x380
[  275.781840]  ? inet_del_offload+0x30/0x30
[  275.785860]  __netif_receive_skb_core+0x751/0xb00
[  275.790593]  ? tcp_gro_receive+0x24d/0x310
[  275.794716]  ? netif_receive_skb_internal+0x47/0xf0
[  275.799620]  netif_receive_skb_internal+0x47/0xf0
[  275.804381]  napi_gro_flush+0x50/0x70
[  275.808071]  napi_complete_done+0x90/0xd0
[  275.812111]  igb_poll+0x8fd/0xe80
[  275.815458]  net_rx_action+0x1fc/0x310
[  275.819227]  __do_softirq+0xd5/0x1cf
[  275.822834]  irq_exit+0xa3/0xb0
[  275.826003]  do_IRQ+0x45/0xc0
[  275.829004]  common_interrupt+0x95/0x95
[  275.832868]  
[  275.835002] RIP: 0010:cpuidle_enter_state+0x120/0x200
[  275.840076] RSP: 0018:b0af00093eb8 EFLAGS: 0282 ORIG_RAX: 
ff7d
[  275.847685] RAX: 9b43ffd9ea80 RBX: 0002 RCX: 00402e53956c
[  275.854844] RDX:  RSI: 36ca RDI: 
[  275.862039] RBP: 9b43ffda71e8 R08: 0003 R09: 0018
[  275.869222] R10:  R11: 0102 R12: 00402e53956c
[  275.876398] R13: 00402e4e8838 R14: 0002 R15: 
[  275.883568]  ? cpuidle_enter_state+0x11c/0x200
[  275.888023]  do_idle+0xd6/0x170
[  275.891177]  cpu_startup_entry+0x67/0x70
[  275.895129]  start_secondary+0x167/0x190
[  275.899080]  secondary_startup_64+0xa5/0xb0
[  275.903291] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 89 
d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 <48> 8b 
46 20 48 85 c9 44 0f b7 38 c7 44 
[  275.922273] RIP: xfrm_lookup+0x2a/0x7d0 RSP: 9b43ffd83bd0
[  275.928070] CR2: 0020
[  275.931417] ---[ end trace 453df6e200be3ed0 ]---
[  275.936061] Kernel panic - not syncing: Fatal exception in interrupt
[  275.942566] Kernel Offset: 0xc00 from 0x8100 (relocation 
range: 0x8000-0xbfff)
[  275.953309] Rebooting in 10 seconds..

> 
> -h
> 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Holger Hoffstätte]

On Fri, 05 Jan 2018 22:13:23 +0100, Tobias Hommel wrote:

> Hi,
> 
> I'm running into a NULL pointer dereference after updating from Linux 4.1.6 to
> 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> either.
> Anyone has an idea what is happening here?

Try 4.14.12 because of:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.14.y=2d01ac8cc12b973668bf898b03bf9ffb12d83b83

-h

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel]

On Sat, Jan 06, 2018 at 12:27:11AM +0300, Ozgur wrote:
> 
> 
> 06.01.2018, 00:20, "Tobias Hommel" :
> > Hi,
> 
> Hi Tobias,
> 
> > I'm running into a NULL pointer dereference after updating from Linux 4.1.6 
> > to
> > 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> > either.
> > Anyone has an idea what is happening here?
> >
> > The affected machine has 2 active ethernet interfaces (igb driver) and acts 
> > as
> > a VPN gateway running strongswan. There are several hundreds of IPSec
> > roadwarriors connecting to eth1. eth0 connects to an infrastructure running 
> > an
> > HTTP server.
> > During my tests these roadwarriors connect to the gateway, sometimes 
> > download a
> > large file from the HTTP server, disconnect and after a random delay repeat
> > these steps.
> >
> > Some observations I made:
> > * SMP Affinity for IRQs of the NICs Rx/Tx queues 
> > (/proc/irq/$IRQ/smp_affinity)
> >   * all affinities set to default ff is broken
> >   * setting affinity for all queues of both interfaces to the same CPU 
> > seems to
> > work fine (running stable for more than 1 day now)
> >   * setting affinity of eth0 queues to CPU 1 and affinity of eth1 queues to 
> > CPU
> > 2 is broken and seems to always trigger the bug on CPU 1
> > * the top 6 entries of the call trace are the same every time the system
> >   crashes, the other entries differ sometimes
> >
> > The bug is 100% reproducible on the Intel Atom machine from the log below 
> > and
> > also on a HP ProLiant Gen6 (also igb driver).
> > I can, of course, provide further information (CPU, NIC, kernel config, more
> > traces, etc.) if required.
> > If helpful I could also run tests on HP ProLiant Gen9 which has different 
> > NICs
> > (tg3).
> >
> > [ 7998.489094] BUG: unable to handle kernel NULL pointer dereference at 
> > 0020
> > [ 7998.496993] IP: xfrm_lookup+0x2a/0x7e0
> > [ 7998.500759] PGD 0 P4D 0
> > [ 7998.503316] Oops:  [#1] SMP PTI
> > [ 7998.506835] Modules linked in:
> > [ 7998.509929] CPU: 2 PID: 22 Comm: ksoftirqd/2 Not tainted 4.14.11 #3
> > [ 7998.516244] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 
> > 1.01 07/11/2016
> > [ 7998.524039] task: 8826bb118000 task.stack: 947ac00f
> > [ 7998.530004] RIP: 0010:xfrm_lookup+0x2a/0x7e0
> > [ 7998.534298] RSP: 0018:947ac00f3b60 EFLAGS: 00010246
> > [ 7998.539550] RAX:  RBX: 93074040 RCX: 
> > 
> > [ 7998.546709] RDX: 947ac00f3bd8 RSI:  RDI: 
> > 93074040
> > [ 7998.553868] RBP: 93074040 R08: 0002 R09: 
> > 0001
> > [ 7998.561026] R10: 0032 R11:  R12: 
> > 947ac00f3bd8
> > [ 7998.568212] R13:  R14: 0002 R15: 
> > 8826b69a8078
> > [ 7998.575395] FS: () GS:8826bfc8() 
> > knlGS:
> > [ 7998.583550] CS: 0010 DS:  ES:  CR0: 80050033
> > [ 7998.589324] CR2: 0020 CR3: 0001781da000 CR4: 
> > 001006e0
> > [ 7998.596482] Call Trace:
> > [ 7998.598959] __xfrm_route_forward+0xa4/0x110
> > [ 7998.603263] ip_forward+0x3e0/0x450
> > [ 7998.606778] ? ip_rcv_finish+0x61/0x3a0
> > [ 7998.610645] ip_rcv+0x2c4/0x390
> > [ 7998.613818] ? inet_del_offload+0x30/0x30
> > [ 7998.617857] __netif_receive_skb_core+0x751/0xb00
> > [ 7998.622562] ? skb_send_sock+0x40/0x40
> > [ 7998.626356] ? netif_receive_skb_internal+0x47/0xf0
> > [ 7998.631252] netif_receive_skb_internal+0x47/0xf0
> > [ 7998.635987] napi_gro_receive+0x70/0x90
> > [ 7998.639835] gro_cell_poll+0x53/0x90
> > [ 7998.643439] net_rx_action+0x1fc/0x310
> > [ 7998.647210] ? rebalance_domains+0x101/0x2b0
> > [ 7998.651500] __do_softirq+0xd5/0x1cf
> > [ 7998.655105] run_ksoftirqd+0x14/0x30
> > [ 7998.658712] smpboot_thread_fn+0xf9/0x150
> > [ 7998.662723] kthread+0xef/0x130
> > [ 7998.665893] ? sort_range+0x20/0x20
> > [ 7998.669404] ? kthread_park+0x60/0x60
> > [ 7998.673098] ret_from_fork+0x1f/0x30
> > [ 7998.676674] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 
> > 89 d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 
> > <48> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84
> > [ 7998.695681] RIP: xfrm_lookup+0x2a/0x7e0 RSP: 947ac00f3b60
> > [ 7998.701479] CR2: 0020
> > [ 7998.704799] ---[ end trace 0544b1946919baad ]---
> > [ 7998.709442] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 7998.715918] Kernel Offset: 0x1100 from 0x8100 
> > (relocation range: 0x8000-0xbfff)
> 
> 
> this error doesn't look like the last version kernel, I think this problem 
> NIC driver.
> What is the use network ethernet card model?
This is what lspci shows for both NICs:
# lspci -nns 00:14.0
00:14.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I354 
[8086:1f41] (rev 03)

I have 

Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Ozgur]

06.01.2018, 00:20, "Tobias Hommel" :
> Hi,

Hi Tobias,

> I'm running into a NULL pointer dereference after updating from Linux 4.1.6 to
> 4.14.11 (see kernel log below). I tried 4.14.3 initially which did not work
> either.
> Anyone has an idea what is happening here?
>
> The affected machine has 2 active ethernet interfaces (igb driver) and acts as
> a VPN gateway running strongswan. There are several hundreds of IPSec
> roadwarriors connecting to eth1. eth0 connects to an infrastructure running an
> HTTP server.
> During my tests these roadwarriors connect to the gateway, sometimes download 
> a
> large file from the HTTP server, disconnect and after a random delay repeat
> these steps.
>
> Some observations I made:
> * SMP Affinity for IRQs of the NICs Rx/Tx queues (/proc/irq/$IRQ/smp_affinity)
>   * all affinities set to default ff is broken
>   * setting affinity for all queues of both interfaces to the same CPU seems 
> to
> work fine (running stable for more than 1 day now)
>   * setting affinity of eth0 queues to CPU 1 and affinity of eth1 queues to 
> CPU
> 2 is broken and seems to always trigger the bug on CPU 1
> * the top 6 entries of the call trace are the same every time the system
>   crashes, the other entries differ sometimes
>
> The bug is 100% reproducible on the Intel Atom machine from the log below and
> also on a HP ProLiant Gen6 (also igb driver).
> I can, of course, provide further information (CPU, NIC, kernel config, more
> traces, etc.) if required.
> If helpful I could also run tests on HP ProLiant Gen9 which has different NICs
> (tg3).
>
> [ 7998.489094] BUG: unable to handle kernel NULL pointer dereference at 
> 0020
> [ 7998.496993] IP: xfrm_lookup+0x2a/0x7e0
> [ 7998.500759] PGD 0 P4D 0
> [ 7998.503316] Oops:  [#1] SMP PTI
> [ 7998.506835] Modules linked in:
> [ 7998.509929] CPU: 2 PID: 22 Comm: ksoftirqd/2 Not tainted 4.14.11 #3
> [ 7998.516244] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 1.01 
> 07/11/2016
> [ 7998.524039] task: 8826bb118000 task.stack: 947ac00f
> [ 7998.530004] RIP: 0010:xfrm_lookup+0x2a/0x7e0
> [ 7998.534298] RSP: 0018:947ac00f3b60 EFLAGS: 00010246
> [ 7998.539550] RAX:  RBX: 93074040 RCX: 
> 
> [ 7998.546709] RDX: 947ac00f3bd8 RSI:  RDI: 
> 93074040
> [ 7998.553868] RBP: 93074040 R08: 0002 R09: 
> 0001
> [ 7998.561026] R10: 0032 R11:  R12: 
> 947ac00f3bd8
> [ 7998.568212] R13:  R14: 0002 R15: 
> 8826b69a8078
> [ 7998.575395] FS: () GS:8826bfc8() 
> knlGS:
> [ 7998.583550] CS: 0010 DS:  ES:  CR0: 80050033
> [ 7998.589324] CR2: 0020 CR3: 0001781da000 CR4: 
> 001006e0
> [ 7998.596482] Call Trace:
> [ 7998.598959] __xfrm_route_forward+0xa4/0x110
> [ 7998.603263] ip_forward+0x3e0/0x450
> [ 7998.606778] ? ip_rcv_finish+0x61/0x3a0
> [ 7998.610645] ip_rcv+0x2c4/0x390
> [ 7998.613818] ? inet_del_offload+0x30/0x30
> [ 7998.617857] __netif_receive_skb_core+0x751/0xb00
> [ 7998.622562] ? skb_send_sock+0x40/0x40
> [ 7998.626356] ? netif_receive_skb_internal+0x47/0xf0
> [ 7998.631252] netif_receive_skb_internal+0x47/0xf0
> [ 7998.635987] napi_gro_receive+0x70/0x90
> [ 7998.639835] gro_cell_poll+0x53/0x90
> [ 7998.643439] net_rx_action+0x1fc/0x310
> [ 7998.647210] ? rebalance_domains+0x101/0x2b0
> [ 7998.651500] __do_softirq+0xd5/0x1cf
> [ 7998.655105] run_ksoftirqd+0x14/0x30
> [ 7998.658712] smpboot_thread_fn+0xf9/0x150
> [ 7998.662723] kthread+0xef/0x130
> [ 7998.665893] ? sort_range+0x20/0x20
> [ 7998.669404] ? kthread_park+0x60/0x60
> [ 7998.673098] ret_from_fork+0x1f/0x30
> [ 7998.676674] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 89 
> d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 <48> 
> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84
> [ 7998.695681] RIP: xfrm_lookup+0x2a/0x7e0 RSP: 947ac00f3b60
> [ 7998.701479] CR2: 0020
> [ 7998.704799] ---[ end trace 0544b1946919baad ]---
> [ 7998.709442] Kernel panic - not syncing: Fatal exception in interrupt
> [ 7998.715918] Kernel Offset: 0x1100 from 0x8100 (relocation 
> range: 0x8000-0xbfff)


this error doesn't look like the last version kernel, I think this problem NIC 
driver.
What is the use network ethernet card model?
And which driver version you use?

> Best regards,
>
> Tobias Hommel

Ozgur