Package: linux-image-3.16.0-4-amd64 Version: 3.16.43-2 Jessie kernels, e.g. linux-image-3.16.0-4-amd64, (also applicable to Wheezy) when used on Xen guests that require a large number of grant table references (many vCPUs and/or vNICs in multi-queue xennet) leads to a boot-time kernel BUG assertion and panic due to a type casting error. On a 2-socket E5-2698v3 Xen 4.4 hypervisor (gnttab_max_nr_frames=256) and a Debian 8 guest with (40) vCPUs, 128GB RAM, (3) xvd block devices, (2) xennet vNICs, and xennet in the default all CPUs multi-queue mode in both dom0 and domU, the console output looks like this:
[ 1.317129] zswap: loaded using pool lzo/zbud [ 1.317259] xenbus_probe_frontend: Device with no driver: device/vbd/51713 [ 1.317265] xenbus_probe_frontend: Device with no driver: device/vbd/51714 [ 1.317269] xenbus_probe_frontend: Device with no driver: device/vbd/51715 [ 1.317273] xenbus_probe_frontend: Device with no driver: device/vif/0 [ 1.317277] xenbus_probe_frontend: Device with no driver: device/vif/1 [ 1.317479] hctosys: unable to open rtc device (rtc0) [ 1.318306] Freeing unused kernel memory: 1316K (ffffffff81b1f000 - ffffffff81c68000) [ 1.318315] Write protecting the kernel read-only data: 10240k [ 1.321142] Freeing unused kernel memory: 392K (ffff88000159e000 - ffff880001600000) [ 1.321862] Freeing unused kernel memory: 1200K (ffff8800018d4000 - ffff880001a00000) Loading, please wait... [ 1.361904] systemd-udevd[260]: starting version 215 [ 1.362768] random: systemd-udevd: uninitialized urandom read (16 bytes read, 121 bits of entropy available) [ 1.395602] xen_netfront: Initialising Xen virtual ethernet driver [ 1.400679] random: nonblocking pool is initialized [ 1.604799] blkfront: xvda1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled; [ 1.722225] ------------[ cut here ]------------ [ 1.722242] kernel BUG at /build/kernel/orig/linux-4.4.43/drivers/net/xen-netfront.c:307! [ 1.722255] invalid opcode: 0000 [#1] SMP [ 1.722265] Modules linked in: xen_netfront(+) xen_blkfront(+) crc32c_intel [ 1.722282] CPU: 8 PID: 209 Comm: xenwatch Not tainted 4.4.0-1-amd64 #1 Debian 4.4.43-1 [ 1.722298] task: ffff881f51c8b240 ti: ffff881f51c8c000 task.ti: ffff881f51c8c000 [ 1.722310] RIP: e030:[<ffffffffa00732e5>] [<ffffffffa00732e5>] xennet_alloc_rx_buffers+0x215/0x2b0 [xen_netfront] [ 1.722333] RSP: e02b:ffff881f51c8fdb8 EFLAGS: 00010286 [ 1.722341] RAX: 0000000000008000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1.722351] RDX: 0000000000008000 RSI: ffff881f51c6f400 RDI: ffff881f498b2bf8 [ 1.722360] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000001000 [ 1.722369] R10: ffff881f47c64540 R11: ffff881f47c64540 R12: 0000000000000000 [ 1.722378] R13: 0000000000008000 R14: ffff881f498b0e00 R15: ffff881f47c68600 [ 1.722394] FS: 0000000000000000(0000) GS:ffff881f5f500000(0000) knlGS:ffff881f5f500000 [ 1.722409] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.722417] CR2: 00007f6eb6205095 CR3: 0000000001a0b000 CR4: 0000000000042660 [ 1.722427] Stack: [ 1.722434] ffffffff810bcab1 ffff881f498b2bf8 ffff881f51a8ee90 ffff881f52392800 [ 1.722449] ffff881f4dbac000 ffff881f498b0e00 0000000000033000 ffff881f498b2380 [ 1.722466] 0000000000055000 ffffffffa0075528 ffff881f00000028 ffff881f52392800 [ 1.722484] Call Trace: [ 1.722503] [<ffffffff810bcab1>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [ 1.722531] [<ffffffffa0075528>] ? netback_changed+0xcd8/0xd67 [xen_netfront] [ 1.722557] [<ffffffff813b5ae0>] ? split+0xf0/0xf0 [ 1.722569] [<ffffffff813b5b6a>] ? xenwatch_thread+0x8a/0x140 [ 1.722581] [<ffffffff810b7a30>] ? wait_woken+0x90/0x90 [ 1.722596] [<ffffffff810962ff>] ? kthread+0xdf/0x100 [ 1.722608] [<ffffffff81096220>] ? kthread_park+0x50/0x50 [ 1.722623] [<ffffffff8159721f>] ? ret_from_fork+0x3f/0x70 [ 1.722633] [<ffffffff81096220>] ? kthread_park+0x50/0x50 [ 1.722641] Code: 8b 05 90 b4 a9 e1 48 8b 04 f8 48 83 f8 ff 0f 84 98 00 00 00 48 89 c2 48 b8 ff ff ff ff ff ff ff 3f 48 21 c2 e9 26 ff ff ff 0f 0b <0f> 0b 48 b8 00 00 00 00 00 00 00 40 48 09 c2 48 3b 3d 45 b4 a9 [ 1.722745] RIP [<ffffffffa00732e5>] xennet_alloc_rx_buffers+0x215/0x2b0 [xen_netfront] [ 1.722762] RSP <ffff881f51c8fdb8> [ 1.722772] ---[ end trace a266a8dd13d8465b ]--- [ 1.722780] Kernel panic - not syncing: Fatal exception in interrupt [ 1.722909] Kernel Offset: disabled The offending xen_netfront code is: static void xennet_alloc_rx_buffers(struct netfront_queue *queue) { ... ref = gnttab_claim_grant_reference(&queue->gref_rx_head); BUG_ON((signed short)ref < 0); queue->grant_rx_ref[id] = ref; ... } Each vNIC requires 514 grant refs per tx+rx queue pair / CPU, 20560 refs for each vNIC on the guest. This mis-casting has been fixed in later Linux 4.x kernels: https://github.com/torvalds/linux/commit/87557efc27f6a50140fb20df06a917f368ce3c66 https://github.com/torvalds/linux/commit/269ebce4531b8edc4224259a02143181a1c1d77c