On Friday, August 10, 2018 at 9:02:31 PM UTC+2, Kelly Dean wrote:
> Has anybody else used both Qubes 3.2 and 4.0 on a system with a HD, not SSD?
> Have you noticed the disk thrashing to be far worse under 4.0? I suspect it
> might have something to do with the new use of LVM combining snapshots with
> thin provisioning.
>
> The problem seems to be triggered by individual qubes doing ordinary bursts
> of disk access, such as loading a program or accessing swap, which would
> normally take just a few seconds on Qubes 3.2, but dom0 then massively
> multiplies that I/O on Qubes 4.0, leading to disk thrashing that drags on for
> minutes at a time, and in some cases, more than an hour.
>
> iotop in dom0 says the thrashing procs are e.g. [21.xvda-0] and [21.xvda-1],
> reading the disk at rates ranging from 10 to 50 MBps (max throughput of the
> disk is about 100). At this rate, for how prolonged the thrashing is, it
> could have read and re-read the entire virtual disk multiple times over, so
> there's something extremely inefficient going on.
>
> Is there any solution other than installing a SSD? I'd prefer not to have to
> add hardware to solve a software performance regression.
Interestingly, I've just encountered this thrashing, but on SSD(it's just
reading 192MiB/sec constantly), Qubes R4.0 up to date, inside a qube while
compiling firefox: typing in any of 3 of its terminal windows does not even
echo anything and the firefox compilation terminal is frozen; the swap (of 1G)
was turned off a while ago (via swapoff); I used Qube Manager to Pause the
offending cube and the thrashing stopped. I don't see much on logs.
Ok so I resumed the qube, the thrashing resumed for a few seconds then stopped
and all terminals were alive again (I can type into them). The log spewed some
new things (since the updatedb audit which was last while Paused), I'm
including some long lines from before, note that the log after the unpause
starts from "[ 6862.846945] INFO: rcu_sched self-detected stall on CPU", as
follows:
[ 0.000000] Linux version 4.14.57-1.pvops.qubes.x86_64 (user@build-fedora4)
(gcc version 6.4.1 20170727 (Red Hat 6.4.1-1) (GCC)) #1 SMP Mon Jul 23 16:28:54
UTC 2018
[ 0.000000] Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0
rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 nopat
...
[ 2769.581919] audit: type=1101 audit(1534434741.005:133): pid=10290 uid=1000
auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2769.582396] audit: type=1123 audit(1534434741.005:134): pid=10290 uid=1000
auid=1000 ses=1 msg='cwd="/home/user" cmd=737761706F6666202F6465762F7876646331
terminal=pts/3 res=success'
[ 2769.582525] audit: type=1110 audit(1534434741.006:135): pid=10290 uid=0
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2769.583384] audit: type=1105 audit(1534434741.007:136): pid=10290 uid=0
auid=1000 ses=1 msg='op=PAM:session_open
grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3
res=success'
[ 2776.388700] audit: type=1106 audit(1534434747.812:137): pid=10290 uid=0
auid=1000 ses=1 msg='op=PAM:session_close
grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3
res=success'
[ 2776.388735] audit: type=1104 audit(1534434747.812:138): pid=10290 uid=0
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 4093.008056] audit: type=1116 audit(1534436064.432:139): pid=29167 uid=0
auid=4294967295 ses=4294967295 msg='op=add-group id=982
exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4093.030620] audit: type=1132 audit(1534436064.454:140): pid=29167 uid=0
auid=4294967295 ses=4294967295 msg='op=add-shadow-group id=982
exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4093.304708] audit: type=1130 audit(1534436064.728:141): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=run-rfbdacad57c5f4bc183d36a7c402c9ae7
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 4094.576065] audit: type=1130 audit(1534436065.999:142): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4094.576138] audit: type=1131 audit(1534436065.999:143): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4094.577822] audit: type=1131 audit(1534436066.001:144): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=run-rfbdacad57c5f4bc183d36a7c402c9ae7
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 4219.569289] audit: type=1100 audit(1534436190.993:145): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.569317] audit: type=1101 audit(1534436190.993:146): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_succeed_if
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.569499] audit: type=1103 audit(1534436190.993:147): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.575980] audit: type=1105 audit(1534436190.999:148): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:session_open
grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.596837] audit: type=1106 audit(1534436191.020:149): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:session_close
grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.596882] audit: type=1104 audit(1534436191.020:150): pid=30881 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528350] audit: type=1100 audit(1534436507.950:151): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528387] audit: type=1101 audit(1534436507.950:152): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_succeed_if
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528426] audit: type=1103 audit(1534436507.951:153): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.532307] audit: type=1105 audit(1534436507.956:154): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:session_open
grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.555390] audit: type=1106 audit(1534436507.979:155): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:session_close
grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.555432] audit: type=1104 audit(1534436507.979:156): pid=580 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok
acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4562.928666] audit: type=1116 audit(1534436534.352:157): pid=1009 uid=0
auid=4294967295 ses=4294967295 msg='op=add-group id=21 exe="/usr/sbin/groupadd"
hostname=? addr=? terminal=? res=success'
[ 4562.939364] audit: type=1132 audit(1534436534.363:158): pid=1009 uid=0
auid=4294967295 ses=4294967295 msg='op=add-shadow-group id=21
exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4563.070979] audit: type=1130 audit(1534436534.494:159): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 4563.675418] audit: type=1130 audit(1534436535.099:160): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.675458] audit: type=1131 audit(1534436535.099:161): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.676716] audit: type=1130 audit(1534436535.100:162): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 4563.676754] audit: type=1131 audit(1534436535.100:163): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 4570.160514] audit: type=1101 audit(1534436541.584:164): pid=1473 uid=1000
auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4570.162254] audit: type=1123 audit(1534436541.584:165): pid=1473 uid=1000
auid=1000 ses=1 msg='cwd="/home/user" cmd="updatedb" terminal=pts/5 res=success'
[ 4570.162328] audit: type=1110 audit(1534436541.585:166): pid=1473 uid=0
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4570.165188] audit: type=1105 audit(1534436541.588:167): pid=1473 uid=0
auid=1000 ses=1 msg='op=PAM:session_open
grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5
res=success'
[ 4580.680238] audit: type=1106 audit(1534436552.103:168): pid=1473 uid=0
auid=1000 ses=1 msg='op=PAM:session_close
grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix
acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5
res=success'
[ 4580.680282] audit: type=1104 audit(1534436552.103:169): pid=1473 uid=0
auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 6862.846945] INFO: rcu_sched self-detected stall on CPU
[ 6862.846946] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 6862.846950] 0-...: (4 GPs behind) idle=3f6/1/0 softirq=1100880/1100880
fqs=0
[ 6862.846956] 2-...: (14 GPs behind) idle=844/0/0 softirq=1162047/1162047
fqs=0
[ 6862.846958] 3-...: (1 ticks this GP) idle=37e/1/0 softirq=1154645/1154645
fqs=0
[ 6862.846960] 4-...: (4 GPs behind) idle=fb2/1/0 softirq=1208765/1208765
fqs=0
[ 6862.846961] 5-...: (10 GPs behind) idle=f66/1/0 softirq=1157598/1157598
fqs=0
[ 6862.846963] 6-...: (14 GPs behind) idle=8dc/0/0 softirq=1263636/1263636
fqs=0
[ 6862.846965] 7-...: (14 GPs behind) idle=202/1/0 softirq=1156389/1156389
fqs=0
[ 6862.846966] 8-...: (14 GPs behind) idle=212/1/0 softirq=1136417/1136417
fqs=0
[ 6862.846968] 9-...: (136 GPs behind) idle=dea/1/0 softirq=1055775/1055775
fqs=0
[ 6862.846969] 10-...: (47 GPs behind) idle=b7a/1/0 softirq=1132407/1132407
fqs=0
[ 6862.846970] 11-...: (15 GPs behind) idle=efe/1/0 softirq=1283226/1283226
fqs=0
[ 6862.846970] (detected by 1, t=342825 jiffies, g=429028, c=429027, q=0)
[ 6862.846975] Sending NMI from CPU 1 to CPUs 0:
[ 6862.847001] NMI backtrace for cpu 0
[ 6862.847020] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6862.847021] task: ffffffff82212480 task.stack: ffffffff82200000
[ 6862.847025] RIP: 0010:blkif_copy_from_grant+0x21/0x70 [xen_blkfront]
[ 6862.847026] RSP: 0000:ffff88035d803d90 EFLAGS: 00000006
[ 6862.847027] RAX: ffffea0006b99380 RBX: ffff88035d803e58 RCX: ffff88035d803e58
[ 6862.847027] RDX: 0000000000000017 RSI: 0000000000000000 RDI: 0000000000024d1a
[ 6862.847028] RBP: 0000000000001000 R08: 0000000000001000 R09: 0000000000001000
[ 6862.847029] R10: 0000000000000040 R11: ffff88035b57b000 R12: 0000000000001000
[ 6862.847029] R13: ffffffffc00027e0 R14: 0000000000024d1b R15: 0000000000001000
[ 6862.847031] FS: 0000000000000000(0000) GS:ffff88035d800000(0000)
knlGS:0000000000000000
[ 6862.847032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6862.847032] CR2: 00000000010d6640 CR3: 000000000220a004 CR4: 00000000003606f0
[ 6862.847034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6862.847035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6862.847035] Call Trace:
[ 6862.847037] <IRQ>
[ 6862.847041] gnttab_foreach_grant_in_range+0x66/0x100
[ 6862.847045] blkif_interrupt+0x2f7/0xd70 [xen_blkfront]
[ 6862.847047] __handle_irq_event_percpu+0x7b/0x190
[ 6862.847049] handle_irq_event_percpu+0x30/0x70
[ 6862.847051] handle_irq_event+0x36/0x60
[ 6862.847052] handle_edge_irq+0x73/0x190
[ 6862.847053] generic_handle_irq+0x24/0x30
[ 6862.847055] __evtchn_fifo_handle_events+0x187/0x1b0
[ 6862.847056] __xen_evtchn_do_upcall+0x42/0x80
[ 6862.847059] xen_evtchn_do_upcall+0x27/0x40
[ 6862.847061] xen_hvm_callback_vector+0x87/0x90
[ 6862.847062] </IRQ>
[ 6862.847063] RIP: 0010:native_safe_halt+0x2/0x10
[ 6862.847063] RSP: 0000:ffffffff82203e88 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffff0c
[ 6862.847064] RAX: ffffffff8186b8c0 RBX: ffffffff82212480 RCX: 0000000000000000
[ 6862.847065] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 6862.847065] RBP: 0000000000000000 R08: 000000001bc5621b R09: ffffffff82974398
[ 6862.847066] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82212480
[ 6862.847066] R13: ffffffff82212480 R14: 0000000000000000 R15: 0000000000000000
[ 6862.847068] ? __sched_text_end+0x3/0x3
[ 6862.847071] default_idle+0x1a/0xf0
[ 6862.847073] do_idle+0x166/0x1d0
[ 6862.847075] cpu_startup_entry+0x6f/0x80
[ 6862.847077] start_kernel+0x682/0x6c3
[ 6862.847079] secondary_startup_64+0xa5/0xb0
[ 6862.847081] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 01 41
54 41 89 d4 8b 51 08 55 53 48 89 cb 48 8b 40 78 48 8b 04 d0 48 8b 40 08 <65> 48
8b 2c 25 40 5c 01 00 83 85 a0 19 00 00 01 48 2b 05 48 52
[ 6862.847546] 5-...: (10 GPs behind) idle=f66/1/0 softirq=1157598/1157598
fqs=0
[ 6862.847559] (t=342826 jiffies g=429028 c=429027 q=6)
[ 6862.847569] rcu_sched kthread starved for 342826 jiffies! g429028 c429027
f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
[ 6862.847582] rcu_sched R running task 0 8 2 0x80000000
[ 6862.847595] Call Trace:
[ 6862.847601] ? __schedule+0x3dc/0x870
[ 6862.847607] schedule+0x32/0x80
[ 6862.847613] schedule_timeout+0x164/0x380
[ 6862.847619] ? call_timer_fn+0x130/0x130
[ 6862.847625] ? rcu_gp_kthread+0x577/0x980
[ 6862.847631] rcu_gp_kthread+0x577/0x980
[ 6862.847637] kthread+0xff/0x130
[ 6862.847643] ? _synchronize_rcu_expedited.constprop.65+0x370/0x370
[ 6862.847651] ? kthread_create_on_node+0x70/0x70
[ 6862.847658] ret_from_fork+0x35/0x40
[ 6862.847998] Sending NMI from CPU 1 to CPUs 2:
[ 6862.848023] NMI backtrace for cpu 2 skipped: idling at pc 0xffffffff8186bbe2
[ 6862.849023] Sending NMI from CPU 1 to CPUs 3:
[ 6862.849058] NMI backtrace for cpu 3
[ 6862.849072] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6862.849072] task: ffff88035ba85c40 task.stack: ffffc9000190c000
[ 6862.849074] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6862.849075] RSP: 0018:ffff88035d8c3e60 EFLAGS: 00000002
[ 6862.849076] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000017
[ 6862.849076] RDX: 0000000000000000 RSI: ffff88035d8c3e70 RDI: 0000000000000003
[ 6862.849077] RBP: ffff88035d8e1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6862.849077] R10: 0000000000000040 R11: ffff88035bbd9200 R12: ffff88035da21a40
[ 6862.849078] R13: 0000000000000001 R14: 0000000000100000 R15: ffff88035d8e1a84
[ 6862.849078] FS: 0000000000000000(0000) GS:ffff88035d8c0000(0000)
knlGS:0000000000000000
[ 6862.849079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6862.849079] CR2: 00007c81d1d743b0 CR3: 000000000220a004 CR4: 00000000003606e0
[ 6862.849081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6862.849081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6862.849081] Call Trace:
[ 6862.849082] <IRQ>
[ 6862.849084] ? xen_poll_irq_timeout+0x50/0x80
[ 6862.849085] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6862.849087] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6862.849088] ? rcu_process_callbacks+0x384/0xa40
[ 6862.849089] ? rebalance_domains+0x253/0x2b0
[ 6862.849090] ? __do_softirq+0x112/0x2b4
[ 6862.849092] ? irq_exit+0xee/0x100
[ 6862.849093] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6862.849094] ? xen_hvm_callback_vector+0x87/0x90
[ 6862.849094] </IRQ>
[ 6862.849095] ? __sched_text_end+0x3/0x3
[ 6862.849096] ? native_safe_halt+0x2/0x10
[ 6862.849097] ? default_idle+0x1a/0xf0
[ 6862.849098] ? do_idle+0x166/0x1d0
[ 6862.849099] ? cpu_startup_entry+0x6f/0x80
[ 6862.849100] ? start_secondary+0x1ae/0x200
[ 6862.849101] ? secondary_startup_64+0xa5/0xb0
[ 6862.849101] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6862.850051] Sending NMI from CPU 1 to CPUs 4:
[ 6863.047982] NMI backtrace for cpu 4
[ 6863.047984] CPU: 4 PID: 6264 Comm: top Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.047985] task: ffff8802d1151ec0 task.stack: ffffc9000452c000
[ 6863.047987] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.047988] RSP: 0000:ffffc9000452f780 EFLAGS: 00000202
[ 6863.047988] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000001d
[ 6863.047989] RDX: 0000000000000000 RSI: ffffc9000452f790 RDI: 0000000000000003
[ 6863.047989] RBP: ffff88035d921a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.047990] R10: 0000000000000040 R11: ffff88035b42c200 R12: ffff88035d8a1a40
[ 6863.047990] R13: 0000000000000001 R14: 0000000000140000 R15: ffff88035d921a84
[ 6863.047991] FS: 000079f4d5048900(0000) GS:ffff88035d900000(0000)
knlGS:0000000000000000
[ 6863.047991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.047992] CR2: 000079f4d409eab0 CR3: 00000001aa87e001 CR4: 00000000003606e0
[ 6863.047993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.047995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.047995] Call Trace:
[ 6863.047998] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.047999] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.048003] ? _raw_spin_lock+0x1d/0x20
[ 6863.048005] ? list_lru_count_one+0x28/0x70
[ 6863.048007] ? super_cache_count+0x7d/0xf0
[ 6863.048008] ? shrink_slab.part.50+0x12f/0x400
[ 6863.048011] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.048013] ? shrink_node+0x10e/0x320
[ 6863.048015] ? do_try_to_free_pages+0xe3/0x350
[ 6863.048016] ? try_to_free_pages+0x107/0x1d0
[ 6863.048018] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.048019] ? __switch_to_asm+0x34/0x70
[ 6863.048020] ? __switch_to_asm+0x40/0x70
[ 6863.048021] ? __switch_to_asm+0x40/0x70
[ 6863.048022] ? __switch_to_asm+0x40/0x70
[ 6863.048023] ? __switch_to_asm+0x40/0x70
[ 6863.048024] ? __switch_to_asm+0x40/0x70
[ 6863.048024] ? __switch_to_asm+0x40/0x70
[ 6863.048025] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.048027] ? __do_page_cache_readahead+0x129/0x340
[ 6863.048028] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.048029] ? find_get_entry+0x19/0x170
[ 6863.048030] ? filemap_fault+0x6b3/0x860
[ 6863.048030] ? filemap_fault+0x6b3/0x860
[ 6863.048031] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.048032] ? alloc_set_pte+0x343/0x520
[ 6863.048033] ? filemap_map_pages+0x2d5/0x510
[ 6863.048035] ? ext4_filemap_fault+0x2c/0x40
[ 6863.048036] ? __do_fault+0x1e/0xb5
[ 6863.048036] ? __handle_mm_fault+0xc40/0xda0
[ 6863.048038] ? handle_mm_fault+0xc6/0x1b0
[ 6863.048039] ? __do_page_fault+0x255/0x4f0
[ 6863.048040] ? do_page_fault+0x33/0x120
[ 6863.048041] ? page_fault+0x2f/0x50
[ 6863.048042] ? page_fault+0x45/0x50
[ 6863.048042] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.048980] Sending NMI from CPU 1 to CPUs 5:
[ 6863.049010] NMI backtrace for cpu 5
[ 6863.049012] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.049013] task: ffff88035ba83d80 task.stack: ffffc9000191c000
[ 6863.049015] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.049015] RSP: 0000:ffff88035d943cb0 EFLAGS: 00000002
[ 6863.049016] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000023
[ 6863.049017] RDX: 0000000000000000 RSI: ffff88035d943cc0 RDI: 0000000000000003
[ 6863.049018] RBP: ffff88035d961a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.049018] R10: 0000000000000040 R11: ffff88035b47d800 R12: ffff88035d8e1a40
[ 6863.049019] R13: 0000000000000001 R14: 0000000000180000 R15: ffff88035d961a84
[ 6863.049020] FS: 0000000000000000(0000) GS:ffff88035d940000(0000)
knlGS:0000000000000000
[ 6863.049021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.049021] CR2: 00000000005d37c0 CR3: 000000000220a005 CR4: 00000000003606e0
[ 6863.049023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.049023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.049024] Call Trace:
[ 6863.049024] <IRQ>
[ 6863.049026] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.049027] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.049028] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.049029] ? rcu_dump_cpu_stacks+0x44/0xc9
[ 6863.049030] ? rcu_check_callbacks+0x842/0x990
[ 6863.049031] ? irqtime_account_process_tick.isra.7+0xed/0x180
[ 6863.049032] ? tick_sched_do_timer+0x60/0x60
[ 6863.049033] ? update_process_times+0x28/0x50
[ 6863.049034] ? tick_sched_handle+0x23/0x60
[ 6863.049035] ? tick_sched_timer+0x34/0x70
[ 6863.049035] ? __hrtimer_run_queues+0xde/0x230
[ 6863.049036] ? hrtimer_interrupt+0xa6/0x1f0
[ 6863.049037] ? xen_timer_interrupt+0x1e/0x30
[ 6863.049038] ? __handle_irq_event_percpu+0x7b/0x190
[ 6863.049039] ? handle_irq_event_percpu+0x30/0x70
[ 6863.049040] ? handle_percpu_irq+0x37/0x50
[ 6863.049043] ? generic_handle_irq+0x24/0x30
[ 6863.049044] ? __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.049045] ? __xen_evtchn_do_upcall+0x42/0x80
[ 6863.049046] ? xen_evtchn_do_upcall+0x27/0x40
[ 6863.049047] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.049047] </IRQ>
[ 6863.049049] ? __sched_text_end+0x3/0x3
[ 6863.049050] ? native_safe_halt+0x2/0x10
[ 6863.049050] ? default_idle+0x1a/0xf0
[ 6863.049051] ? do_idle+0x166/0x1d0
[ 6863.049052] ? cpu_startup_entry+0x6f/0x80
[ 6863.049053] ? start_secondary+0x1ae/0x200
[ 6863.049054] ? secondary_startup_64+0xa5/0xb0
[ 6863.049055] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.050000] Sending NMI from CPU 1 to CPUs 6:
[ 6863.248957] NMI backtrace for cpu 6
[ 6863.248958] CPU: 6 PID: 4295 Comm: rustc Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.248959] task: ffff88019fbbdc40 task.stack: ffffc90006390000
[ 6863.248960] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.248961] RSP: 0000:ffffc90006393780 EFLAGS: 00000202
[ 6863.248961] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000029
[ 6863.248962] RDX: 0000000000000000 RSI: ffffc90006393790 RDI: 0000000000000003
[ 6863.248962] RBP: ffff88035d9a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.248962] R10: 0000000000000040 R11: ffff88035b4cf800 R12: 0000000000000001
[ 6863.248963] R13: 0000000000000100 R14: 00000000001c0000 R15: 0000000000000000
[ 6863.248963] FS: 00007ed99a4a1700(0000) GS:ffff88035d980000(0000)
knlGS:0000000000000000
[ 6863.248964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.248964] CR2: 00007ed9a2119e50 CR3: 00000001516c6006 CR4: 00000000003606e0
[ 6863.248966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.248966] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.248966] Call Trace:
[ 6863.248968] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.248969] ? __pv_queued_spin_lock_slowpath+0x22c/0x260
[ 6863.248971] ? _raw_spin_lock+0x1d/0x20
[ 6863.248972] ? list_lru_count_one+0x28/0x70
[ 6863.248973] ? super_cache_count+0x7d/0xf0
[ 6863.248974] ? shrink_slab.part.50+0x12f/0x400
[ 6863.248976] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.248977] ? shrink_node+0x10e/0x320
[ 6863.248978] ? do_try_to_free_pages+0xe3/0x350
[ 6863.248979] ? try_to_free_pages+0x107/0x1d0
[ 6863.248980] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.248981] ? __switch_to_asm+0x34/0x70
[ 6863.248982] ? __switch_to_asm+0x40/0x70
[ 6863.248983] ? __switch_to_asm+0x40/0x70
[ 6863.248983] ? __switch_to_asm+0x40/0x70
[ 6863.248984] ? __switch_to_asm+0x40/0x70
[ 6863.248985] ? __switch_to_asm+0x40/0x70
[ 6863.248986] ? __switch_to_asm+0x40/0x70
[ 6863.248987] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.248988] ? __do_page_cache_readahead+0x129/0x340
[ 6863.248993] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.248996] ? find_get_entry+0x19/0x170
[ 6863.248996] ? filemap_fault+0x6b3/0x860
[ 6863.248997] ? filemap_fault+0x6b3/0x860
[ 6863.248999] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.248999] ? alloc_set_pte+0x343/0x520
[ 6863.249004] ? filemap_map_pages+0x2d5/0x510
[ 6863.249006] ? ext4_filemap_fault+0x2c/0x40
[ 6863.249008] ? __do_fault+0x1e/0xb5
[ 6863.249009] ? __handle_mm_fault+0xc40/0xda0
[ 6863.249010] ? __switch_to_asm+0x34/0x70
[ 6863.249012] ? handle_mm_fault+0xc6/0x1b0
[ 6863.249014] ? __do_page_fault+0x255/0x4f0
[ 6863.249015] ? do_page_fault+0x33/0x120
[ 6863.249016] ? page_fault+0x2f/0x50
[ 6863.249018] ? page_fault+0x45/0x50
[ 6863.249018] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.249953] Sending NMI from CPU 1 to CPUs 7:
[ 6863.249977] NMI backtrace for cpu 7
[ 6863.249980] CPU: 7 PID: 266 Comm: systemd-journal Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.249981] task: ffff88033eba5c40 task.stack: ffffc90001f38000
[ 6863.249982] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.249983] RSP: 0000:ffffc90001f3b780 EFLAGS: 00000202
[ 6863.249983] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002f
[ 6863.249984] RDX: 0000000000000000 RSI: ffffc90001f3b790 RDI: 0000000000000003
[ 6863.249984] RBP: ffff88035d9e1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.249984] R10: 0000000000000040 R11: ffff88035b52fe00 R12: ffff88035d821a40
[ 6863.249985] R13: 0000000000000001 R14: 0000000000200000 R15: ffff88035d9e1a84
[ 6863.249985] FS: 00007db8406e2940(0000) GS:ffff88035d9c0000(0000)
knlGS:0000000000000000
[ 6863.249986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.249986] CR2: 00007db8401c7ab0 CR3: 0000000039bcc004 CR4: 00000000003606e0
[ 6863.249988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.249988] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.249988] Call Trace:
[ 6863.249990] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.249991] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.249992] ? _raw_spin_lock+0x1d/0x20
[ 6863.249993] ? list_lru_count_one+0x28/0x70
[ 6863.249994] ? super_cache_count+0x7d/0xf0
[ 6863.249995] ? shrink_slab.part.50+0x12f/0x400
[ 6863.249996] ? shrink_node+0x10e/0x320
[ 6863.249997] ? do_try_to_free_pages+0xe3/0x350
[ 6863.249998] ? try_to_free_pages+0x107/0x1d0
[ 6863.250002] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.250003] ? __switch_to_asm+0x34/0x70
[ 6863.250005] ? __switch_to_asm+0x40/0x70
[ 6863.250006] ? __switch_to_asm+0x40/0x70
[ 6863.250007] ? __switch_to_asm+0x40/0x70
[ 6863.250009] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.250011] ? __do_page_cache_readahead+0x129/0x340
[ 6863.250012] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.250014] ? find_get_entry+0x19/0x170
[ 6863.250015] ? filemap_fault+0x6b3/0x860
[ 6863.250016] ? filemap_fault+0x6b3/0x860
[ 6863.250017] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.250018] ? alloc_set_pte+0x343/0x520
[ 6863.250020] ? filemap_map_pages+0x209/0x510
[ 6863.250021] ? ext4_filemap_fault+0x2c/0x40
[ 6863.250023] ? __do_fault+0x1e/0xb5
[ 6863.250024] ? __handle_mm_fault+0xc40/0xda0
[ 6863.250025] ? handle_mm_fault+0xc6/0x1b0
[ 6863.250027] ? __do_page_fault+0x255/0x4f0
[ 6863.250028] ? do_page_fault+0x33/0x120
[ 6863.250030] ? page_fault+0x2f/0x50
[ 6863.250031] ? page_fault+0x45/0x50
[ 6863.250032] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.250973] Sending NMI from CPU 1 to CPUs 8:
[ 6863.250993] NMI backtrace for cpu 8
[ 6863.251008] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.251009] task: ffff88035ba95c40 task.stack: ffffc90001934000
[ 6863.251011] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.251011] RSP: 0000:ffff88035da03e60 EFLAGS: 00000002
[ 6863.251012] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000035
[ 6863.251013] RDX: 0000000000000000 RSI: ffff88035da03e70 RDI: 0000000000000003
[ 6863.251013] RBP: ffff88035da21a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.251014] R10: 0000000000000040 R11: ffff88035b57b600 R12: 0000000000000001
[ 6863.251014] R13: 0000000000000100 R14: 0000000000240000 R15: 0000000000000000
[ 6863.251015] FS: 0000000000000000(0000) GS:ffff88035da00000(0000)
knlGS:0000000000000000
[ 6863.251016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.251017] CR2: 00007f1cf2ac0510 CR3: 000000000220a005 CR4: 00000000003606e0
[ 6863.251018] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.251019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.251019] Call Trace:
[ 6863.251020] <IRQ>
[ 6863.251025] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.251026] ? __pv_queued_spin_lock_slowpath+0x22c/0x260
[ 6863.251028] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.251029] ? rcu_process_callbacks+0x384/0xa40
[ 6863.251030] ? rebalance_domains+0x253/0x2b0
[ 6863.251032] ? __do_softirq+0x112/0x2b4
[ 6863.251033] ? irq_exit+0xee/0x100
[ 6863.251035] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6863.251036] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.251036] </IRQ>
[ 6863.251039] ? __sched_text_end+0x3/0x3
[ 6863.251040] ? native_safe_halt+0x2/0x10
[ 6863.251041] ? default_idle+0x1a/0xf0
[ 6863.251042] ? do_idle+0x166/0x1d0
[ 6863.251044] ? cpu_startup_entry+0x6f/0x80
[ 6863.251048] ? start_secondary+0x1ae/0x200
[ 6863.251049] ? secondary_startup_64+0xa5/0xb0
[ 6863.251050] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.251985] Sending NMI from CPU 1 to CPUs 9:
[ 6863.252003] NMI backtrace for cpu 9
[ 6863.252004] CPU: 9 PID: 435 Comm: systemd-logind Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.252005] task: ffff88033cfc5c40 task.stack: ffffc90001f58000
[ 6863.252006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x207/0x260
[ 6863.252006] RSP: 0000:ffffc90001f5b7c0 EFLAGS: 00000206
[ 6863.252007] RAX: 0000000000000000 RBX: ffff880039271280 RCX: 000000000000003b
[ 6863.252008] RDX: 0000000000001839 RSI: 0000000000000000 RDI: 0000000000000006
[ 6863.252008] RBP: ffff88035da61a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.252009] R10: 0000000000000040 R11: ffff88035b5f0400 R12: ffff88035d9a1a40
[ 6863.252010] R13: 0000000000000001 R14: 0000000000280000 R15: ffff88035da61a84
[ 6863.252011] FS: 0000717a785909c0(0000) GS:ffff88035da40000(0000)
knlGS:0000000000000000
[ 6863.252011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.252012] CR2: 0000717a7803b6c0 CR3: 0000000034d76004 CR4: 00000000003606e0
[ 6863.252014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.252014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.252014] Call Trace:
[ 6863.252017] _raw_spin_lock+0x1d/0x20
[ 6863.252018] list_lru_count_one+0x28/0x70
[ 6863.252020] super_cache_count+0x7d/0xf0
[ 6863.252021] shrink_slab.part.50+0x12f/0x400
[ 6863.252024] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.252025] shrink_node+0x10e/0x320
[ 6863.252027] do_try_to_free_pages+0xe3/0x350
[ 6863.252029] try_to_free_pages+0x107/0x1d0
[ 6863.252031] __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.252032] ? __switch_to_asm+0x34/0x70
[ 6863.252033] ? __switch_to_asm+0x40/0x70
[ 6863.252034] ? __switch_to_asm+0x40/0x70
[ 6863.252035] ? __switch_to_asm+0x40/0x70
[ 6863.252037] __alloc_pages_nodemask+0x270/0x2b0
[ 6863.252039] __do_page_cache_readahead+0x129/0x340
[ 6863.252041] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.252042] ? find_get_entry+0x19/0x170
[ 6863.252043] ? filemap_fault+0x6b3/0x860
[ 6863.252044] filemap_fault+0x6b3/0x860
[ 6863.252045] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.252046] ? alloc_set_pte+0x343/0x520
[ 6863.252048] ? filemap_map_pages+0x2d5/0x510
[ 6863.252050] ext4_filemap_fault+0x2c/0x40
[ 6863.252051] __do_fault+0x1e/0xb5
[ 6863.252052] __handle_mm_fault+0xc40/0xda0
[ 6863.252054] ? hrtimer_start_range_ns+0x18d/0x340
[ 6863.252055] handle_mm_fault+0xc6/0x1b0
[ 6863.252057] __do_page_fault+0x255/0x4f0
[ 6863.252058] do_page_fault+0x33/0x120
[ 6863.252060] ? page_fault+0x2f/0x50
[ 6863.252061] page_fault+0x45/0x50
[ 6863.252062] RIP: 7808db31:0x717a78384cc0
[ 6863.252063] RSP: 825f5aa0:00005eb4825e9e80 EFLAGS: 717a780b8f40
[ 6863.252063] Code: 08 85 c0 74 f7 4c 8b 7d 00 4d 85 ff 0f 84 af fe ff ff 41
0f 0d 0f e9 a6 fe ff ff 49 63 7c 24 40 ff 14 25 20 69 22 82 84 c0 75 a1 <f3> 90
83 ea 01 75 85 eb 98 48 83 3c 24 00 c6 43 01 00 74 18 c6
[ 6863.252999] Sending NMI from CPU 1 to CPUs 10:
[ 6863.253023] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.254011] Sending NMI from CPU 1 to CPUs 11:
[ 6863.254035] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.255028] rcu_sched kthread starved for 343233 jiffies! g429028 c429027
f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
[ 6863.255029] rcu_sched R running task 0 8 2 0x80000000
[ 6863.255030] Call Trace:
[ 6863.255032] ? __schedule+0x3dc/0x870
[ 6863.255033] schedule+0x32/0x80
[ 6863.255034] schedule_timeout+0x164/0x380
[ 6863.255036] ? call_timer_fn+0x130/0x130
[ 6863.255037] ? rcu_gp_kthread+0x577/0x980
[ 6863.255037] rcu_gp_kthread+0x577/0x980
[ 6863.255039] kthread+0xff/0x130
[ 6863.255040] ? _synchronize_rcu_expedited.constprop.65+0x370/0x370
[ 6863.255041] ? kthread_create_on_node+0x70/0x70
[ 6863.255042] ret_from_fork+0x35/0x40
[ 6863.255054] Sending NMI from CPU 5 to CPUs 0:
[ 6863.255089] NMI backtrace for cpu 0
[ 6863.255090] CPU: 0 PID: 640 Comm: Xorg Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.255091] task: ffff88003267dc40 task.stack: ffffc900022f4000
[ 6863.255092] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.255093] RSP: 0000:ffffc900022f7780 EFLAGS: 00000202
[ 6863.255093] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005
[ 6863.255094] RDX: 0000000000000000 RSI: ffffc900022f7790 RDI: 0000000000000003
[ 6863.255094] RBP: ffff88035d821a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.255094] R10: 0000000000000040 R11: ffff88035d54c600 R12: ffff88035d921a40
[ 6863.255095] R13: 0000000000000001 R14: 0000000000040000 R15: ffff88035d821a84
[ 6863.255096] FS: 000074f9a54f8ac0(0000) GS:ffff88035d800000(0000)
knlGS:0000000000000000
[ 6863.255096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.255096] CR2: 0000000000578ed0 CR3: 0000000032788001 CR4: 00000000003606f0
[ 6863.255098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.255098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.255098] Call Trace:
[ 6863.255100] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.255101] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.255102] ? _raw_spin_lock+0x1d/0x20
[ 6863.255103] ? list_lru_count_one+0x28/0x70
[ 6863.255118] ? super_cache_count+0x7d/0xf0
[ 6863.255118] ? shrink_slab.part.50+0x12f/0x400
[ 6863.255120] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.255121] ? shrink_node+0x10e/0x320
[ 6863.255122] ? do_try_to_free_pages+0xe3/0x350
[ 6863.255123] ? try_to_free_pages+0x107/0x1d0
[ 6863.255125] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.255126] ? __switch_to_asm+0x34/0x70
[ 6863.255127] ? __switch_to_asm+0x40/0x70
[ 6863.255128] ? __switch_to_asm+0x40/0x70
[ 6863.255129] ? __switch_to_asm+0x40/0x70
[ 6863.255131] ? __switch_to_asm+0x40/0x70
[ 6863.255132] ? __switch_to_asm+0x40/0x70
[ 6863.255133] ? __switch_to_asm+0x40/0x70
[ 6863.255134] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.255135] ? __do_page_cache_readahead+0x129/0x340
[ 6863.255137] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.255138] ? find_get_entry+0x19/0x170
[ 6863.255139] ? filemap_fault+0x6b3/0x860
[ 6863.255140] ? filemap_fault+0x6b3/0x860
[ 6863.255141] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.255142] ? alloc_set_pte+0x343/0x520
[ 6863.255143] ? filemap_map_pages+0x209/0x510
[ 6863.255145] ? ext4_filemap_fault+0x2c/0x40
[ 6863.255146] ? __do_fault+0x1e/0xb5
[ 6863.255147] ? __handle_mm_fault+0xc40/0xda0
[ 6863.255148] ? handle_mm_fault+0xc6/0x1b0
[ 6863.255150] ? __do_page_fault+0x255/0x4f0
[ 6863.255151] ? do_page_fault+0x33/0x120
[ 6863.255152] ? page_fault+0x2f/0x50
[ 6863.255153] ? page_fault+0x45/0x50
[ 6863.255154] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.256086] Sending NMI from CPU 5 to CPUs 2:
[ 6863.256107] NMI backtrace for cpu 2
[ 6863.256108] CPU: 2 PID: 1 Comm: systemd Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.256108] task: ffff88035ba10000 task.stack: ffffc900018a4000
[ 6863.256110] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.256111] RSP: 0000:ffffc900018a7780 EFLAGS: 00000206
[ 6863.256111] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000011
[ 6863.256112] RDX: 0000000000000000 RSI: ffffc900018a7790 RDI: 0000000000000003
[ 6863.256112] RBP: ffff88035d8a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.256113] R10: 0000000000000040 R11: ffff88035bb8ac00 R12: ffff88035d821a40
[ 6863.256113] R13: 0000000000000001 R14: 00000000000c0000 R15: ffff88035d8a1a84
[ 6863.256114] FS: 00007773dbc39e40(0000) GS:ffff88035d880000(0000)
knlGS:0000000000000000
[ 6863.256114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.256115] CR2: 0000618326569e11 CR3: 00000003587f8005 CR4: 00000000003606e0
[ 6863.256116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.256116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.256116] Call Trace:
[ 6863.256119] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.256120] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.256122] ? _raw_spin_lock+0x1d/0x20
[ 6863.256123] ? list_lru_count_one+0x28/0x70
[ 6863.256124] ? super_cache_count+0x7d/0xf0
[ 6863.256125] ? shrink_slab.part.50+0x12f/0x400
[ 6863.256126] ? shrink_node+0x10e/0x320
[ 6863.256127] ? do_try_to_free_pages+0xe3/0x350
[ 6863.256128] ? try_to_free_pages+0x107/0x1d0
[ 6863.256129] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.256130] ? __switch_to_asm+0x34/0x70
[ 6863.256131] ? __switch_to_asm+0x40/0x70
[ 6863.256132] ? __switch_to_asm+0x40/0x70
[ 6863.256133] ? __switch_to_asm+0x40/0x70
[ 6863.256134] ? __switch_to_asm+0x40/0x70
[ 6863.256135] ? __switch_to_asm+0x40/0x70
[ 6863.256135] ? __switch_to_asm+0x40/0x70
[ 6863.256136] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.256138] ? __do_page_cache_readahead+0x129/0x340
[ 6863.256139] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.256140] ? find_get_entry+0x19/0x170
[ 6863.256141] ? filemap_fault+0x6b3/0x860
[ 6863.256141] ? filemap_fault+0x6b3/0x860
[ 6863.256142] ? page_add_file_rmap+0x18/0x1f0
[ 6863.256143] ? alloc_set_pte+0x343/0x520
[ 6863.256144] ? filemap_map_pages+0x2d5/0x510
[ 6863.256145] ? ext4_filemap_fault+0x2c/0x40
[ 6863.256146] ? __do_fault+0x1e/0xb5
[ 6863.256147] ? __handle_mm_fault+0xc40/0xda0
[ 6863.256148] ? timerfd_read+0xc7/0x280
[ 6863.256150] ? handle_mm_fault+0xc6/0x1b0
[ 6863.256151] ? __do_page_fault+0x255/0x4f0
[ 6863.256152] ? do_page_fault+0x33/0x120
[ 6863.256153] ? page_fault+0x2f/0x50
[ 6863.256154] ? page_fault+0x45/0x50
[ 6863.256154] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.257103] Sending NMI from CPU 5 to CPUs 4:
[ 6863.257117] NMI backtrace for cpu 4
[ 6863.257118] CPU: 4 PID: 6264 Comm: top Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.257118] task: ffff8802d1151ec0 task.stack: ffffc9000452c000
[ 6863.257119] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.257120] RSP: 0018:ffffc9000452f780 EFLAGS: 00000202
[ 6863.257120] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000001d
[ 6863.257121] RDX: 0000000000000000 RSI: ffffc9000452f790 RDI: 0000000000000003
[ 6863.257121] RBP: ffff88035d921a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.257122] R10: 0000000000000040 R11: ffff88035b42c200 R12: ffff88035d8a1a40
[ 6863.257122] R13: 0000000000000001 R14: 0000000000140000 R15: ffff88035d921a84
[ 6863.257123] FS: 000079f4d5048900(0000) GS:ffff88035d900000(0000)
knlGS:0000000000000000
[ 6863.257123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.257123] CR2: 00007efb66abfb30 CR3: 00000001aa87e001 CR4: 00000000003606e0
[ 6863.257127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.257127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.257128] Call Trace:
[ 6863.257129] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.257130] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.257131] ? _raw_spin_lock+0x1d/0x20
[ 6863.257132] ? list_lru_count_one+0x28/0x70
[ 6863.257133] ? super_cache_count+0x7d/0xf0
[ 6863.257134] ? shrink_slab.part.50+0x12f/0x400
[ 6863.257135] ? shrink_node+0x10e/0x320
[ 6863.257136] ? do_try_to_free_pages+0xe3/0x350
[ 6863.257137] ? try_to_free_pages+0x107/0x1d0
[ 6863.257138] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.257139] ? __switch_to_asm+0x34/0x70
[ 6863.257140] ? __switch_to_asm+0x40/0x70
[ 6863.257141] ? __switch_to_asm+0x40/0x70
[ 6863.257141] ? __switch_to_asm+0x40/0x70
[ 6863.257142] ? __switch_to_asm+0x40/0x70
[ 6863.257143] ? __switch_to_asm+0x40/0x70
[ 6863.257144] ? __switch_to_asm+0x40/0x70
[ 6863.257145] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.257146] ? __do_page_cache_readahead+0x129/0x340
[ 6863.257147] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.257148] ? find_get_entry+0x19/0x170
[ 6863.257149] ? filemap_fault+0x6b3/0x860
[ 6863.257150] ? filemap_fault+0x6b3/0x860
[ 6863.257151] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.257151] ? alloc_set_pte+0x343/0x520
[ 6863.257152] ? filemap_map_pages+0x2d5/0x510
[ 6863.257154] ? ext4_filemap_fault+0x2c/0x40
[ 6863.257155] ? __do_fault+0x1e/0xb5
[ 6863.257156] ? __handle_mm_fault+0xc40/0xda0
[ 6863.257157] ? handle_mm_fault+0xc6/0x1b0
[ 6863.257158] ? __do_page_fault+0x255/0x4f0
[ 6863.257159] ? do_page_fault+0x33/0x120
[ 6863.257160] ? page_fault+0x2f/0x50
[ 6863.257161] ? page_fault+0x45/0x50
[ 6863.257161] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.258115] NMI backtrace for cpu 5
[ 6863.258116] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.258117] Call Trace:
[ 6863.258118] <IRQ>
[ 6863.258119] dump_stack+0x5c/0x85
[ 6863.258120] nmi_cpu_backtrace+0xbf/0xd0
[ 6863.258122] ? irq_force_complete_move+0xe0/0xe0
[ 6863.258123] nmi_trigger_cpumask_backtrace+0x86/0xc0
[ 6863.258124] rcu_dump_cpu_stacks+0x92/0xc9
[ 6863.258125] rcu_check_callbacks+0x842/0x990
[ 6863.258126] ? irqtime_account_process_tick.isra.7+0xed/0x180
[ 6863.258128] ? tick_sched_do_timer+0x60/0x60
[ 6863.258128] update_process_times+0x28/0x50
[ 6863.258129] tick_sched_handle+0x23/0x60
[ 6863.258130] tick_sched_timer+0x34/0x70
[ 6863.258131] __hrtimer_run_queues+0xde/0x230
[ 6863.258132] hrtimer_interrupt+0xa6/0x1f0
[ 6863.258133] xen_timer_interrupt+0x1e/0x30
[ 6863.258135] __handle_irq_event_percpu+0x7b/0x190
[ 6863.258136] handle_irq_event_percpu+0x30/0x70
[ 6863.258137] handle_percpu_irq+0x37/0x50
[ 6863.258138] generic_handle_irq+0x24/0x30
[ 6863.258139] __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.258141] __xen_evtchn_do_upcall+0x42/0x80
[ 6863.258142] xen_evtchn_do_upcall+0x27/0x40
[ 6863.258143] xen_hvm_callback_vector+0x87/0x90
[ 6863.258144] </IRQ>
[ 6863.258145] RIP: 0010:native_safe_halt+0x2/0x10
[ 6863.258146] RSP: 0000:ffffc9000191feb8 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffff0c
[ 6863.258146] RAX: ffffffff8186b8c0 RBX: ffff88035ba83d80 RCX: 0000000000000000
[ 6863.258147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 6863.258147] RBP: 0000000000000005 R08: 000000001bba9704 R09: ffff88035d95b7e0
[ 6863.258148] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035ba83d80
[ 6863.258148] R13: ffff88035ba83d80 R14: 0000000000000000 R15: 0000000000000000
[ 6863.258149] ? __sched_text_end+0x3/0x3
[ 6863.258150] default_idle+0x1a/0xf0
[ 6863.258151] do_idle+0x166/0x1d0
[ 6863.258153] cpu_startup_entry+0x6f/0x80
[ 6863.258154] start_secondary+0x1ae/0x200
[ 6863.258159] secondary_startup_64+0xa5/0xb0
[ 6863.258162] Sending NMI from CPU 5 to CPUs 6:
[ 6863.258177] NMI backtrace for cpu 6
[ 6863.258178] CPU: 6 PID: 4295 Comm: rustc Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.258178] task: ffff88019fbbdc40 task.stack: ffffc90006390000
[ 6863.258179] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.258180] RSP: 0000:ffffc90006393780 EFLAGS: 00000202
[ 6863.258181] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000029
[ 6863.258181] RDX: 0000000000000000 RSI: ffffc90006393790 RDI: 0000000000000003
[ 6863.258181] RBP: ffff88035d9a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.258182] R10: 0000000000000040 R11: ffff88035b4cf800 R12: ffff88035d9e1a40
[ 6863.258182] R13: 0000000000000001 R14: 00000000001c0000 R15: ffff88035d9a1a84
[ 6863.258183] FS: 00007ed99a4a1700(0000) GS:ffff88035d980000(0000)
knlGS:0000000000000000
[ 6863.258183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.258184] CR2: 00007cf603db71c0 CR3: 00000001516c6006 CR4: 00000000003606e0
[ 6863.258185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.258185] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.258186] Call Trace:
[ 6863.258187] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.258188] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.258189] ? _raw_spin_lock+0x1d/0x20
[ 6863.258190] ? list_lru_count_one+0x28/0x70
[ 6863.258191] ? super_cache_count+0x7d/0xf0
[ 6863.258192] ? shrink_slab.part.50+0x12f/0x400
[ 6863.258194] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.258195] ? shrink_node+0x10e/0x320
[ 6863.258196] ? do_try_to_free_pages+0xe3/0x350
[ 6863.258197] ? try_to_free_pages+0x107/0x1d0
[ 6863.258198] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.258199] ? __switch_to_asm+0x34/0x70
[ 6863.258199] ? __switch_to_asm+0x40/0x70
[ 6863.258200] ? __switch_to_asm+0x40/0x70
[ 6863.258201] ? __switch_to_asm+0x40/0x70
[ 6863.258202] ? __switch_to_asm+0x40/0x70
[ 6863.258203] ? __switch_to_asm+0x40/0x70
[ 6863.258204] ? __switch_to_asm+0x40/0x70
[ 6863.258205] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.258206] ? __do_page_cache_readahead+0x129/0x340
[ 6863.258207] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.258208] ? find_get_entry+0x19/0x170
[ 6863.258209] ? filemap_fault+0x6b3/0x860
[ 6863.258209] ? filemap_fault+0x6b3/0x860
[ 6863.258210] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.258211] ? alloc_set_pte+0x343/0x520
[ 6863.258212] ? filemap_map_pages+0x2d5/0x510
[ 6863.258214] ? ext4_filemap_fault+0x2c/0x40
[ 6863.258215] ? __do_fault+0x1e/0xb5
[ 6863.258215] ? __handle_mm_fault+0xc40/0xda0
[ 6863.258216] ? __switch_to_asm+0x34/0x70
[ 6863.258217] ? handle_mm_fault+0xc6/0x1b0
[ 6863.258218] ? __do_page_fault+0x255/0x4f0
[ 6863.258219] ? do_page_fault+0x33/0x120
[ 6863.258220] ? page_fault+0x2f/0x50
[ 6863.258221] ? page_fault+0x45/0x50
[ 6863.258222] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.259175] Sending NMI from CPU 5 to CPUs 7:
[ 6863.260188] NMI backtrace for cpu 7
[ 6863.260189] CPU: 7 PID: 266 Comm: systemd-journal Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.260189] task: ffff88033eba5c40 task.stack: ffffc90001f38000
[ 6863.260189] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.260189] RSP: 0000:ffff88035d9c3cc0 EFLAGS: 00000046
[ 6863.260190] RAX: 0000000000000000 RBX: 000000000000000c RCX: ffff88035d9c3d14
[ 6863.260190] RDX: 0000000000000800 RSI: 0000000000000000 RDI: 0000000000000000
[ 6863.260190] RBP: ffff88035d420f40 R08: ffffc90000000000 R09: 00000000000121b2
[ 6863.260191] R10: 00000000000129b6 R11: ffffffff82977ded R12: 0000000000000004
[ 6863.260191] R13: ffff88035d9c3d14 R14: 0000000000000010 R15: ffffffff82977de0
[ 6863.260193] FS: 00007db8406e2940(0000) GS:ffff88035d9c0000(0000)
knlGS:0000000000000000
[ 6863.260193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.260194] CR2: 00007db8401c7ab0 CR3: 0000000039bcc004 CR4: 00000000003606e0
[ 6863.260194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.260194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.260194] Call Trace:
[ 6863.260194] <IRQ>
[ 6863.260195] ? domU_write_console+0x12d/0x1b0
[ 6863.260195] ? hvc_console_print+0xcc/0x120
[ 6863.260195] ? console_unlock+0x3f3/0x4e0
[ 6863.260195] ? wake_up_klogd_work_func+0x56/0x60
[ 6863.260195] ? irq_work_run_list+0x49/0x70
[ 6863.260196] ? tick_sched_do_timer+0x60/0x60
[ 6863.260196] ? update_process_times+0x3b/0x50
[ 6863.260196] ? tick_sched_handle+0x23/0x60
[ 6863.260196] ? tick_sched_timer+0x34/0x70
[ 6863.260196] ? __hrtimer_run_queues+0xde/0x230
[ 6863.260197] ? hrtimer_interrupt+0xa6/0x1f0
[ 6863.260197] ? xen_timer_interrupt+0x1e/0x30
[ 6863.260197] ? __handle_irq_event_percpu+0x7b/0x190
[ 6863.260197] ? handle_irq_event_percpu+0x30/0x70
[ 6863.260197] ? handle_percpu_irq+0x37/0x50
[ 6863.260198] ? generic_handle_irq+0x24/0x30
[ 6863.260198] ? __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.260198] ? __xen_evtchn_do_upcall+0x42/0x80
[ 6863.260198] ? xen_evtchn_do_upcall+0x27/0x40
[ 6863.260198] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.260199] </IRQ>
[ 6863.260199] ? xen_hypercall_sched_op+0x8/0x20
[ 6863.260199] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.260199] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.260199] ? _raw_spin_lock+0x1d/0x20
[ 6863.260200] ? list_lru_count_one+0x28/0x70
[ 6863.260200] ? super_cache_count+0x7d/0xf0
[ 6863.260200] ? shrink_slab.part.50+0x12f/0x400
[ 6863.260200] ? shrink_node+0x10e/0x320
[ 6863.260200] ? do_try_to_free_pages+0xe3/0x350
[ 6863.260201] ? try_to_free_pages+0x107/0x1d0
[ 6863.260201] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.260201] ? __switch_to_asm+0x34/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260202] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.260202] ? __do_page_cache_readahead+0x129/0x340
[ 6863.260202] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.260202] ? find_get_entry+0x19/0x170
[ 6863.260202] ? filemap_fault+0x6b3/0x860
[ 6863.260203] ? filemap_fault+0x6b3/0x860
[ 6863.260203] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.260203] ? alloc_set_pte+0x343/0x520
[ 6863.260203] ? filemap_map_pages+0x209/0x510
[ 6863.260203] ? ext4_filemap_fault+0x2c/0x40
[ 6863.260204] ? __do_fault+0x1e/0xb5
[ 6863.260204] ? __handle_mm_fault+0xc40/0xda0
[ 6863.260204] ? handle_mm_fault+0xc6/0x1b0
[ 6863.260204] ? __do_page_fault+0x255/0x4f0
[ 6863.260204] ? do_page_fault+0x33/0x120
[ 6863.260205] ? page_fault+0x2f/0x50
[ 6863.260205] ? page_fault+0x45/0x50
[ 6863.260205] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.260212] Sending NMI from CPU 5 to CPUs 8:
[ 6863.260225] NMI backtrace for cpu 8
[ 6863.260226] CPU: 8 PID: 921 Comm: gnome-terminal- Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.260227] task: ffff88008478bd80 task.stack: ffffc90002474000
[ 6863.260228] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.260228] RSP: 0000:ffff88035da03e60 EFLAGS: 00000002
[ 6863.260229] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000035
[ 6863.260229] RDX: 0000000000000000 RSI: ffff88035da03e70 RDI: 0000000000000003
[ 6863.260230] RBP: ffff88035da21a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.260230] R10: 0000000000000040 R11: ffff88035b57b600 R12: ffff88035d8e1a40
[ 6863.260231] R13: 0000000000000001 R14: 0000000000240000 R15: ffff88035da21a84
[ 6863.260231] FS: 00007cf606eb0ac0(0000) GS:ffff88035da00000(0000)
knlGS:0000000000000000
[ 6863.260234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.260234] CR2: 00007cf605f5d5bf CR3: 000000012b34a003 CR4: 00000000003606e0
[ 6863.260236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.260236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.260236] Call Trace:
[ 6863.260237] <IRQ>
[ 6863.260238] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.260239] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.260241] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.260241] ? rcu_process_callbacks+0x384/0xa40
[ 6863.260243] ? rebalance_domains+0x253/0x2b0
[ 6863.260244] ? __do_softirq+0x112/0x2b4
[ 6863.260245] ? irq_exit+0xee/0x100
[ 6863.260246] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6863.260247] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.260247] </IRQ>
[ 6863.260249] ? __raw_callee_save___pv_queued_spin_unlock+0xc/0x12
[ 6863.260250] ? list_lru_count_one+0x47/0x70
[ 6863.260251] ? super_cache_count+0x64/0xf0
[ 6863.260252] ? shrink_slab.part.50+0x12f/0x400
[ 6863.260253] ? shrink_node+0x10e/0x320
[ 6863.260254] ? do_try_to_free_pages+0xe3/0x350
[ 6863.260255] ? try_to_free_pages+0x107/0x1d0
[ 6863.260256] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.260257] ? __switch_to_asm+0x34/0x70
[ 6863.260258] ? __switch_to_asm+0x40/0x70
[ 6863.260259] ? __switch_to_asm+0x40/0x70
[ 6863.260260] ? __switch_to_asm+0x40/0x70
[ 6863.260261] ? __switch_to_asm+0x40/0x70
[ 6863.260261] ? __switch_to_asm+0x40/0x70
[ 6863.260262] ? __switch_to_asm+0x40/0x70
[ 6863.260263] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.260264] ? __do_page_cache_readahead+0x129/0x340
[ 6863.260266] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.260267] ? find_get_entry+0x19/0x170
[ 6863.260267] ? filemap_fault+0x6b3/0x860
[ 6863.260268] ? filemap_fault+0x6b3/0x860
[ 6863.260269] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.260270] ? alloc_set_pte+0x343/0x520
[ 6863.260271] ? filemap_map_pages+0x2d5/0x510
[ 6863.260272] ? ext4_filemap_fault+0x2c/0x40
[ 6863.260273] ? __do_fault+0x1e/0xb5
[ 6863.260274] ? __handle_mm_fault+0xc40/0xda0
[ 6863.260275] ? handle_mm_fault+0xc6/0x1b0
[ 6863.260276] ? __do_page_fault+0x255/0x4f0
[ 6863.260277] ? do_page_fault+0x33/0x120
[ 6863.260278] ? page_fault+0x2f/0x50
[ 6863.260279] ? page_fault+0x45/0x50
[ 6863.260279] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.261224] Sending NMI from CPU 5 to CPUs 9:
[ 6863.261238] NMI backtrace for cpu 9
[ 6863.261239] CPU: 9 PID: 435 Comm: systemd-logind Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.261239] task: ffff88033cfc5c40 task.stack: ffffc90001f58000
[ 6863.261240] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.261241] RSP: 0000:ffffc90001f5b780 EFLAGS: 00000202
[ 6863.261241] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000003b
[ 6863.261241] RDX: 0000000000000000 RSI: ffffc90001f5b790 RDI: 0000000000000003
[ 6863.261242] RBP: ffff88035da61a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.261242] R10: 0000000000000040 R11: ffff88035b5f0400 R12: ffff88035d861a40
[ 6863.261243] R13: 0000000000000001 R14: 0000000000280000 R15: ffff88035da61a84
[ 6863.261243] FS: 0000717a785909c0(0000) GS:ffff88035da40000(0000)
knlGS:0000000000000000
[ 6863.261244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.261244] CR2: 0000717a7803b6c0 CR3: 0000000034d76004 CR4: 00000000003606e0
[ 6863.261245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.261246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.261246] Call Trace:
[ 6863.261247] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.261248] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.261249] ? _raw_spin_lock+0x1d/0x20
[ 6863.261250] ? list_lru_count_one+0x28/0x70
[ 6863.261251] ? super_cache_count+0x7d/0xf0
[ 6863.261254] ? shrink_slab.part.50+0x12f/0x400
[ 6863.261256] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.261257] ? shrink_node+0x10e/0x320
[ 6863.261258] ? do_try_to_free_pages+0xe3/0x350
[ 6863.261259] ? try_to_free_pages+0x107/0x1d0
[ 6863.261260] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.261261] ? __switch_to_asm+0x34/0x70
[ 6863.261262] ? __switch_to_asm+0x40/0x70
[ 6863.261262] ? __switch_to_asm+0x40/0x70
[ 6863.261263] ? __switch_to_asm+0x40/0x70
[ 6863.261265] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.261266] ? __do_page_cache_readahead+0x129/0x340
[ 6863.261267] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.261268] ? find_get_entry+0x19/0x170
[ 6863.261269] ? filemap_fault+0x6b3/0x860
[ 6863.261269] ? filemap_fault+0x6b3/0x860
[ 6863.261270] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.261271] ? alloc_set_pte+0x343/0x520
[ 6863.261272] ? filemap_map_pages+0x2d5/0x510
[ 6863.261273] ? ext4_filemap_fault+0x2c/0x40
[ 6863.261274] ? __do_fault+0x1e/0xb5
[ 6863.261275] ? __handle_mm_fault+0xc40/0xda0
[ 6863.261276] ? hrtimer_start_range_ns+0x18d/0x340
[ 6863.261277] ? handle_mm_fault+0xc6/0x1b0
[ 6863.261278] ? __do_page_fault+0x255/0x4f0
[ 6863.261279] ? do_page_fault+0x33/0x120
[ 6863.261280] ? page_fault+0x2f/0x50
[ 6863.261281] ? page_fault+0x45/0x50
[ 6863.261281] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.262236] Sending NMI from CPU 5 to CPUs 10:
[ 6863.262250] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.263248] Sending NMI from CPU 5 to CPUs 11:
[ 6863.263266] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffff8186bbe2
[ 6865.272330] audit: type=1701 audit(1534438836.696:170): auid=4294967295
uid=0 gid=0 ses=4294967295 pid=435 comm="systemd-logind"
exe="/usr/lib/systemd/systemd-logind" sig=6 res=1
[ 6870.370170] cc1plus invoked oom-killer:
gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0,
oom_score_adj=0
[ 6870.370197] cc1plus cpuset=/ mems_allowed=0
[ 6870.370209] CPU: 8 PID: 5092 Comm: cc1plus Tainted: G O
4.14.57-1.pvops.qubes.x86_64 #1
[ 6870.370225] Call Trace:
[ 6870.370236] dump_stack+0x5c/0x85
[ 6870.370248] dump_header+0x94/0x21c
[ 6870.370258] ? get_page_from_freelist+0x184/0xbd0
[ 6870.370270] oom_kill_process+0x212/0x410
[ 6870.370279] out_of_memory+0x131/0x4b0
[ 6870.370289] __alloc_pages_slowpath+0xda8/0xe90
[ 6870.370301] __alloc_pages_nodemask+0x270/0x2b0
[ 6870.370312] filemap_fault+0x54c/0x860
[ 6870.370322] ? alloc_set_pte+0x343/0x520
[ 6870.370332] ? filemap_map_pages+0x2d5/0x510
[ 6870.370345] ext4_filemap_fault+0x2c/0x40
[ 6870.370355] __do_fault+0x1e/0xb5
[ 6870.370364] __handle_mm_fault+0xc40/0xda0
[ 6870.370374] ? __switch_to_xtra+0x354/0x360
[ 6870.370384] handle_mm_fault+0xc6/0x1b0
[ 6870.370394] __do_page_fault+0x255/0x4f0
[ 6870.370404] do_page_fault+0x33/0x120
[ 6870.370414] ? page_fault+0x2f/0x50
[ 6870.370423] page_fault+0x45/0x50
[ 6870.370433] RIP: ce647960:0x79a0e57cb3f0
[ 6870.370441] RSP: ea817000:00007ffeb55d1f40 EFLAGS: 000000e4
[ 6870.370442] Mem-Info:
[ 6870.370461] active_anon:3357705 inactive_anon:24251 isolated_anon:0
[ 6870.370461] active_file:104 inactive_file:2 isolated_file:0
[ 6870.370461] unevictable:18392 dirty:0 writeback:2 unstable:0
[ 6870.370461] slab_reclaimable:10221 slab_unreclaimable:15302
[ 6870.370461] mapped:1245 shmem:1275 pagetables:10643 bounce:0
[ 6870.370461] free:30157 free_pcp:3654 free_cma:0
[ 6870.370523] Node 0 active_anon:13430820kB inactive_anon:97004kB
active_file:416kB inactive_file:8kB unevictable:73568kB isolated(anon):0kB
isolated(file):0kB mapped:4980kB dirty:0kB writeback:8kB shmem:5100kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
[ 6870.370569] Node 0 DMA free:15908kB min:76kB low:92kB high:108kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB
kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB
[ 6870.370621] lowmem_reserve[]: 0 3956 13655 13655 13655
[ 6870.370634] Node 0 DMA32 free:57708kB min:19556kB low:24444kB high:29332kB
active_anon:3865036kB inactive_anon:39792kB active_file:620kB
inactive_file:632kB unevictable:32436kB writepending:8kB present:4159452kB
managed:4071632kB mlocked:32436kB kernel_stack:2512kB pagetables:15532kB
bounce:0kB free_pcp:7576kB local_pcp:532kB free_cma:0kB
[ 6870.370686] lowmem_reserve[]: 0 0 9699 9699 9699
[ 6870.370699] Node 0 Normal free:47012kB min:47948kB low:59932kB high:71916kB
active_anon:9565784kB inactive_anon:57212kB active_file:1692kB
inactive_file:708kB unevictable:41132kB writepending:0kB present:10160128kB
managed:9915580kB mlocked:41132kB kernel_stack:3552kB pagetables:27040kB
bounce:0kB free_pcp:7040kB local_pcp:736kB free_cma:0kB
[ 6870.370753] lowmem_reserve[]: 0 0 0 0 0
[ 6870.370762] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB
(U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 6870.370798] Node 0 DMA32: 662*4kB (UME) 256*8kB (UME) 314*16kB (UME)
331*32kB (UME) 181*64kB (UME) 89*128kB (UE) 32*256kB (UME) 13*512kB (UME)
1*1024kB (E) 0*2048kB 0*4096kB = 59160kB
[ 6870.370834] Node 0 Normal: 461*4kB (UME) 28*8kB (UME) 55*16kB (UM) 1385*32kB
(UME) 5*64kB (M) 5*128kB (M) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
48996kB
[ 6870.370872] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=1048576kB
[ 6870.370894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB
[ 6870.370910] 1946 total pagecache pages
[ 6870.370918] 0 pages in swap cache
[ 6870.370926] Swap cache stats: add 3510, delete 3510, find 44/106
[ 6870.370937] Free swap = 0kB
[ 6870.370945] Total swap = 0kB
[ 6870.370952] 3583894 pages RAM
[ 6870.370959] 0 pages HighMem/MovableOnly
[ 6870.370966] 83114 pages reserved
[ 6870.370974] 0 pages cma reserved
[ 6870.370981] 0 pages hwpoisoned
[ 6870.370988] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents
oom_score_adj name
[ 6870.371035] [ 266] 0 266 27000 204 45 4 0
0 systemd-journal
[ 6870.371054] [ 282] 0 282 30805 100 34 3 0
0 qubesdb-daemon
[ 6870.371073] [ 326] 0 326 23697 686 46 3 0
-1000 systemd-udevd
[ 6870.371091] [ 432] 0 432 3042 781 11 3 0
0 haveged
[ 6870.371108] [ 433] 81 433 13278 231 30 3 0
-900 dbus-daemon
[ 6870.371125] [ 435] 0 435 19358 189 42 3 0
0 systemd-logind
[ 6870.371143] [ 438] 0 438 10243 73 23 3 0
0 meminfo-writer
[ 6870.371162] [ 450] 0 450 34209 116 38 3 0
0 xl
[ 6870.371176] [ 454] 0 454 18919 152 43 3 0
0 qubes-gui
[ 6870.371194] [ 455] 0 455 16536 106 36 3 0
0 qrexec-agent
[ 6870.371211] [ 503] 0 503 73994 189 50 3 0
0 su
[ 6870.371226] [ 519] 0 519 52863 27 10 3 0
0 agetty
[ 6870.570122] [ 520] 0 520 52775 29 11 3 0
0 agetty
[ 6870.570136] [ 521] 1000 521 21958 334 45 3 0
0 systemd
[ 6870.570150] [ 527] 1000 527 34755 598 64 3 0
0 (sd-pam)
[ 6870.570163] [ 570] 1000 570 54160 86 14 3 0
0 bash
[ 6870.570177] [ 627] 1000 627 3500 30 12 3 0
0 xinit
[ 6870.570189] [ 640] 1000 640 323803 26706 180 4 0
0 Xorg
[ 6870.570205] [ 663] 1000 663 53597 67 12 4 0
0 qubes-session
[ 6870.570218] [ 675] 1000 675 13197 181 31 3 0
0 dbus-daemon
[ 6870.570231] [ 691] 1000 691 7242 119 17 3 0
0 ssh-agent
[ 6870.570244] [ 709] 1000 709 16562 105 38 3 0
0 qrexec-client-v
[ 6870.570257] [ 725] 1000 725 48074 134 31 3 0
0 dconf-service
[ 6870.570270] [ 731] 1000 731 428388 2600 194 5 0
0 gsd-xsettings
[ 6870.570282] [ 732] 1000 732 62744 1790 29 3 0
0 icon-sender
[ 6870.570295] [ 734] 1000 734 122405 246 40 3 0
0 gnome-keyring-d
[ 6870.570308] [ 735] 1000 735 120207 121 37 3 0
0 agent
[ 6870.570320] [ 748] 1000 748 438372 2843 212 5 0
0 nm-applet
[ 6870.570333] [ 755] 1000 755 128956 339 91 4 0
0 pulseaudio
[ 6870.570346] [ 757] 172 757 47723 83 29 3 0
0 rtkit-daemon
[ 6870.570360] [ 768] 998 768 657135 1585 91 6 1
0 polkitd
[ 6870.570372] [ 771] 1000 771 16528 100 34 3 0
0 qrexec-fork-ser
[ 6870.570385] [ 774] 1000 774 52238 17 11 3 0
0 sleep
[ 6870.570397] [ 848] 1000 848 87396 143 38 4 0
0 at-spi-bus-laun
[ 6870.570410] [ 854] 1000 854 13134 120 31 3 0
0 dbus-daemon
[ 6870.570423] [ 858] 1000 858 56364 197 46 3 0
0 at-spi2-registr
[ 6870.570436] [ 868] 1000 868 123835 208 46 3 0
0 gvfsd
[ 6870.570448] [ 902] 1000 902 105683 143 41 4 0
0 gvfsd-fuse
[ 6870.570462] [ 921] 1000 921 213915 6618 145 4 0
0 gnome-terminal-
[ 6870.570474] [ 925] 1000 925 206356 367 66 4 0
0 xdg-desktop-por
[ 6870.570486] [ 930] 1000 930 157249 145 39 4 0
0 xdg-document-po
[ 6870.570499] [ 933] 1000 933 117667 117 33 3 0
0 xdg-permission-
[ 6870.570511] [ 943] 1000 943 193292 1109 110 3 0
0 xdg-desktop-por
[ 6870.570524] [ 951] 1000 951 54193 171 16 3 0
0 bash
[ 6870.570536] [ 1012] 0 1012 252781 46241 218 4 0
0 packagekitd
[ 6870.570549] [ 2375] 1000 2375 54193 165 14 3 0
0 bash
[ 6870.570562] [ 6264] 1000 6264 63294 229 30 3 0
0 top
[ 6870.570579] [ 6748] 1000 6748 54193 160 13 3 0
0 bash
[ 6870.570595] [10247] 1000 10247 54193 161 13 3 0
0 bash
[ 6870.570607] [30948] 1000 30948 54193 158 16 3 0
0 bash
[ 6870.570620] [ 664] 1000 664 54193 156 15 4 0
0 bash
[ 6870.570633] [ 3687] 1000 3687 54023 213 14 3 0
0 watch
[ 6870.570645] [24682] 1000 24682 65858 761 34 3 0
0 rpmbuild
[ 6870.570657] [24781] 1000 24781 53597 75 13 3 0
0 sh
[ 6870.570668] [24795] 1000 24795 67125 4750 66 3 0
0 python2.7
[ 6870.570680] [24810] 1000 24810 31461 5860 62 3 0
0 python2.7
[ 6870.570692] [24848] 1000 24848 9587 583 23 3 0
0 gmake
[ 6870.570706] [24851] 1000 24851 9516 507 20 3 0
0 gmake
[ 6870.570719] [24952] 1000 24952 9521 484 22 3 0
0 gmake
[ 6870.570731] [24956] 1000 24956 9588 614 23 3 0
0 gmake
[ 6870.570743] [24997] 1000 24997 9145 139 21 3 0
0 gmake
[ 6870.570755] [25002] 1000 25002 218452 5840 215 4 0
0 cargo
[ 6870.570767] [ 1886] 1000 1886 9202 234 22 3 0
0 gmake
[ 6870.570781] [ 2121] 1000 2121 9284 298 22 3 0
0 gmake
[ 6870.570794] [ 4048] 1000 4048 755277 605078 1338 6 0
0 rustc
[ 6870.570807] [ 4540] 1000 4540 3006 320 11 3 0
0 ccache
[ 6870.570819] [ 4613] 1000 4613 3003 318 12 3 0
0 g++
[ 6870.570831] [ 4645] 1000 4645 2305 24 9 3 0
0 g++
[ 6870.570844] [ 4646] 1000 4646 498700 469312 952 5 0
0 cc1plus
[ 6870.570856] [ 4647] 1000 4647 37091 33394 79 4 0
0 as
[ 6870.770910] [ 4846] 1000 4846 2775 222 12 3 0
0 ccache
[ 6870.770924] [ 4861] 1000 4861 2773 220 10 3 0
0 g++
[ 6870.770937] [ 4877] 1000 4877 2305 23 10 3 0
0 g++
[ 6870.770950] [ 4878] 1000 4878 309707 283062 591 4 0
0 cc1plus
[ 6870.770963] [ 4879] 1000 4879 18182 14491 41 4 0
0 as
[ 6870.770974] [ 4897] 1000 4897 2780 202 11 3 0
0 ccache
[ 6870.770987] [ 4924] 1000 4924 2777 198 10 3 0
0 g++
[ 6870.771014] [ 4937] 1000 4937 2305 23 9 3 0
0 g++
[ 6870.771027] [ 4938] 1000 4938 306512 279904 587 4 0
0 cc1plus
[ 6870.771040] [ 4939] 1000 4939 9404 5674 23 3 0
0 as
[ 6870.771051] [ 4960] 1000 4960 2780 185 10 4 0
0 ccache
[ 6870.771064] [ 4967] 1000 4967 2814 200 11 3 0
0 ccache
[ 6870.771077] [ 4974] 1000 4974 2786 216 11 3 0
0 ccache
[ 6870.771090] [ 4979] 1000 4979 2813 184 11 3 0
0 ccache
[ 6870.771103] [ 4990] 1000 4990 2777 183 12 3 0
0 g++
[ 6870.771115] [ 4997] 1000 4997 2774 194 11 3 0
0 ccache
[ 6870.771128] [ 5006] 1000 5006 2812 187 10 3 0
0 g++
[ 6870.771141] [ 5009] 1000 5009 2813 201 10 3 0
0 g++
[ 6870.771154] [ 5012] 1000 5012 2785 216 11 3 0
0 g++
[ 6870.771166] [ 5015] 1000 5015 2305 22 9 3 0
0 g++
[ 6870.771179] [ 5016] 1000 5016 274112 247749 523 4 0
0 cc1plus
[ 6870.771192] [ 5017] 1000 5017 4982 1249 15 3 0
0 as
[ 6870.771203] [ 5020] 1000 5020 2772 191 10 3 0
0 g++
[ 6870.771216] [ 5023] 1000 5023 2305 23 10 3 0
0 g++
[ 6870.771230] [ 5024] 1000 5024 260164 241258 507 4 0
0 cc1plus
[ 6870.771244] [ 5025] 1000 5025 5511 1776 17 3 0
0 as
[ 6870.771256] [ 5026] 1000 5026 2305 23 9 3 0
0 g++
[ 6870.771268] [ 5027] 1000 5027 265514 241877 506 4 0
0 cc1plus
[ 6870.771280] [ 5028] 1000 5028 4983 1248 15 3 0
0 as
[ 6870.771294] [ 5031] 1000 5031 2305 23 8 3 0
0 g++
[ 6870.771306] [ 5032] 1000 5032 274929 251196 524 4 0
0 cc1plus
[ 6870.771318] [ 5033] 1000 5033 4983 1251 16 3 0
0 as
[ 6870.771329] [ 5034] 1000 5034 2305 24 9 3 0
0 g++
[ 6870.771341] [ 5035] 1000 5035 248405 222601 471 4 0
0 cc1plus
[ 6870.771354] [ 5036] 1000 5036 4982 1187 14 3 0
0 as
[ 6870.771364] [ 5051] 1000 5051 9145 136 21 3 0
0 gmake
[ 6870.771376] [ 5056] 1000 5056 2801 198 11 3 0
0 ccache
[ 6870.771389] [ 5066] 1000 5066 2799 196 10 3 0
0 g++
[ 6870.771401] [ 5071] 1000 5071 2795 175 10 3 0
0 ccache
[ 6870.771413] [ 5076] 1000 5076 2305 23 9 3 0
0 g++
[ 6870.771425] [ 5077] 1000 5077 175559 158120 341 3 0
0 cc1plus
[ 6870.771437] [ 5078] 1000 5078 4981 1252 17 3 0
0 as
[ 6870.771448] [ 5081] 1000 5081 2793 174 11 3 0
0 g++
[ 6870.771462] [ 5086] 1000 5086 2812 211 9 3 0
0 ccache
[ 6870.771474] [ 5091] 1000 5091 2305 23 10 3 0
0 g++
[ 6870.771486] [ 5092] 1000 5092 146716 125096 278 3 0
0 cc1plus
[ 6870.771498] [ 5093] 1000 5093 4982 1248 15 3 0
0 as
[ 6870.771508] [ 5096] 1000 5096 2811 210 11 3 0
0 g++
[ 6870.971546] [ 5101] 1000 5101 2305 23 9 3 0
0 g++
[ 6870.971560] [ 5102] 1000 5102 107775 89047 209 3 0
0 cc1plus
[ 6870.971573] [ 5103] 1000 5103 4983 1254 13 3 0
0 as
[ 6870.971584] [ 5118] 0 5118 7366 35 16 3 0
0 systemd-coredum
[ 6870.971597] [ 5120] 1000 5120 54023 213 11 3 0
0 watch
[ 6870.971610] [ 5121] 1000 5121 1465 16 7 3 0
0 sh
[ 6870.971620] Out of memory: Kill process 4048 (rustc) score 173 or sacrifice
child
[ 6870.971643] Killed process 4048 (rustc) total-vm:3021108kB,
anon-rss:2420312kB, file-rss:8kB, shmem-rss:0kB
[ 6871.117954] audit: type=1701 audit(1534438842.541:171): auid=4294967295
uid=0 gid=0 ses=4294967295 pid=266 comm="systemd-journal"
exe="/usr/lib/systemd/systemd-journald" sig=6 res=1
[ 6871.322611] systemd[1]: Created slice system-systemd\x2dcoredump.slice.
[ 6871.338365] systemd[1]: Started Process Core Dump (PID 5118/UID 0).
[ 6871.340483] audit: type=1130 audit(1534438842.764:172): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@0-5118-0
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=success'
[ 6871.357122] systemd[1]: systemd-journald.service: Main process exited,
code=dumped, status=6/ABRT
[ 6871.357205] systemd[1]: systemd-journald.service: Failed with result
'watchdog'.
[ 6871.357751] audit: type=1130 audit(1534438842.781:173): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 6871.358161] systemd[1]: systemd-journald.service: Service has no hold-off
time, scheduling restart.
[ 6871.358248] systemd[1]: systemd-journald.service: Scheduled restart job,
restart counter is at 1.
[ 6871.358919] systemd[1]: Stopped Flush Journal to Persistent Storage.
[ 6871.358988] systemd[1]: Stopping Flush Journal to Persistent Storage...
[ 6871.358999] audit: type=1130 audit(1534438842.782:174): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359002] audit: type=1131 audit(1534438842.782:175): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359096] systemd[1]: Stopped Journal Service.
[ 6871.359138] audit: type=1130 audit(1534438842.783:176): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359177] audit: type=1131 audit(1534438842.783:177): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.364043] systemd[1]: Starting Journal Service...
[ 6871.396019] audit: type=1305 audit(1534438842.819:178): audit_enabled=1
old=1 auid=4294967295 ses=4294967295 res=1
[ 6871.418446] systemd-journald[5125]: File
/var/log/journal/12c5e521101c444594b96b53751551a8/system.journal corrupted or
uncleanly shut down, renaming and replacing.
[ 6871.431865] systemd-coredump[5122]: MESSAGE=Process 266 (systemd-journal) of
user 0 dumped core.
[ 6871.431884] systemd-coredump[5122]: Coredump diverted to
/var/lib/systemd/coredump/core.systemd-journal.0.f4dc34a546c24e2aa7f99cdcd659015f.266.1534438842000000.lz4
[ 6871.431902] systemd-coredump[5122]: Stack trace of thread 266:
[ 6871.431912] systemd-coredump[5122]: #0 0x00007db8401c3ae4
journal_file_find_data_object_with_hash (libsystemd-shared-238.so)
[ 6871.431927] systemd-coredump[5122]: #1 0x00007db8401c3d59
journal_file_append_data (libsystemd-shared-238.so)
[ 6871.431941] systemd-coredump[5122]: #2 0x00007db8401c60d1
journal_file_append_entry (libsystemd-shared-238.so)
[ 6871.431955] systemd-coredump[5122]: #3 0x0000620290cfd6c8
dispatch_message_real (systemd-journald)
[ 6871.431967] systemd-coredump[5122]: #4 0x0000620290cf95e5
server_read_dev_kmsg (systemd-journald)
[ 6871.431980] systemd-coredump[5122]: #5 0x00007db8401b6cb8 source_dispatch
(libsystemd-shared-238.so)
[ 6871.431992] systemd-coredump[5122]: #6 0x00007db8401b848c sd_event_dispatch
(libsystemd-shared-238.so)
[ 6871.435899] systemd-coredum: 4 output lines suppressed due to ratelimiting
[ 6871.485869] audit: type=1130 audit(1534438842.909:179): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 6871.486357] audit: type=1130 audit(1534438842.910:180): pid=1 uid=0
auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
So it looks like it ran out of memory and killed rustc, OR it was a side effect
of qube Pause/Resume ?
10:28.79 Compiling style_traits v0.0.1
(file:///home/user/rpmbuild/BUILD/firefox-61.0.2/servo/components/style_traits)
...
13:09.57
/home/user/rpmbuild/BUILD/firefox-61.0.2/dom/crypto/CryptoKey.cpp:39:3: note:
in expansion of macro 'NS_INTERFACE_MAP_ENTRY'
13:09.57 NS_INTERFACE_MAP_ENTRY(nsISupports)
13:09.57 ^~~~~~~~~~~~~~~~~~~~~~
21:52.81 error: Could not compile `style`.
21:52.82 To learn more, run the command again with --verbose.
21:52.87 gmake[4]: ***
[/home/user/rpmbuild/BUILD/firefox-61.0.2/config/rules.mk:950:
force-cargo-library-build] Error 101
21:52.89 gmake[3]: ***
[/home/user/rpmbuild/BUILD/firefox-61.0.2/config/recurse.mk:73:
toolkit/library/rust/target] Error 2
21:52.89 gmake[3]: *** Waiting for unfinished jobs....
Either way, I can't explain the constant reading speed of 192MiB/sec (according
to dom0 xfce-panel plugin Disk Performance Monitor for /dev/sda while the qube
was running)
$ cat /proc/meminfo
MemTotal: 7454500 kB
MemFree: 5635088 kB
MemAvailable: 6574676 kB
Buffers: 53832 kB
Cached: 1094368 kB
SwapCached: 0 kB
Active: 724832 kB
Inactive: 747696 kB
Active(anon): 233816 kB
Inactive(anon): 95768 kB
Active(file): 491016 kB
Inactive(file): 651928 kB
Unevictable: 73568 kB
Mlocked: 73568 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 292 kB
Writeback: 0 kB
AnonPages: 398016 kB
Mapped: 54320 kB
Shmem: 5256 kB
Slab: 134680 kB
SReclaimable: 74124 kB
SUnreclaim: 60556 kB
KernelStack: 4800 kB
PageTables: 10524 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3727248 kB
Committed_AS: 1332236 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 327644 kB
DirectMap2M: 14008320 kB
DirectMap1G: 0 kB
Hmm, mem total is 7.4Gig, it's set to 1400MB initial, and 14000MB max with
"Include in memory balancing" enabled.
I'm kinda wondering if the OOM is the one that caused the huge disk reading,
even though it doesn't make any sense...
--
You received this message because you are subscribed to the Google Groups
"qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to qubes-users+unsubscr...@googlegroups.com.
To post to this group, send email to qubes-users@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/qubes-users/5f5e6148-8256-4c2a-808f-c9f35c2cf7fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
