Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
On 2018年07月18日 08:24, Marc MERLIN wrote: > On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: >> No OOM triggers? That's a little strange. >> Maybe it's related to how kernel handles memory over-commit? > > Yes, I think you are correct. > >> And for the hang, I think it's related to some memory allocation failure >> and error handler just didn't handle it well, so it's causing deadlock >> for certain page. > > That indeed matches what I'm seeing. > >> ENOMEM handling is pretty common but hardly verified, so it's not that >> strange, but we must locate the problem. > > I seem to be getting deadlocks in the kernel, so I'm hoping that at least > it's checked there, but maybe not? > >> In my system, at least I'm not using btrfs as root fs, and for the >> memory eating program I normally ensure it's eating all the memory + >> swap, so OOM killer is always triggered, maybe that's the cause. >> >> So in your case, maybe it's btrfs not really taking up all memory, thus >> OOM killer not triggered. > > Correct, the swap is not used. > >> Any kernel dmesg about OOM killer triggered? > > Nothing at all. It never gets triggered. > >>> Here is my system when it virtually died: >>> ER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND >>> root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs >>> check /dev/mapper/dshelf2 > > See how btrs was taking 29GB in that ps output (that's before it takes > everything and I can't even type ps anymore) > Note that VSZ is almost equal to RSS. Nothing gets swapped. > > Then see free output: > >>> total used free sharedbuffers cached >>> Mem: 32643788 32180100 463688 0 44664 119508 >>> -/+ buffers/cache: 32015928 627860 >>> Swap: 15616764 443676 15173088 >> >> For swap, it looks like only some other program's memory is swapped out, >> not btrfs'. > > That's exactly correct. btrfs check never goes to swap, I'm not sure why, > and because there is virtual memory free, maybe that's why OOM does not > trigger? > So I guess I can probably "fix" my problem by removing swap, but ultimately > it would be useful to know why memory taken by btrfs check does not end up > in swap. > >> And unfortunately, I'm not so familiar with OOM/MM code outside of >> filesystem. >> Any help from other experienced developers would definitely help to >> solve why memory of 'btrfs check' is not swapped out or why OOM killer >> is not triggered. > > Do you have someone from linux-vm you might be able to ask, or should we Cc > this thread there? Michal Hocho gives me a brief session about this. Which is super helpful in this case, thank you Michal! Firstly, btrfs-progs usage of malloc() will result anonymous pages, thus they can be swapped out. Secondly, kernel doesn't like to swap out anonymous pages at all, thus kernel won't try to aggressively swap out such pages. Thirdly, for user anonymous memory, there is LRU-like algorithm to determine which memory should go swapped out. But considering how btrfs check uses pages, it would only make it harder to be swapped out. So it's not an easy way thing to aggressively swap out memory of btrfs check to swap. Although Michal mentioned some cgroup way to limit the memory usage so it can be more aggressively swapped out, I'm still digging into it. Thanks, Qu > > Thanks, > Marc > signature.asc Description: OpenPGP digital signature
Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
On Wed, Jul 18, 2018 at 10:42:21PM +0300, Andrei Borzenkov wrote: > > Any help from other experienced developers would definitely help to > > solve why memory of 'btrfs check' is not swapped out or why OOM killer > > is not triggered. > > Almost all used memory is marked as "active" and active pages are not > swapped. Page is active if it was accessed recently. Is it possible that > btrfs logic does frequent scans across all allocated memory? > >> > >> Active: 30381404 kB > >> Inactive: 585952 kB That is a very good find. Yes, the linux kernel VM may be smart enough not to swap pages that got used recently and when btrfs slurps all the extents to cross check everything, I think it does cross reference them all many times. This is why it can run in a few hours when btrfs check lowmem requires days to run in a similar situation. I'm not sure if there is a good way around this, but it's good to know that btrfs repair can effectively abuse the linux VM in a way that it'll take everything down without OOM having a chance to trigger. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
18.07.2018 03:05, Qu Wenruo пишет: > > > On 2018年07月18日 04:59, Marc MERLIN wrote: >> Ok, I did more testing. Qu is right that btrfs check does not crash the >> kernel. >> It just takes all the memory until linux hangs everywhere, and somehow (no >> idea why) >> the OOM killer never triggers. > > No OOM triggers? That's a little strange. > Maybe it's related to how kernel handles memory over-commit? > > And for the hang, I think it's related to some memory allocation failure > and error handler just didn't handle it well, so it's causing deadlock > for certain page. > > ENOMEM handling is pretty common but hardly verified, so it's not that > strange, but we must locate the problem. > >> Details below: >> >> On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: >>> Here is what I got when the system was not doing well (it took minutes to >>> run): >>> >>> total used free sharedbuffers cached >>> Mem: 32643788 32070952 572836 0 1021604378772 >>> -/+ buffers/cache: 275900205053768 >>> Swap: 15616764 973596 14643168 >> >> ok, the reason it was not that close to 0 was due to /dev/shm it seems. >> I cleared that, and now I can get it to go to near 0 again. >> I'm wrong about the system being fully crashed, it's not, it's just very >> close to being hung. >> I can type killall -9 btrfs in the serial console and wait a few minutes. >> The system eventually recovers, but it's impossible to fix anything via ssh >> apparently because networking does not get to run when I'm in this state. >> >> I'm not sure why my system reproduces this easy while Qu's system does not, >> but Qu was right that the kernel is not dead and that it's merely a problem >> of userspace >> taking all the RAM and somehow not being killed by OOM > > In my system, at least I'm not using btrfs as root fs, and for the > memory eating program I normally ensure it's eating all the memory + > swap, so OOM killer is always triggered, maybe that's the cause. > > So in your case, maybe it's btrfs not really taking up all memory, thus > OOM killer not triggered. > >> >> I checked the PID and don't see why it's not being killed: >> gargamel:/proc/31006# grep . oom* >> oom_adj:0 >> oom_score:221 << this increases a lot, but OOM never kills it >> oom_score_adj:0 >> >> I have these variables: >> /proc/sys/vm/oom_dump_tasks:1 >> /proc/sys/vm/oom_kill_allocating_task:0 >> /proc/sys/vm/overcommit_kbytes:0 >> /proc/sys/vm/overcommit_memory:0 >> /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) > > Any kernel dmesg about OOM killer triggered? > >> >> Here is my system when it virtually died: >> ER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND >> root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs >> check /dev/mapper/dshelf2 >> >> total used free sharedbuffers cached >> Mem: 32643788 32180100 463688 0 44664 119508 >> -/+ buffers/cache: 32015928 627860 >> Swap: 15616764 443676 15173088 > > For swap, it looks like only some other program's memory is swapped out, > not btrfs'. > > And unfortunately, I'm not so familiar with OOM/MM code outside of > filesystem. > Any help from other experienced developers would definitely help to > solve why memory of 'btrfs check' is not swapped out or why OOM killer > is not triggered. > Almost all used memory is marked as "active" and active pages are not swapped. Page is active if it was accessed recently. Is it possible that btrfs logic does frequent scans across all allocated memory? > Thanks, > Qu > >> >> MemTotal: 32643788 kB >> MemFree: 463440 kB >> MemAvailable: 44864 kB >> Buffers: 44664 kB >> Cached: 120360 kB >> SwapCached:87064 kB >> Active: 30381404 kB >> Inactive: 585952 kB >> Active(anon): 30334696 kB >> Inactive(anon): 474624 kB >> Active(file): 46708 kB >> Inactive(file): 111328 kB >> Unevictable:5616 kB >> Mlocked:5616 kB >> SwapTotal: 15616764 kB >> SwapFree: 15173088 kB >> Dirty: 1636 kB >> Writeback: 4 kB >> AnonPages: 30734240 kB >> Mapped:67236 kB >> Shmem: 3036 kB >> Slab: 267884 kB >> SReclaimable: 51528 kB >> SUnreclaim: 216356 kB >> KernelStack: 10144 kB >> PageTables:69284 kB >> NFS_Unstable: 0 kB >> Bounce:0 kB >> WritebackTmp: 0 kB >> CommitLimit:31938656 kB >> Committed_AS: 32865492 kB >> VmallocTotal: 34359738367 kB >> VmallocUsed: 0 kB >> VmallocChunk: 0 kB >> HardwareCorrupted: 0 kB >> AnonHugePages: 0 kB >> ShmemHugePages:0 kB >> ShmemPmdMapped:0 kB >> CmaTotal: 16384 kB >> CmaFree: 0 kB >> HugePages_Total: 0 >> HugePages_Free:
Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: > No OOM triggers? That's a little strange. > Maybe it's related to how kernel handles memory over-commit? Yes, I think you are correct. > And for the hang, I think it's related to some memory allocation failure > and error handler just didn't handle it well, so it's causing deadlock > for certain page. That indeed matches what I'm seeing. > ENOMEM handling is pretty common but hardly verified, so it's not that > strange, but we must locate the problem. I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not? > In my system, at least I'm not using btrfs as root fs, and for the > memory eating program I normally ensure it's eating all the memory + > swap, so OOM killer is always triggered, maybe that's the cause. > > So in your case, maybe it's btrfs not really taking up all memory, thus > OOM killer not triggered. Correct, the swap is not used. > Any kernel dmesg about OOM killer triggered? Nothing at all. It never gets triggered. > > Here is my system when it virtually died: > > ER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND > > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs > > check /dev/mapper/dshelf2 See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore) Note that VSZ is almost equal to RSS. Nothing gets swapped. Then see free output: > > total used free sharedbuffers cached > > Mem: 32643788 32180100 463688 0 44664 119508 > > -/+ buffers/cache: 32015928 627860 > > Swap: 15616764 443676 15173088 > > For swap, it looks like only some other program's memory is swapped out, > not btrfs'. That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger? So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap. > And unfortunately, I'm not so familiar with OOM/MM code outside of > filesystem. > Any help from other experienced developers would definitely help to > solve why memory of 'btrfs check' is not swapped out or why OOM killer > is not triggered. Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
On 2018年07月18日 04:59, Marc MERLIN wrote: > Ok, I did more testing. Qu is right that btrfs check does not crash the > kernel. > It just takes all the memory until linux hangs everywhere, and somehow (no > idea why) > the OOM killer never triggers. No OOM triggers? That's a little strange. Maybe it's related to how kernel handles memory over-commit? And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page. ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem. > Details below: > > On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: >> Here is what I got when the system was not doing well (it took minutes to >> run): >> >> total used free sharedbuffers cached >> Mem: 32643788 32070952 572836 0 1021604378772 >> -/+ buffers/cache: 275900205053768 >> Swap: 15616764 973596 14643168 > > ok, the reason it was not that close to 0 was due to /dev/shm it seems. > I cleared that, and now I can get it to go to near 0 again. > I'm wrong about the system being fully crashed, it's not, it's just very > close to being hung. > I can type killall -9 btrfs in the serial console and wait a few minutes. > The system eventually recovers, but it's impossible to fix anything via ssh > apparently because networking does not get to run when I'm in this state. > > I'm not sure why my system reproduces this easy while Qu's system does not, > but Qu was right that the kernel is not dead and that it's merely a problem > of userspace > taking all the RAM and somehow not being killed by OOM In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause. So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered. > > I checked the PID and don't see why it's not being killed: > gargamel:/proc/31006# grep . oom* > oom_adj:0 > oom_score:221 << this increases a lot, but OOM never kills it > oom_score_adj:0 > > I have these variables: > /proc/sys/vm/oom_dump_tasks:1 > /proc/sys/vm/oom_kill_allocating_task:0 > /proc/sys/vm/overcommit_kbytes:0 > /proc/sys/vm/overcommit_memory:0 > /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) Any kernel dmesg about OOM killer triggered? > > Here is my system when it virtually died: > ER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs > check /dev/mapper/dshelf2 > > total used free sharedbuffers cached > Mem: 32643788 32180100 463688 0 44664 119508 > -/+ buffers/cache: 32015928 627860 > Swap: 15616764 443676 15173088 For swap, it looks like only some other program's memory is swapped out, not btrfs'. And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem. Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered. Thanks, Qu > > MemTotal: 32643788 kB > MemFree: 463440 kB > MemAvailable: 44864 kB > Buffers: 44664 kB > Cached: 120360 kB > SwapCached:87064 kB > Active: 30381404 kB > Inactive: 585952 kB > Active(anon): 30334696 kB > Inactive(anon): 474624 kB > Active(file): 46708 kB > Inactive(file): 111328 kB > Unevictable:5616 kB > Mlocked:5616 kB > SwapTotal: 15616764 kB > SwapFree: 15173088 kB > Dirty: 1636 kB > Writeback: 4 kB > AnonPages: 30734240 kB > Mapped:67236 kB > Shmem: 3036 kB > Slab: 267884 kB > SReclaimable: 51528 kB > SUnreclaim: 216356 kB > KernelStack: 10144 kB > PageTables:69284 kB > NFS_Unstable: 0 kB > Bounce:0 kB > WritebackTmp: 0 kB > CommitLimit:31938656 kB > Committed_AS: 32865492 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > HardwareCorrupted: 0 kB > AnonHugePages: 0 kB > ShmemHugePages:0 kB > ShmemPmdMapped:0 kB > CmaTotal: 16384 kB > CmaFree: 0 kB > HugePages_Total: 0 > HugePages_Free:0 > HugePages_Rsvd:0 > HugePages_Surp:0 > Hugepagesize: 2048 kB > Hugetlb: 0 kB > DirectMap4k: 560404 kB > DirectMap2M:32692224 kB > > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)
Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers. Details below: On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: > Here is what I got when the system was not doing well (it took minutes to > run): > > total used free sharedbuffers cached > Mem: 32643788 32070952 572836 0 1021604378772 > -/+ buffers/cache: 275900205053768 > Swap: 15616764 973596 14643168 ok, the reason it was not that close to 0 was due to /dev/shm it seems. I cleared that, and now I can get it to go to near 0 again. I'm wrong about the system being fully crashed, it's not, it's just very close to being hung. I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state. I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM I checked the PID and don't see why it's not being killed: gargamel:/proc/31006# grep . oom* oom_adj:0 oom_score:221 << this increases a lot, but OOM never kills it oom_score_adj:0 I have these variables: /proc/sys/vm/oom_dump_tasks:1 /proc/sys/vm/oom_kill_allocating_task:0 /proc/sys/vm/overcommit_kbytes:0 /proc/sys/vm/overcommit_memory:0 /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) Here is my system when it virtually died: ER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 total used free sharedbuffers cached Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860 Swap: 15616764 443676 15173088 MemTotal: 32643788 kB MemFree: 463440 kB MemAvailable: 44864 kB Buffers: 44664 kB Cached: 120360 kB SwapCached:87064 kB Active: 30381404 kB Inactive: 585952 kB Active(anon): 30334696 kB Inactive(anon): 474624 kB Active(file): 46708 kB Inactive(file): 111328 kB Unevictable:5616 kB Mlocked:5616 kB SwapTotal: 15616764 kB SwapFree: 15173088 kB Dirty: 1636 kB Writeback: 4 kB AnonPages: 30734240 kB Mapped:67236 kB Shmem: 3036 kB Slab: 267884 kB SReclaimable: 51528 kB SUnreclaim: 216356 kB KernelStack: 10144 kB PageTables:69284 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit:31938656 kB Committed_AS: 32865492 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages:0 kB ShmemPmdMapped:0 kB CmaTotal: 16384 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 560404 kB DirectMap2M:32692224 kB -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs check (not lowmem) and OOM-like hangs (4.17.6)
On Tue, Jul 17, 2018 at 10:50:32AM -0700, Marc MERLIN wrote: > I got the following on 4.17.6 while running btrfs check --repair on an > unmounted filesystem (not the lowmem version) > > I understand that btrfs check is userland only, although it seems that > it caused these FS hangs on a different filesystem (the trace of course > does not provide info on which FS) > > Any idea what happened here? > I'm going to wait a few hours without running btrfs check to see if it > happens again and then if running btrfs check will re-create this issue, > but other suggestions (if any), are welcome: Hi Qu, I know we were talking about this last week and then, btrfs check just worked for me so I wasn't able to reproduce. Now I'm able to reproduce again. I tried again, it's definitely triggered by btrfs check --repair I tried to capture what happens, and memory didn't dip to 0, but the system got very slow and things started failing. btrfs was never killed though while ssh was. Is there a chance that maybe btrfs is in some kernel OOM exclude list? Here is what I got when the system was not doing well (it took minutes to run): total used free sharedbuffers cached Mem: 32643788 32070952 572836 0 1021604378772 -/+ buffers/cache: 275900205053768 Swap: 15616764 973596 14643168 gargamel:~# cat /proc/meminfo MemTotal: 32643788 kB MemFree: 2726276 kB MemAvailable:2502200 kB Buffers: 12360 kB Cached: 1676388 kB SwapCached: 11048580 kB Active: 16443004 kB Inactive: 12010456 kB Active(anon): 16287780 kB Inactive(anon): 11651692 kB Active(file): 155224 kB Inactive(file): 358764 kB Unevictable:5776 kB Mlocked:5776 kB SwapTotal: 15616764 kB SwapFree: 294592 kB Dirty: 3032 kB Writeback: 76064 kB AnonPages: 15723272 kB Mapped: 612124 kB Shmem: 1171032 kB Slab: 399824 kB SReclaimable: 84568 kB SUnreclaim: 315256 kB KernelStack: 20576 kB PageTables:94268 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit:31938656 kB Committed_AS: 37909452 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 98304 kB ShmemHugePages:0 kB ShmemPmdMapped:0 kB CmaTotal: 16384 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 355604 kB DirectMap2M:32897024 kB and console: [ 9184.345329] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds. [ 9184.366258] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4 [ 9184.385323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9184.408803] zmtrigger.plD0 9981 9804 0x20020080 [ 9184.425249] Call Trace: [ 9184.432580] ? __schedule+0x53e/0x59b [ 9184.443551] schedule+0x7f/0x98 [ 9184.452960] io_schedule+0x16/0x38 [ 9184.463154] wait_on_page_bit_common+0x10c/0x199 [ 9184.476996] ? file_check_and_advance_wb_err+0xd7/0xd7 [ 9184.493339] shmem_getpage_gfp+0x2dd/0x975 [ 9184.506558] shmem_fault+0x188/0x1c3 [ 9184.518199] ? filemap_map_pages+0x6f/0x295 [ 9184.531680] __do_fault+0x1d/0x6e [ 9184.542505] __handle_mm_fault+0x675/0xa61 [ 9184.555653] ? list_move+0x21/0x3a [ 9184.566737] handle_mm_fault+0x11c/0x16b [ 9184.579355] __do_page_fault+0x324/0x41c [ 9184.591996] ? page_fault+0x8/0x30 [ 9184.603059] page_fault+0x1e/0x30 [ 9184.613846] RIP: 0023:0xf7d2d022 [ 9184.624366] RSP: 002b:ffeb9fe8 EFLAGS: 00010202 [ 9184.640868] RAX: f7eed000 RBX: 567e6000 RCX: 0004 [ 9184.663095] RDX: 587fecb0 RSI: 5876538c RDI: 0004 [ 9184.685308] RBP: 58185160 R08: R09: [ 9184.707524] R10: R11: 0286 R12: [ 9184.729757] R13: R14: R15: [ 9184.751988] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds. [ 9184.775106] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4 [ 9184.795072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9184.819423] /usr/sbin/apach D0 11868 11311 0x20020080 [ 9184.836748] Call Trace: [ 9184.844926] ? __schedule+0x53e/0x59b [ 9184.856811] schedule+0x7f/0x98 [ 9184.867075] io_schedule+0x16/0x38 [ 9184.878114] wait_on_page_bit_common+0x10c/0x199 [ 9184.892807] ? file_check_and_advance_wb_err+0xd7/0xd7 [ 9184.909036] shmem_getpage_gfp+0x2dd/0x975 [ 9184.922157] shmem_fault+0x188/0x1c3 [ 9184.933667] ? filemap_map_pages+0x6f/0x295 [ 9184.947504] __do_fault+0x1d/0x6e [ 9184.958234] __handle_mm_fault+0x675/0xa61 [
