Hello everybody. This night server stopped responding for some time. About 3 hours. During this period of time hdd became completely full (there was about 1Tb oа free space) and 90 of 230 (aprox) processes became uninterruptiblle . And suddenly space was freed and everything became stable again. Here what was in dmesg.
[9417724.085399] INFO: task exim4:23580 blocked for more than 120 seconds. [9417724.085403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085406] exim4 D 0000000000000004 0 23580 6911 0x00000000 [9417724.085412] ffff88000d9a1de8 0000000000000086 ffff8807ea54c500 0000000000000041 [9417724.085418] ffff88000d9a1fd8 ffff88000d9a1fd8 ffff88000d9a1fd8 00000000000137c0 [9417724.085422] ffff8807e7c22e00 ffff8807ea54c500 0000000000000000 ffff8806acde0f10 [9417724.085427] Call Trace: [9417724.085437] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085442] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085447] [<ffffffff81187361>] ? do_path_lookup+0x31/0xc0 [9417724.085450] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085454] [<ffffffff8118774e>] do_unlinkat+0x8e/0x1d0 [9417724.085459] [<ffffffff81178290>] ? vfs_write+0x110/0x180 [9417724.085462] [<ffffffff8117855a>] ? sys_write+0x4a/0x90 [9417724.085466] [<ffffffff811883e6>] sys_unlink+0x16/0x20 [9417724.085470] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9417724.085473] INFO: task exim4:23608 blocked for more than 120 seconds. [9417724.085475] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085478] exim4 D ffffffff81806200 0 23608 6911 0x00000000 [9417724.085482] ffff88006d161c78 0000000000000086 ffff8807e49f37b0 ffff88006d161e38 [9417724.085486] ffff88006d161fd8 ffff88006d161fd8 ffff88006d161fd8 00000000000137c0 [9417724.085491] ffffffff81c0d020 ffff880220d6ae00 0000000000000081 ffff8806acde0f10 [9417724.085495] Call Trace: [9417724.085499] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085502] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085506] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085509] [<ffffffff811862b4>] do_last+0x2b4/0x730 [9417724.085513] [<ffffffff81187ad1>] path_openat+0xd1/0x3f0 [9417724.085516] [<ffffffff81183415>] ? putname+0x35/0x50 [9417724.085520] [<ffffffff81187f12>] do_filp_open+0x42/0xa0 [9417724.085526] [<ffffffff81319811>] ? strncpy_from_user+0x31/0x40 [9417724.085529] [<ffffffff8118325a>] ? do_getname+0x10a/0x180 [9417724.085533] [<ffffffff8165b00e>] ? _raw_spin_lock+0xe/0x20 [9417724.085538] [<ffffffff81195227>] ? alloc_fd+0xf7/0x150 [9417724.085541] [<ffffffff81177538>] do_sys_open+0xf8/0x240 [9417724.085544] [<ffffffff811776a0>] sys_open+0x20/0x30 [9417724.085548] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9417724.085551] INFO: task exim4:23710 blocked for more than 120 seconds. [9417724.085553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085555] exim4 D ffffffff81806200 0 23710 6911 0x00000000 [9417724.085559] ffff880005f87c78 0000000000000086 ffff8807e49f37b0 ffff880005f87e38 [9417724.085563] ffff880005f87fd8 ffff880005f87fd8 ffff880005f87fd8 00000000000137c0 [9417724.085567] ffff8807ea099700 ffff88000d419700 0000000000000081 ffff8806acde0f10 [9417724.085572] Call Trace: [9417724.085576] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085579] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085582] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085586] [<ffffffff811862b4>] do_last+0x2b4/0x730 [9417724.085589] [<ffffffff81187ad1>] path_openat+0xd1/0x3f0 [9417724.085593] [<ffffffff81183415>] ? putname+0x35/0x50 [9417724.085596] [<ffffffff81187f12>] do_filp_open+0x42/0xa0 [9417724.085601] [<ffffffff81319811>] ? strncpy_from_user+0x31/0x40 [9417724.085604] [<ffffffff8118325a>] ? do_getname+0x10a/0x180 [9417724.085607] [<ffffffff8165b00e>] ? _raw_spin_lock+0xe/0x20 [9417724.085611] [<ffffffff81195227>] ? alloc_fd+0xf7/0x150 [9417724.085614] [<ffffffff81177538>] do_sys_open+0xf8/0x240 [9417724.085617] [<ffffffff811776a0>] sys_open+0x20/0x30 [9417724.085621] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9417724.085624] INFO: task exim4:23741 blocked for more than 120 seconds. [9417724.085626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085628] exim4 D ffffffff81806200 0 23741 6911 0x00000000 [9417724.085631] ffff8800b3df1c78 0000000000000082 ffff8807e49f37b0 ffff8800b3df1e38 [9417724.085636] ffff8800b3df1fd8 ffff8800b3df1fd8 ffff8800b3df1fd8 00000000000137c0 [9417724.085640] ffff8807ea5e9700 ffff8806d37c4500 0000000000000081 ffff8806acde0f10 [9417724.085644] Call Trace: [9417724.085648] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085651] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085655] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085658] [<ffffffff811862b4>] do_last+0x2b4/0x730 [9417724.085662] [<ffffffff81187ad1>] path_openat+0xd1/0x3f0 [9417724.085665] [<ffffffff81183415>] ? putname+0x35/0x50 [9417724.085669] [<ffffffff81187f12>] do_filp_open+0x42/0xa0 [9417724.085673] [<ffffffff81319811>] ? strncpy_from_user+0x31/0x40 [9417724.085676] [<ffffffff8118325a>] ? do_getname+0x10a/0x180 [9417724.085680] [<ffffffff8165b00e>] ? _raw_spin_lock+0xe/0x20 [9417724.085683] [<ffffffff81195227>] ? alloc_fd+0xf7/0x150 [9417724.085686] [<ffffffff81177538>] do_sys_open+0xf8/0x240 [9417724.085690] [<ffffffff811776a0>] sys_open+0x20/0x30 [9417724.085693] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9417724.085696] INFO: task exim4:23742 blocked for more than 120 seconds. [9417724.085698] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085700] exim4 D ffffffff81806200 0 23742 6911 0x00000000 [9417724.085704] ffff880010d0bc78 0000000000000086 ffff8807e49f37b0 ffff880010d0be38 [9417724.085708] ffff880010d0bfd8 ffff880010d0bfd8 ffff880010d0bfd8 00000000000137c0 [9417724.085712] ffff8807ea001700 ffff88000d41dc00 0000000000000081 ffff8806acde0f10 [9417724.085716] Call Trace: [9417724.085720] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085724] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085727] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085730] [<ffffffff811862b4>] do_last+0x2b4/0x730 [9417724.085734] [<ffffffff81187ad1>] path_openat+0xd1/0x3f0 [9417724.085738] [<ffffffff81187f12>] do_filp_open+0x42/0xa0 [9417724.085742] [<ffffffff81319811>] ? strncpy_from_user+0x31/0x40 [9417724.085745] [<ffffffff8118325a>] ? do_getname+0x10a/0x180 [9417724.085749] [<ffffffff8165b00e>] ? _raw_spin_lock+0xe/0x20 [9417724.085752] [<ffffffff81195227>] ? alloc_fd+0xf7/0x150 [9417724.085755] [<ffffffff81177538>] do_sys_open+0xf8/0x240 [9417724.085759] [<ffffffff811776a0>] sys_open+0x20/0x30 [9417724.085762] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9417724.085765] INFO: task exim4:23744 blocked for more than 120 seconds. [9417724.085766] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9417724.085769] exim4 D ffffffff81806200 0 23744 6911 0x00000000 [9417724.085772] ffff88006d14fc78 0000000000000082 ffff8807e49f37b0 ffff88006d14fe38 [9417724.085777] ffff88006d14ffd8 ffff88006d14ffd8 ffff88006d14ffd8 00000000000137c0 [9417724.085781] ffff8807ea5e9700 ffff880010d01700 0000000000000081 ffff8806acde0f10 [9417724.085785] Call Trace: [9417724.085789] [<ffffffff816590ff>] schedule+0x3f/0x60 [9417724.085792] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9417724.085796] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9417724.085799] [<ffffffff811862b4>] do_last+0x2b4/0x730 [9417724.085803] [<ffffffff81187ad1>] path_openat+0xd1/0x3f0 [9417724.085806] [<ffffffff81187f12>] do_filp_open+0x42/0xa0 [9417724.085811] [<ffffffff81319811>] ? strncpy_from_user+0x31/0x40 [9417724.085814] [<ffffffff8118325a>] ? do_getname+0x10a/0x180 [9417724.085817] [<ffffffff8165b00e>] ? _raw_spin_lock+0xe/0x20 [9417724.085821] [<ffffffff81195227>] ? alloc_fd+0xf7/0x150 [9417724.085824] [<ffffffff81177538>] do_sys_open+0xf8/0x240 [9417724.085827] [<ffffffff811776a0>] sys_open+0x20/0x30 [9417724.085831] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9418083.051777] INFO: task exim4:23744 blocked for more than 120 seconds. [9418083.051782] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9418083.051785] exim4 D 0000000000000005 0 23744 6911 0x00000000 [9418083.051790] ffff88006d14fde8 0000000000000082 ffff880010d01700 0000000000000041 [9418083.051796] ffff88006d14ffd8 ffff88006d14ffd8 ffff88006d14ffd8 00000000000137c0 [9418083.051801] ffff8802799eae00 ffff880010d01700 0000000000000000 ffff8806acde0f10 [9418083.051805] Call Trace: [9418083.051815] [<ffffffff816590ff>] schedule+0x3f/0x60 [9418083.051821] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9418083.051826] [<ffffffff81187361>] ? do_path_lookup+0x31/0xc0 [9418083.051839] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9418083.051841] [<ffffffff8118774e>] do_unlinkat+0x8e/0x1d0 [9418083.051843] [<ffffffff81178290>] ? vfs_write+0x110/0x180 [9418083.051845] [<ffffffff8117855a>] ? sys_write+0x4a/0x90 [9418083.051846] [<ffffffff811883e6>] sys_unlink+0x16/0x20 [9418083.051848] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9418083.051850] INFO: task exim4:23949 blocked for more than 120 seconds. [9418083.051851] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9418083.051852] exim4 D 0000000000000004 0 23949 6911 0x00000000 [9418083.051853] ffff88000d95bde8 0000000000000082 ffff880026654500 0000000000000041 [9418083.051855] ffff88000d95bfd8 ffff88000d95bfd8 ffff88000d95bfd8 00000000000137c0 [9418083.051857] ffff8807ea398000 ffff880026654500 0000000000000000 ffff8806acde0f10 [9418083.051859] Call Trace: [9418083.051861] [<ffffffff816590ff>] schedule+0x3f/0x60 [9418083.051862] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9418083.051864] [<ffffffff81187361>] ? do_path_lookup+0x31/0xc0 [9418083.051865] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9418083.051866] [<ffffffff8118774e>] do_unlinkat+0x8e/0x1d0 [9418083.051868] [<ffffffff81178290>] ? vfs_write+0x110/0x180 [9418083.051869] [<ffffffff8117855a>] ? sys_write+0x4a/0x90 [9418083.051871] [<ffffffff811883e6>] sys_unlink+0x16/0x20 [9418083.051872] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9418083.051874] INFO: task exim4:24110 blocked for more than 120 seconds. [9418083.051875] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9418083.051875] exim4 D 0000000000000006 0 24110 6911 0x00000000 [9418083.051877] ffff8800222dbe38 0000000000000082 ffff8800222dbf68 ffffffff8118875d [9418083.051879] ffff8800222dbfd8 ffff8800222dbfd8 ffff8800222dbfd8 00000000000137c0 [9418083.051881] ffff8807e5ce4500 ffff880218a21700 0000000000000003 ffff8806acde0f10 [9418083.051882] Call Trace: [9418083.051884] [<ffffffff8118875d>] ? sys_renameat+0xbd/0x240 [9418083.051886] [<ffffffff816590ff>] schedule+0x3f/0x60 [9418083.051887] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9418083.051889] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9418083.051892] [<ffffffff8121194f>] ext4_sync_file+0xef/0x2d0 [9418083.051893] [<ffffffff811775bc>] ? do_sys_open+0x17c/0x240 [9418083.051896] [<ffffffff811a6ae6>] do_fsync+0x56/0x80 [9418083.051898] [<ffffffff811a6e10>] sys_fsync+0x10/0x20 [9418083.051899] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9418083.051901] INFO: task exim4:24185 blocked for more than 120 seconds. [9418083.051902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [9418083.051903] exim4 D 0000000000000000 0 24185 1 0x00000000 [9418083.051905] ffff8803242bfde8 0000000000000082 ffff8803242bfe38 ffffffff81646dda [9418083.051907] ffff8803242bffd8 ffff8803242bffd8 ffff8803242bffd8 00000000000137c0 [9418083.051908] ffff8807ea3a9700 ffff88000dd89700 0000000000000000 ffff8806acde0f10 [9418083.051910] Call Trace: [9418083.051913] [<ffffffff81646dda>] ? __slab_free+0x1d2/0x2f3 [9418083.051915] [<ffffffff816590ff>] schedule+0x3f/0x60 [9418083.051916] [<ffffffff81659f07>] __mutex_lock_slowpath+0xd7/0x150 [9418083.051918] [<ffffffff81187361>] ? do_path_lookup+0x31/0xc0 [9418083.051919] [<ffffffff81659b1a>] mutex_lock+0x2a/0x50 [9418083.051921] [<ffffffff8118774e>] do_unlinkat+0x8e/0x1d0 [9418083.051924] [<ffffffff81014721>] ? math_state_restore+0x51/0x80 [9418083.051925] [<ffffffff811883e6>] sys_unlink+0x16/0x20 [9418083.051927] [<ffffffff81663602>] system_call_fastpath+0x16/0x1b [9429960.233032] Process accounting paused [9430317.176019] Process accounting resumed I'n not sure i't exim to blame, but "exim" seems to be the only word I know in this list:) Exim version 4.76 #1 built 25-Oct-2012 14:23:36 Copyright (c) University of Cambridge, 1995 - 2007 Berkeley DB: Berkeley DB 5.1.25: (January 28, 2011) Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DKIM Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch nis nis0 passwd Authenticators: cram_md5 plaintext Routers: accept dnslookup ipliteral manualroute queryprogram redirect Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp Fixed never_users: 0 Size of off_t: 8 Actually i't used only for delivery of our local mail - less then 1 mail/sec. Can anyone say what happened? Thank you. -- С уважением, Антипов Илья -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
