Hello, This is the second time I experienced this so I thought and post to get some perspective. When this first happened, I suspected the kernel, and upgraded from 3.18.22 to 3.18.23.
Scenario: - lab scenario - single osd host — osdhost01. Supermicro X8DTE-F - 2x X5570 + 48G RAM + 20x 4TB OSD + SSD journals — 5:1 osd:journal ratio. All OSDs stable, no active rebuilds / scrubbing, etc. - osdhost01 also runs a mon - Separate mon-only node — two monitors in cluster. - osdhost01 maps rbds, exports via NFS. - rbds = xfs formatted. - osds = btrfs formatted. - ceph 0.94.5 - calltrace only on NFS use. I haven’t observed a specific trigger IO pattern. - When OSDs exported via block methods — e.g. SCST, things are stable. Relevant / recent log entires — nothing significant preceding the below: - system returned back to normal after a reboot. — Nov 9 06:56:11 osdhost01 bash: 2015-11-09 06:56:11.209312 7f60a6669700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:51.209310) Nov 9 06:56:11 osdhost01 bash: 2015-11-09 06:56:11.424398 7f608a0d5700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:51.424395) Nov 9 06:56:11 osdhost01 bash: 2015-11-09 06:56:11.982586 7fe3d2cf0700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:51.982583) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.287997 7fe3ee2f1700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:52.287995) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.290600 7f60a6669700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:52.290598) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.675519 7f6acff5c700 -1 osd.7 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.795557 front 2015-11-09 06:55:51.795557 (cutoff 2015-11-09 06:55:52.675515) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.763544 7f03b2c1d700 -1 osd.11 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.641513 front 2015-11-09 06:55:52.641513 (cutoff 2015-11-09 06:55:52.763540) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.804794 7f6ab4604700 -1 osd.7 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.795557 front 2015-11-09 06:55:51.795557 (cutoff 2015-11-09 06:55:52.804791) Nov 9 06:56:12 osdhost01 bash: 2015-11-09 06:56:12.708600 7fe64670f700 -1 osd.14 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.701280 front 2015-11-09 06:55:52.701280 (cutoff 2015-11-09 06:55:52.708597) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.056720 7f0395812700 -1 osd.11 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.641513 front 2015-11-09 06:55:52.641513 (cutoff 2015-11-09 06:55:53.056717) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.127314 7f608a0d5700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:53.127302) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.288281 7fe3ee2f1700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:53.288278) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.290793 7f60a6669700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:53.290791) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.469732 7f7809459700 -1 osd.3 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.637819 front 2015-11-09 06:55:52.637819 (cutoff 2015-11-09 06:55:53.469719) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.598682 7fe66401a700 -1 osd.14 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.701280 front 2015-11-09 06:55:52.701280 (cutoff 2015-11-09 06:55:53.598678) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.674622 7f9f225f1700 -1 osd.9 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.803479 front 2015-11-09 06:55:52.803479 (cutoff 2015-11-09 06:55:53.674619) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.675760 7f6acff5c700 -1 osd.7 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.795557 front 2015-11-09 06:55:51.795557 (cutoff 2015-11-09 06:55:53.675757) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.683551 7fe3d2cf0700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:53.683548) Nov 9 06:56:13 osdhost01 bash: 2015-11-09 06:56:13.763759 7f03b2c1d700 -1 osd.11 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.641513 front 2015-11-09 06:55:52.641513 (cutoff 2015-11-09 06:55:53.763756) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.031501 7f986f40c700 -1 osd.4 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:53.848296 front 2015-11-09 06:55:53.848296 (cutoff 2015-11-09 06:55:54.031498) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.080978 7efc5cccb700 -1 osd.1 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:53.169337 front 2015-11-09 06:55:53.169337 (cutoff 2015-11-09 06:55:54.080975) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.288519 7fe3ee2f1700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:54.288517) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.291009 7f60a6669700 -1 osd.10 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.113587 front 2015-11-09 06:55:55.814376 (cutoff 2015-11-09 06:55:54.291007) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.469982 7f7809459700 -1 osd.3 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.637819 front 2015-11-09 06:55:52.637819 (cutoff 2015-11-09 06:55:54.469979) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.512457 7f55a36db700 -1 osd.2 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:53.592724 front 2015-11-09 06:55:53.592724 (cutoff 2015-11-09 06:55:54.512454) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.598847 7fe66401a700 -1 osd.14 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.701280 front 2015-11-09 06:55:52.701280 (cutoff 2015-11-09 06:55:54.598844) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.674790 7f9f225f1700 -1 osd.9 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.803479 front 2015-11-09 06:55:52.803479 (cutoff 2015-11-09 06:55:54.674788) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.675914 7f6acff5c700 -1 osd.7 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.795557 front 2015-11-09 06:55:51.795557 (cutoff 2015-11-09 06:55:54.675911) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.763922 7f03b2c1d700 -1 osd.11 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.641513 front 2015-11-09 06:55:52.641513 (cutoff 2015-11-09 06:55:54.763919) Nov 9 06:56:14 osdhost01 bash: 2015-11-09 06:56:14.784489 7fe3d2cf0700 -1 osd.13 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:51.670920 front 2015-11-09 06:55:51.670920 (cutoff 2015-11-09 06:55:54.784486) Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.031711 7f986f40c700 -1 osd.4 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:53.848296 front 2015-11-09 06:55:53.848296 (cutoff 2015-11-09 06:55:55.031708) Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.045013 7f77edc85700 -1 osd.3 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:52.637819 front 2015-11-09 06:55:52.637819 (cutoff 2015-11-09 06:55:55.045009) Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.081213 7efc5cccb700 -1 osd.1 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:53.169337 front 2015-11-09 06:55:53.169337 (cutoff 2015-11-09 06:55:55.081209) Nov 9 06:56:15 osdhost01 kernel: libceph: osd19 down Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.186846 7f2c862f4700 -1 osd.18 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:55.001621 front 2015-11-09 06:55:55.001621 (cutoff 2015-11-09 06:55:55.186842) Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.323558 7f87a7d18700 -1 osd.5 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:54.729777 front 2015-11-09 06:55:54.729777 (cutoff 2015-11-09 06:55:55.323553) Nov 9 06:56:15 osdhost01 bash: 2015-11-09 06:56:15.541786 7fe57705f700 -1 osd.12 504 heartbeat_check: no reply from osd.19 since back 2015-11-09 06:55:54.908156 front 2015-11-09 06:55:54.908156 (cutoff 2015-11-09 06:55:55.541781) Nov 9 06:58:52 osdhost01 kernel: INFO: task kswapd0:154 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: kswapd0 D ffff880627c13500 0 154 2 0x00000000 Nov 9 06:58:52 osdhost01 kernel: ffff8806198077b8 0000000000000046 ffff88061de843d0 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff880619807fd8 0000000000013500 ffffffff819a3460 ffff88061de843d0 Nov 9 06:58:52 osdhost01 kernel: ffff8806198077f8 ffff8806168baf50 ffff8806168baf58 7fffffffffffffff Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166ca49>] schedule+0x29/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166fbdc>] schedule_timeout+0x20c/0x280 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812dd878>] ? blk_finish_plug+0x18/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01793f7>] ? _xfs_buf_ioapply+0x327/0x430 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166e386>] wait_for_completion+0x116/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017b394>] ? xfs_bwrite+0x24/0x60 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017af7d>] xfs_buf_submit_wait+0xbd/0x1d0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017b394>] xfs_bwrite+0x24/0x60 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0182c3b>] xfs_reclaim_inode+0x32b/0x350 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8101d365>] ? native_sched_clock+0x35/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8101d3c9>] ? sched_clock+0x9/0x10 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a6b35>] ? sched_clock_cpu+0x85/0xc0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a30f1>] ? try_to_wake_up+0x1c1/0x320 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a3277>] ? wake_up_process+0x27/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811de98b>] ? mem_cgroup_iter+0x13b/0x3d0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81189f55>] kswapd_shrink_zone+0x155/0x210 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118b21d>] kswapd+0x4ad/0x8e0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118ad70>] ? mem_cgroup_shrink_node_zone+0x170/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: INFO: task kswapd1:155 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: kswapd1 D ffff880c3fcb3500 0 155 2 0x00000000 Nov 9 06:58:52 osdhost01 kernel: ffff88061980b888 0000000000000046 ffff88061de85160 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff88061980bfd8 0000000000013500 ffff880c1e1cdef0 ffff88061de85160 Nov 9 06:58:52 osdhost01 kernel: ffff88061980b8d8 ffff880c3fcb3e18 ffff88061980b8d8 ffff88063ff9a0c0 Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa018dca9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166cd41>] io_schedule+0xa1/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa018a609>] __xfs_iunpin_wait+0xd9/0x150 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810b7f20>] ? autoremove_wake_function+0x40/0x40 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa018dca9>] xfs_iunpin_wait+0x19/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa018299c>] xfs_reclaim_inode+0x8c/0x350 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812099cc>] ? destroy_inode+0x3c/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81209b0a>] ? evict+0x10a/0x180 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81189f55>] kswapd_shrink_zone+0x155/0x210 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118b21d>] kswapd+0x4ad/0x8e0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118ad70>] ? mem_cgroup_shrink_node_zone+0x170/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: INFO: task ceph-osd:13820 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: ceph-osd D ffff880c3fcf3500 0 13820 5406 0x00000080 Nov 9 06:58:52 osdhost01 kernel: ffff8809406734f0 0000000000000082 ffff8803dabda8b0 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff880940673fd8 0000000000013500 ffff8809a7f30d90 ffff8803dabda8b0 Nov 9 06:58:52 osdhost01 kernel: ffff880940673530 ffff880112a1b250 ffff880112a1b258 7fffffffffffffff Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166ca49>] schedule+0x29/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166fbdc>] schedule_timeout+0x20c/0x280 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812dd878>] ? blk_finish_plug+0x18/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01793f7>] ? _xfs_buf_ioapply+0x327/0x430 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166e386>] wait_for_completion+0x116/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017b394>] ? xfs_bwrite+0x24/0x60 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017af7d>] xfs_buf_submit_wait+0xbd/0x1d0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017b394>] xfs_bwrite+0x24/0x60 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0182c3b>] xfs_reclaim_inode+0x32b/0x350 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a077d>] ? ttwu_do_activate.constprop.91+0x5d/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a30f1>] ? try_to_wake_up+0x1c1/0x320 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a3277>] ? wake_up_process+0x27/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811de98b>] ? mem_cgroup_iter+0x13b/0x3d0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118a63b>] shrink_zones.constprop.43+0x2fb/0x410 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118a80b>] do_try_to_free_pages+0xbb/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8118a95a>] try_to_free_pages+0xda/0x170 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811d3bee>] ? remove_migration_ptes+0x4e/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81664588>] __alloc_pages_slowpath+0x51e/0x82d Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117e111>] __alloc_pages_nodemask+0x241/0x270 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117e1fd>] alloc_kmem_pages_node+0x6d/0x140 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811ce7c5>] ? kmem_cache_free+0x1c5/0x1f0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81073f43>] copy_process.part.25+0x133/0x1be0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81061dac>] ? __do_page_fault+0x21c/0x550 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8153b6dd>] ? SYSC_accept4+0x1ad/0x220 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81075bb1>] do_fork+0xe1/0x360 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811183a4>] ? __audit_syscall_entry+0xb4/0x110 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81021b8c>] ? do_audit_syscall_entry+0x6c/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810230f3>] ? syscall_trace_enter_phase1+0x143/0x1a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81075eb6>] SyS_clone+0x16/0x20 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671449>] stub_clone+0x69/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671109>] ? system_call_fastpath+0x12/0x17 Nov 9 06:58:52 osdhost01 kernel: INFO: task xfsaild/rbd1:15307 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: xfsaild/rbd1 D ffff880c3fc33500 0 15307 2 0x00000080 Nov 9 06:58:52 osdhost01 kernel: ffff880289363a38 0000000000000046 ffff88003d37c3d0 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff880289363fd8 0000000000013500 ffff8809a98bec80 ffff88003d37c3d0 Nov 9 06:58:52 osdhost01 kernel: ffff8802ba4da8f0 ffff880c3fc33e18 ffff8802ba4da8f0 ffff880c16d87150 Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166cd41>] io_schedule+0xa1/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812db366>] get_request+0x226/0x780 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810b7ee0>] ? prepare_to_wait_event+0xf0/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812dd586>] blk_queue_bio+0xc6/0x3a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812d8680>] generic_make_request+0xe0/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812d8748>] submit_bio+0x78/0x160 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01793b6>] _xfs_buf_ioapply+0x2e6/0x430 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017add8>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017ab08>] xfs_buf_submit+0x68/0x1d0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017add8>] __xfs_buf_delwri_submit+0x168/0x250 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017bb4f>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01a6bc0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017bb4f>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01a6dfd>] xfsaild+0x23d/0x5d0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01a6bc0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: INFO: task rsync:18478 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: rsync D ffff880627c33500 0 18478 18462 0x00000084 Nov 9 06:58:52 osdhost01 kernel: ffff8800b6e13718 0000000000000082 ffff880c04b29b20 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff8800b6e13fd8 0000000000013500 ffff88061e6b3640 ffff880c04b29b20 Nov 9 06:58:52 osdhost01 kernel: ffff8800b6e13718 ffff8800090962b0 7fffffffffffffff ffff880c04b29b20 Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166ca49>] schedule+0x29/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166fbdc>] schedule_timeout+0x20c/0x280 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0151058>] ? xfs_bmbt_get_all+0x18/0x20 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0146983>] ? xfs_bmap_search_multi_extents+0xb3/0x140 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166ee75>] __down_common+0xfe/0x148 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017a43d>] ? _xfs_buf_find+0x15d/0x2c0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166eedc>] __down+0x1d/0x1f Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810bcac1>] down+0x41/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017a240>] xfs_buf_lock+0x40/0xe0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017a43d>] _xfs_buf_find+0x15d/0x2c0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017a5ca>] xfs_buf_get_map+0x2a/0x180 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017b0fc>] xfs_buf_read_map+0x2c/0x140 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01a7f19>] xfs_trans_read_buf_map+0x199/0x400 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa016efae>] xfs_imap_to_bp+0x6e/0xf0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa016f4ae>] xfs_iread+0x7e/0x420 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01986f7>] ? kmem_zone_alloc+0x77/0x100 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0183218>] xfs_iget+0x228/0x6c0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa018bb07>] xfs_lookup+0xe7/0x110 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0188c3b>] xfs_vn_lookup+0x7b/0xd0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f77fd>] lookup_real+0x1d/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f81a2>] __lookup_hash+0x42/0x60 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81666c96>] lookup_slow+0x45/0xab Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811fceac>] path_lookupat+0x73c/0x770 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811cf5e5>] ? kmem_cache_alloc+0x35/0x1e0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811fdf6f>] ? getname_flags+0x4f/0x1a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811fcf0b>] filename_lookup+0x2b/0xc0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811fef93>] user_path_at_empty+0x63/0xc0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0188769>] ? xfs_vn_follow_link+0x49/0xa0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81215e6f>] ? kfree_put_link+0x1f/0x30 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811ff001>] user_path_at+0x11/0x20 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f2b63>] vfs_fstatat+0x63/0xc0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f3131>] SYSC_newlstat+0x31/0x60 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81021b8c>] ? do_audit_syscall_entry+0x6c/0x70 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810230f3>] ? syscall_trace_enter_phase1+0x143/0x1a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811185f6>] ? __audit_syscall_exit+0x1f6/0x2a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff811f33de>] SyS_newlstat+0xe/0x10 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671109>] system_call_fastpath+0x12/0x17 Nov 9 06:58:52 osdhost01 kernel: INFO: task kworker/u50:7:18612 blocked for more than 120 seconds. Nov 9 06:58:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 06:58:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 06:58:52 osdhost01 kernel: kworker/u50:7 D ffff880c3fc13500 0 18612 2 0x00000080 Nov 9 06:58:52 osdhost01 kernel: Workqueue: writeback bdi_writeback_workfn (flush-252:16) Nov 9 06:58:52 osdhost01 kernel: ffff88001eb33688 0000000000000046 ffff8809add30000 0000000000013500 Nov 9 06:58:52 osdhost01 kernel: ffff88001eb33fd8 0000000000013500 ffff88061e6b5ef0 ffff8809add30000 Nov 9 06:58:52 osdhost01 kernel: ffff8808adad2750 ffff880c3fc13e18 ffff8808adad2750 ffff880c16d87150 Nov 9 06:58:52 osdhost01 kernel: Call Trace: Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8166cd41>] io_schedule+0xa1/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812db366>] get_request+0x226/0x780 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812e08d1>] ? ioc_lookup_icq+0x31/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810b7ee0>] ? prepare_to_wait_event+0xf0/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812dd586>] blk_queue_bio+0xc6/0x3a0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812d8680>] generic_make_request+0xe0/0x130 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff812d8748>] submit_bio+0x78/0x160 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa01735a3>] xfs_submit_ioend_bio.isra.13+0x33/0x40 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa017369f>] xfs_submit_ioend+0xef/0x130 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0174418>] xfs_vm_writepage+0x2b8/0x5f0 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117f067>] __writepage+0x17/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117fa35>] write_cache_pages+0x245/0x4b0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117f050>] ? global_dirtyable_memory+0x50/0x50 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8117fced>] generic_writepages+0x4d/0x80 Nov 9 06:58:52 osdhost01 kernel: [<ffffffffa0173c73>] xfs_vm_writepages+0x43/0x50 [xfs] Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81180f2e>] do_writepages+0x1e/0x40 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81217680>] __writeback_single_inode+0x40/0x220 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810a926b>] ? select_idle_sibling+0x2b/0x120 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81218123>] writeback_sb_inodes+0x283/0x450 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8121838f>] __writeback_inodes_wb+0x9f/0xd0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81218c03>] wb_writeback+0x263/0x2f0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8121b19c>] bdi_writeback_workfn+0x1dc/0x470 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8108f7ce>] process_one_work+0x14e/0x3f0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8108fedb>] worker_thread+0x11b/0x4d0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff8108fdc0>] ? rescuer_thread+0x350/0x350 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810795f7>] ? do_exit+0x6e7/0xa60 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 06:58:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 06:59:13 osdhost01 bash: 2015-11-09 06:59:13.324872 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:09.075331 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:53.324870) Nov 9 06:59:14 osdhost01 bash: 2015-11-09 06:59:14.325102 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:09.075331 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:54.325101) Nov 9 06:59:14 osdhost01 bash: 2015-11-09 06:59:14.377276 7fe3d2cf0700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:09.075331 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:54.377274) Nov 9 06:59:15 osdhost01 bash: 2015-11-09 06:59:15.325272 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:14.376696 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:55.325271) Nov 9 06:59:16 osdhost01 bash: 2015-11-09 06:59:16.078781 7fe3d2cf0700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:14.376696 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:56.078779) Nov 9 06:59:16 osdhost01 bash: 2015-11-09 06:59:16.325476 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:16.077627 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:56.325474) Nov 9 06:59:17 osdhost01 bash: 2015-11-09 06:59:17.325632 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:16.077627 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:57.325631) Nov 9 06:59:18 osdhost01 bash: 2015-11-09 06:59:18.325792 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:16.077627 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:58.325791) Nov 9 06:59:18 osdhost01 bash: 2015-11-09 06:59:18.379497 7fe3d2cf0700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:16.077627 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:58.379494) Nov 9 06:59:18 osdhost01 bash: 2015-11-09 06:59:18.880692 7fe3d2cf0700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.379113 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:58.880690) Nov 9 06:59:19 osdhost01 bash: 2015-11-09 06:59:19.325958 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:58:59.325957) Nov 9 06:59:20 osdhost01 bash: 2015-11-09 06:59:20.326074 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:00.326072) Nov 9 06:59:21 osdhost01 bash: 2015-11-09 06:59:21.326255 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:01.326254) Nov 9 06:59:22 osdhost01 bash: 2015-11-09 06:59:22.326442 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:02.326440) Nov 9 06:59:23 osdhost01 bash: 2015-11-09 06:59:23.326682 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:03.326678) Nov 9 06:59:24 osdhost01 bash: 2015-11-09 06:59:24.182479 7fe3d2cf0700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:18.879812 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:04.182477) Nov 9 06:59:24 osdhost01 bash: 2015-11-09 06:59:24.326853 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:24.181049 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:04.326852) Nov 9 06:59:25 osdhost01 bash: 2015-11-09 06:59:25.327009 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:24.181049 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:05.327008) Nov 9 06:59:26 osdhost01 bash: 2015-11-09 06:59:26.327205 7fe3ee2f1700 -1 osd.13 506 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:59:24.181049 front 2015-11-09 06:58:52.768953 (cutoff 2015-11-09 06:59:06.327203) Nov 9 06:59:26 osdhost01 kernel: libceph: osd1 down Nov 9 06:59:41 osdhost01 kernel: libceph: osd1 up Nov 9 06:59:43 osdhost01 bash: 2015-11-09 06:59:43.068228 7f15fd7ae700 -1 osd.8 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:23.068225) Nov 9 06:59:43 osdhost01 bash: 2015-11-09 06:59:43.353261 7f87a7d18700 -1 osd.5 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:23.353258) Nov 9 06:59:43 osdhost01 bash: 2015-11-09 06:59:43.574038 7fe57705f700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:23.574035) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.068432 7f15fd7ae700 -1 osd.8 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:24.068421) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.107312 7fe55c0d0700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:24.107308) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.233835 7f2c862f4700 -1 osd.18 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.809705 front 2015-11-09 06:59:23.809705 (cutoff 2015-11-09 06:59:24.233831) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.353516 7f87a7d18700 -1 osd.5 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:24.353512) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.544725 7f55a36db700 -1 osd.2 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.874609 front 2015-11-09 06:59:23.874609 (cutoff 2015-11-09 06:59:24.544721) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.574217 7fe57705f700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:24.574214) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.609590 7fe55c0d0700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:24.609586) Nov 9 06:59:44 osdhost01 bash: 2015-11-09 06:59:44.721117 7f2c6a135700 -1 osd.18 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.809705 front 2015-11-09 06:59:23.809705 (cutoff 2015-11-09 06:59:24.721114) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.061858 7f986f40c700 -1 osd.4 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:24.254551 front 2015-11-09 06:59:24.254551 (cutoff 2015-11-09 06:59:25.061854) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.068784 7f15fd7ae700 -1 osd.8 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:25.068781) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.234106 7f2c862f4700 -1 osd.18 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.809705 front 2015-11-09 06:59:23.809705 (cutoff 2015-11-09 06:59:25.234101) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.353677 7f87a7d18700 -1 osd.5 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:25.353673) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.544913 7f55a36db700 -1 osd.2 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.874609 front 2015-11-09 06:59:23.874609 (cutoff 2015-11-09 06:59:25.544909) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.574450 7fe57705f700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:25.574445) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.626694 7f878bdac700 -1 osd.5 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:25.626691) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.705895 7f6acff5c700 -1 osd.7 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.113821 front 2015-11-09 06:59:25.113821 (cutoff 2015-11-09 06:59:25.705891) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.710796 7fe55c0d0700 -1 osd.12 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:25.710793) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.722186 7f2bb588e700 -1 osd.16 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525165 front 2015-11-09 06:59:25.525165 (cutoff 2015-11-09 06:59:25.722182) Nov 9 06:59:45 osdhost01 bash: 2015-11-09 06:59:45.916430 7f0167723700 -1 osd.17 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.391105 front 2015-11-09 06:59:25.391105 (cutoff 2015-11-09 06:59:25.916426) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.001068 7f8495a8d700 -1 osd.6 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.696253 front 2015-11-09 06:59:25.696253 (cutoff 2015-11-09 06:59:26.001066) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.061989 7f986f40c700 -1 osd.4 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:24.254551 front 2015-11-09 06:59:24.254551 (cutoff 2015-11-09 06:59:26.061982) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.068905 7f15fd7ae700 -1 osd.8 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:26.068902) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.082426 7f5588367700 -1 osd.2 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.874609 front 2015-11-09 06:59:23.874609 (cutoff 2015-11-09 06:59:26.082423) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.234303 7f2c862f4700 -1 osd.18 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.809705 front 2015-11-09 06:59:23.809705 (cutoff 2015-11-09 06:59:26.234299) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.264922 7fe66401a700 -1 osd.14 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.183596 front 2015-11-09 06:59:26.183596 (cutoff 2015-11-09 06:59:26.264919) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.312961 7f9f503e9700 -1 osd.15 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525498 front 2015-11-09 06:59:25.525498 (cutoff 2015-11-09 06:59:26.312956) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.325140 7f84b1109700 -1 osd.6 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.696253 front 2015-11-09 06:59:25.696253 (cutoff 2015-11-09 06:59:26.325137) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.353794 7f87a7d18700 -1 osd.5 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:26.353791) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.361189 7f9852a8d700 -1 osd.4 509 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:24.254551 front 2015-11-09 06:59:24.254551 (cutoff 2015-11-09 06:59:26.361188) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.415100 7f15e2534700 -1 osd.8 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:26.415097) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.501854 7f7809459700 -1 osd.3 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.840797 front 2015-11-09 06:59:25.840797 (cutoff 2015-11-09 06:59:26.501851) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.545285 7f55a36db700 -1 osd.2 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.874609 front 2015-11-09 06:59:23.874609 (cutoff 2015-11-09 06:59:26.545282) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.574712 7fe57705f700 -1 osd.12 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:26.574709) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.706044 7f6acff5c700 -1 osd.7 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.113821 front 2015-11-09 06:59:25.113821 (cutoff 2015-11-09 06:59:26.706041) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.706366 7f9f225f1700 -1 osd.9 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.100799 front 2015-11-09 06:59:26.100799 (cutoff 2015-11-09 06:59:26.706364) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.722330 7f2bb588e700 -1 osd.16 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525165 front 2015-11-09 06:59:25.525165 (cutoff 2015-11-09 06:59:26.722327) Nov 9 06:59:46 osdhost01 bash: 2015-11-09 06:59:46.916731 7f0167723700 -1 osd.17 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.391105 front 2015-11-09 06:59:25.391105 (cutoff 2015-11-09 06:59:26.916728) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.038581 7f9f346ab700 -1 osd.15 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525498 front 2015-11-09 06:59:25.525498 (cutoff 2015-11-09 06:59:27.038575) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.041196 7f229a9e1700 -1 osd.0 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.370980 front 2015-11-09 06:59:26.370980 (cutoff 2015-11-09 06:59:27.041193) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.062382 7f986f40c700 -1 osd.4 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:24.254551 front 2015-11-09 06:59:24.254551 (cutoff 2015-11-09 06:59:27.062379) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.069322 7f15fd7ae700 -1 osd.8 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.504852 front 2015-11-09 06:59:22.504852 (cutoff 2015-11-09 06:59:27.069320) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.234567 7f2c862f4700 -1 osd.18 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.809705 front 2015-11-09 06:59:23.809705 (cutoff 2015-11-09 06:59:27.234564) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.265240 7fe66401a700 -1 osd.14 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.183596 front 2015-11-09 06:59:26.183596 (cutoff 2015-11-09 06:59:27.265238) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.279601 7f227e3ed700 -1 osd.0 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.370980 front 2015-11-09 06:59:26.370980 (cutoff 2015-11-09 06:59:27.279433) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.313253 7f9f503e9700 -1 osd.15 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525498 front 2015-11-09 06:59:25.525498 (cutoff 2015-11-09 06:59:27.313250) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.325483 7f84b1109700 -1 osd.6 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.696253 front 2015-11-09 06:59:25.696253 (cutoff 2015-11-09 06:59:27.325481) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.354154 7f87a7d18700 -1 osd.5 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:22.917338 front 2015-11-09 06:59:22.917338 (cutoff 2015-11-09 06:59:27.354151) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.461814 7f9852a8d700 -1 osd.4 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:24.254551 front 2015-11-09 06:59:24.254551 (cutoff 2015-11-09 06:59:27.461812) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.502074 7f7809459700 -1 osd.3 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.840797 front 2015-11-09 06:59:25.840797 (cutoff 2015-11-09 06:59:27.502071) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.545455 7f55a36db700 -1 osd.2 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.874609 front 2015-11-09 06:59:23.874609 (cutoff 2015-11-09 06:59:27.545452) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.574978 7fe57705f700 -1 osd.12 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:23.394243 front 2015-11-09 06:59:23.394243 (cutoff 2015-11-09 06:59:27.574975) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.630791 7f2b9a97a700 -1 osd.16 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525165 front 2015-11-09 06:59:25.525165 (cutoff 2015-11-09 06:59:27.630788) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.706325 7f6acff5c700 -1 osd.7 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.113821 front 2015-11-09 06:59:25.113821 (cutoff 2015-11-09 06:59:27.706321) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.706637 7f9f225f1700 -1 osd.9 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.100799 front 2015-11-09 06:59:26.100799 (cutoff 2015-11-09 06:59:27.706634) Nov 9 06:59:47 osdhost01 bash: 2015-11-09 06:59:47.722679 7f2bb588e700 -1 osd.16 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:25.525165 front 2015-11-09 06:59:25.525165 (cutoff 2015-11-09 06:59:27.722676) Nov 9 06:59:48 osdhost01 bash: 2015-11-09 06:59:48.041315 7f229a9e1700 -1 osd.0 510 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:59:26.370980 front 2015-11-09 06:59:26.370980 (cutoff 2015-11-09 06:59:28.041312) Nov 9 06:59:51 osdhost01 kernel: libceph: osd13 down Nov 9 07:00:01 osdhost01 systemd: Starting Session 248 of user root. Nov 9 07:00:01 osdhost01 systemd: Started Session 248 of user root. Nov 9 07:00:52 osdhost01 kernel: INFO: task kswapd0:154 blocked for more than 120 seconds. Nov 9 07:00:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 07:00:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 07:00:52 osdhost01 kernel: kswapd0 D ffff880627c13500 0 154 2 0x00000000 Nov 9 07:00:52 osdhost01 kernel: ffff8806198077b8 0000000000000046 ffff88061de843d0 0000000000013500 Nov 9 07:00:52 osdhost01 kernel: ffff880619807fd8 0000000000013500 ffffffff819a3460 ffff88061de843d0 Nov 9 07:00:52 osdhost01 kernel: ffff8806198077f8 ffff8806168baf50 ffff8806168baf58 7fffffffffffffff Nov 9 07:00:52 osdhost01 kernel: Call Trace: Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166ca49>] schedule+0x29/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166fbdc>] schedule_timeout+0x20c/0x280 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812dd878>] ? blk_finish_plug+0x18/0x50 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01793f7>] ? _xfs_buf_ioapply+0x327/0x430 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166e386>] wait_for_completion+0x116/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017b394>] ? xfs_bwrite+0x24/0x60 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017af7d>] xfs_buf_submit_wait+0xbd/0x1d0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017b394>] xfs_bwrite+0x24/0x60 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0182c3b>] xfs_reclaim_inode+0x32b/0x350 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8101d365>] ? native_sched_clock+0x35/0x90 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8101d3c9>] ? sched_clock+0x9/0x10 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a6b35>] ? sched_clock_cpu+0x85/0xc0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a30f1>] ? try_to_wake_up+0x1c1/0x320 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a3277>] ? wake_up_process+0x27/0x50 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811de98b>] ? mem_cgroup_iter+0x13b/0x3d0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81189f55>] kswapd_shrink_zone+0x155/0x210 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118b21d>] kswapd+0x4ad/0x8e0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118ad70>] ? mem_cgroup_shrink_node_zone+0x170/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:00:52 osdhost01 kernel: INFO: task kswapd1:155 blocked for more than 120 seconds. Nov 9 07:00:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 07:00:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 07:00:52 osdhost01 kernel: kswapd1 D ffff880c3fcb3500 0 155 2 0x00000000 Nov 9 07:00:52 osdhost01 kernel: ffff88061980b888 0000000000000046 ffff88061de85160 0000000000013500 Nov 9 07:00:52 osdhost01 kernel: ffff88061980bfd8 0000000000013500 ffff880c1e1cdef0 ffff88061de85160 Nov 9 07:00:52 osdhost01 kernel: ffff88061980b8d8 ffff880c3fcb3e18 ffff88061980b8d8 ffff88063ff9a0c0 Nov 9 07:00:52 osdhost01 kernel: Call Trace: Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa018dca9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166cd41>] io_schedule+0xa1/0x130 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa018a609>] __xfs_iunpin_wait+0xd9/0x150 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810b7f20>] ? autoremove_wake_function+0x40/0x40 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa018dca9>] xfs_iunpin_wait+0x19/0x20 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa018299c>] xfs_reclaim_inode+0x8c/0x350 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812099cc>] ? destroy_inode+0x3c/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81209b0a>] ? evict+0x10a/0x180 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81189f55>] kswapd_shrink_zone+0x155/0x210 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118b21d>] kswapd+0x4ad/0x8e0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118ad70>] ? mem_cgroup_shrink_node_zone+0x170/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:00:52 osdhost01 kernel: INFO: task ceph-osd:13820 blocked for more than 120 seconds. Nov 9 07:00:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 07:00:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 07:00:52 osdhost01 kernel: ceph-osd D ffff880c3fcf3500 0 13820 5406 0x00000080 Nov 9 07:00:52 osdhost01 kernel: ffff8809406734f0 0000000000000082 ffff8803dabda8b0 0000000000013500 Nov 9 07:00:52 osdhost01 kernel: ffff880940673fd8 0000000000013500 ffff8809a7f30d90 ffff8803dabda8b0 Nov 9 07:00:52 osdhost01 kernel: ffff880940673530 ffff880112a1b250 ffff880112a1b258 7fffffffffffffff Nov 9 07:00:52 osdhost01 kernel: Call Trace: Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166ca49>] schedule+0x29/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166fbdc>] schedule_timeout+0x20c/0x280 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812dd878>] ? blk_finish_plug+0x18/0x50 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01793f7>] ? _xfs_buf_ioapply+0x327/0x430 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166e386>] wait_for_completion+0x116/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017b394>] ? xfs_bwrite+0x24/0x60 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017af7d>] xfs_buf_submit_wait+0xbd/0x1d0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017b394>] xfs_bwrite+0x24/0x60 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0182c3b>] xfs_reclaim_inode+0x32b/0x350 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0182ec7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a077d>] ? ttwu_do_activate.constprop.91+0x5d/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a30f1>] ? try_to_wake_up+0x1c1/0x320 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a3277>] ? wake_up_process+0x27/0x50 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0183a63>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa0193035>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811f14fc>] super_cache_scan+0x17c/0x190 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811855b8>] shrink_slab_node+0x138/0x290 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811de98b>] ? mem_cgroup_iter+0x13b/0x3d0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118737c>] shrink_slab+0x13c/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118a63b>] shrink_zones.constprop.43+0x2fb/0x410 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118a80b>] do_try_to_free_pages+0xbb/0x130 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8118a95a>] try_to_free_pages+0xda/0x170 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811d3bee>] ? remove_migration_ptes+0x4e/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81664588>] __alloc_pages_slowpath+0x51e/0x82d Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8117e111>] __alloc_pages_nodemask+0x241/0x270 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8117e1fd>] alloc_kmem_pages_node+0x6d/0x140 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811ce7c5>] ? kmem_cache_free+0x1c5/0x1f0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81073f43>] copy_process.part.25+0x133/0x1be0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81061dac>] ? __do_page_fault+0x21c/0x550 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8153b6dd>] ? SYSC_accept4+0x1ad/0x220 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81075bb1>] do_fork+0xe1/0x360 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff811183a4>] ? __audit_syscall_entry+0xb4/0x110 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81021b8c>] ? do_audit_syscall_entry+0x6c/0x70 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810230f3>] ? syscall_trace_enter_phase1+0x143/0x1a0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81075eb6>] SyS_clone+0x16/0x20 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81671449>] stub_clone+0x69/0x90 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81671109>] ? system_call_fastpath+0x12/0x17 Nov 9 07:00:52 osdhost01 kernel: INFO: task xfsaild/rbd1:15307 blocked for more than 120 seconds. Nov 9 07:00:52 osdhost01 kernel: Not tainted 3.18.23 #1 Nov 9 07:00:52 osdhost01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 9 07:00:52 osdhost01 kernel: xfsaild/rbd1 D ffff880c3fc33500 0 15307 2 0x00000080 Nov 9 07:00:52 osdhost01 kernel: ffff880289363a38 0000000000000046 ffff88003d37c3d0 0000000000013500 Nov 9 07:00:52 osdhost01 kernel: ffff880289363fd8 0000000000013500 ffff8809a98bec80 ffff88003d37c3d0 Nov 9 07:00:52 osdhost01 kernel: ffff8802ba4da8f0 ffff880c3fc33e18 ffff8802ba4da8f0 ffff880c16d87150 Nov 9 07:00:52 osdhost01 kernel: Call Trace: Nov 9 07:00:52 osdhost01 kernel: [<ffffffff8166cd41>] io_schedule+0xa1/0x130 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812db366>] get_request+0x226/0x780 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810b7ee0>] ? prepare_to_wait_event+0xf0/0xf0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812dd586>] blk_queue_bio+0xc6/0x3a0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812d8680>] generic_make_request+0xe0/0x130 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff812d8748>] submit_bio+0x78/0x160 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01793b6>] _xfs_buf_ioapply+0x2e6/0x430 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810a32c0>] ? wake_up_state+0x20/0x20 Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017add8>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017ab08>] xfs_buf_submit+0x68/0x1d0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017add8>] __xfs_buf_delwri_submit+0x168/0x250 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017bb4f>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01a6bc0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa017bb4f>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01a6dfd>] xfsaild+0x23d/0x5d0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffffa01a6bc0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810956c8>] kthread+0xd8/0xf0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff81671058>] ret_from_fork+0x58/0x90 Nov 9 07:00:52 osdhost01 kernel: [<ffffffff810955f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 9 07:01:01 osdhost01 systemd: Starting Session 249 of user root. Nov 9 07:01:01 osdhost01 systemd: Started Session 249 of user root. Nov 9 07:01:17 osdhost01 kernel: libceph: osd19 weight 0x0 (out) Nov 9 07:01:30 osdhost01 systemd: Starting Session 250 of user root. Nov 9 07:01:30 osdhost01 systemd-logind: New session 250 of user root. Nov 9 07:01:30 osdhost01 systemd: Started Session 250 of user root. Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752026 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.0 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752107 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752128 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.2 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752211 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.3 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752235 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.4 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752248 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.5 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752256 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.6 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752265 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.7 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752275 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.8 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752283 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.9 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752291 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.10 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752304 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.11 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752313 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.12 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752320 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.13 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752328 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.14 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.752335 7efed0c89700 -1 osd.19 504 heartbeat_check: no reply from osd.18 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.752021) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.759964 7efeb225b700 -1 osd.19 504 heartbeat_check: no reply from osd.0 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.759962) Nov 9 07:01:41 osdhost01 bash: 2015-11-09 07:01:41.759982 7efeb225b700 -1 osd.19 504 heartbeat_check: no reply from osd.1 since back 2015-11-09 06:55:55.212501 front 2015-11-09 06:55:55.212501 (cutoff 2015-11-09 07:01:21.759962) — _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
