I remembered one thing I changed some days ago. Cahnging the default io scheduler from cfq to anticipatory. With the latter one, it was impossible to resync the software raid1 md3, as you can see in dmesg logs. Changed it back to defaults and waited for the raid to be synced again. After that started the kvm guests again. But still get lot of kernel messages:
See: [ 248.800024] Clocksource tsc unstable (delta = -270012333 ns) [ 6720.520038] INFO: task flush-9:2:454 blocked for more than 120 seconds. [ 6720.524331] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 6720.530099] flush-9:2 D 0000000000000000 0 454 2 0x00000000 [ 6720.530109] ffff8801938578d0 0000000000000046 0000000000015b80 0000000000015b80 [ 6720.530118] ffff880193859ab0 ffff880193857fd8 0000000000015b80 ffff8801938596f0 [ 6720.530126] 0000000000015b80 ffff880193857fd8 0000000000015b80 ffff880193859ab0 [ 6720.530134] Call Trace: [ 6720.530151] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.530161] [<ffffffff8153e697>] io_schedule+0x47/0x70 [ 6720.530168] [<ffffffff8116c775>] sync_buffer+0x45/0x50 [ 6720.530175] [<ffffffff8153ed9a>] __wait_on_bit_lock+0x5a/0xc0 [ 6720.530182] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.530189] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6720.530196] [<ffffffff8153ee78>] out_of_line_wait_on_bit_lock+0x78/0x90 [ 6720.530205] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40 [ 6720.530212] [<ffffffff8116c8f6>] __lock_buffer+0x36/0x40 [ 6720.530219] [<ffffffff8116d644>] __block_write_full_page+0x374/0x3a0 [ 6720.530227] [<ffffffff810f39e7>] ? unlock_page+0x27/0x30 [ 6720.530234] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6720.530241] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6720.530249] [<ffffffff8116dfd0>] block_write_full_page_endio+0xe0/0x120 [ 6720.530256] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6720.530263] [<ffffffff8116e025>] block_write_full_page+0x15/0x20 [ 6720.530271] [<ffffffff811b636d>] ext3_ordered_writepage+0x1dd/0x200 [ 6720.530279] [<ffffffff810fb907>] __writepage+0x17/0x40 [ 6720.530287] [<ffffffff810fcac7>] write_cache_pages+0x227/0x4d0 [ 6720.530294] [<ffffffff810fb8f0>] ? __writepage+0x0/0x40 [ 6720.530302] [<ffffffff810fcd94>] generic_writepages+0x24/0x30 [ 6720.530309] [<ffffffff810fcdd5>] do_writepages+0x35/0x40 [ 6720.530315] [<ffffffff81164b66>] writeback_single_inode+0xf6/0x3d0 [ 6720.530322] [<ffffffff811657d0>] writeback_inodes_wb+0x410/0x5e0 [ 6720.530328] [<ffffffff81165aaa>] wb_writeback+0x10a/0x1d0 [ 6720.530335] [<ffffffff81077895>] ? try_to_del_timer_sync+0x75/0xd0 [ 6720.530342] [<ffffffff8153eb7b>] ? schedule_timeout+0x19b/0x300 [ 6720.530348] [<ffffffff81165ddc>] wb_do_writeback+0x18c/0x1a0 [ 6720.530355] [<ffffffff81165e43>] bdi_writeback_task+0x53/0xe0 [ 6720.530363] [<ffffffff8110e726>] bdi_start_fn+0x86/0x100 [ 6720.530369] [<ffffffff8110e6a0>] ? bdi_start_fn+0x0/0x100 [ 6720.530375] [<ffffffff81084f86>] kthread+0x96/0xa0 [ 6720.530383] [<ffffffff810141ea>] child_rip+0xa/0x20 [ 6720.530389] [<ffffffff81084ef0>] ? kthread+0x0/0xa0 [ 6720.530395] [<ffffffff810141e0>] ? child_rip+0x0/0x20 [ 6720.530400] INFO: task kjournald:459 blocked for more than 120 seconds. [ 6720.534113] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 6720.541964] kjournald D 00000000ffffffff 0 459 2 0x00000000 [ 6720.541973] ffff880193a2bc30 0000000000000046 0000000000015b80 0000000000015b80 [ 6720.541981] ffff8801938b9ab0 ffff880193a2bfd8 0000000000015b80 ffff8801938b96f0 [ 6720.541995] 0000000000015b80 ffff880193a2bfd8 0000000000015b80 ffff8801938b9ab0 [ 6720.542014] Call Trace: [ 6720.542026] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.542037] [<ffffffff8153e697>] io_schedule+0x47/0x70 [ 6720.542050] [<ffffffff8116c775>] sync_buffer+0x45/0x50 [ 6720.542061] [<ffffffff8153eeef>] __wait_on_bit+0x5f/0x90 [ 6720.542074] [<ffffffff8116b4f1>] ? submit_bh+0x111/0x140 [ 6720.542086] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.542097] [<ffffffff8153ef98>] out_of_line_wait_on_bit+0x78/0x90 [ 6720.542110] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40 [ 6720.542122] [<ffffffff8116c726>] __wait_on_buffer+0x26/0x30 [ 6720.542136] [<ffffffff81212e0b>] journal_commit_transaction+0x86b/0xe90 [ 6720.542152] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10 [ 6720.542164] [<ffffffff81076e0c>] ? lock_timer_base+0x3c/0x70 [ 6720.542175] [<ffffffff81077895>] ? try_to_del_timer_sync+0x75/0xd0 [ 6720.542189] [<ffffffff812167dd>] kjournald+0xed/0x250 [ 6720.542201] [<ffffffff81085300>] ? autoremove_wake_function+0x0/0x40 [ 6720.542214] [<ffffffff812166f0>] ? kjournald+0x0/0x250 [ 6720.542225] [<ffffffff81084f86>] kthread+0x96/0xa0 [ 6720.542236] [<ffffffff810141ea>] child_rip+0xa/0x20 [ 6720.542248] [<ffffffff81084ef0>] ? kthread+0x0/0xa0 [ 6720.542259] [<ffffffff810141e0>] ? child_rip+0x0/0x20 [ 6720.542280] INFO: task openvpn:1591 blocked for more than 120 seconds. [ 6720.546980] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 6720.556669] openvpn D 0000000000000000 0 1591 1 0x00000000 [ 6720.556677] ffff880186b17918 0000000000000082 0000000000015b80 0000000000015b80 [ 6720.556685] ffff880186e7df80 ffff880186b17fd8 0000000000015b80 ffff880186e7dbc0 [ 6720.556703] 0000000000015b80 ffff880186b17fd8 0000000000015b80 ffff880186e7df80 [ 6720.556723] Call Trace: [ 6720.556731] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.556737] [<ffffffff8153e697>] io_schedule+0x47/0x70 [ 6720.556744] [<ffffffff8116c775>] sync_buffer+0x45/0x50 [ 6720.556750] [<ffffffff8153ed9a>] __wait_on_bit_lock+0x5a/0xc0 [ 6720.556758] [<ffffffff815406ff>] ? _spin_lock_irqsave+0x2f/0x40 [ 6720.556764] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6720.556771] [<ffffffff8153ee78>] out_of_line_wait_on_bit_lock+0x78/0x90 [ 6720.556778] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40 [ 6720.556785] [<ffffffff8116b727>] ? __find_get_block_slow+0xb7/0x130 [ 6720.556796] [<ffffffff8116c8f6>] __lock_buffer+0x36/0x40 [ 6720.556808] [<ffffffff81211e34>] do_get_write_access+0x564/0x5e0 [ 6720.556821] [<ffffffff8116c0a6>] ? __getblk+0x36/0x70 [ 6720.556833] [<ffffffff81212041>] journal_get_write_access+0x31/0x50 [ 6720.556847] [<ffffffff811c5b5d>] __ext3_journal_get_write_access+0x2d/0x60 [ 6720.556860] [<ffffffff811b77db>] ext3_reserve_inode_write+0x7b/0xa0 [ 6720.556872] [<ffffffff811b7836>] ext3_mark_inode_dirty+0x36/0x60 [ 6720.556884] [<ffffffff811b79e1>] ext3_dirty_inode+0x61/0xa0 [ 6720.556896] [<ffffffff811651d2>] __mark_inode_dirty+0x42/0x1e0 [ 6720.556909] [<ffffffff8115950b>] file_update_time+0xfb/0x180 [ 6720.556922] [<ffffffff810f5730>] __generic_file_aio_write+0x210/0x470 [ 6720.556934] [<ffffffff811b6bfd>] ? ext3_mark_iloc_dirty+0x1d/0x30 [ 6720.556947] [<ffffffff810f59ff>] generic_file_aio_write+0x6f/0xe0 [ 6720.556960] [<ffffffff811425aa>] do_sync_write+0xfa/0x140 [ 6720.556974] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10 [ 6720.556986] [<ffffffff81085300>] ? autoremove_wake_function+0x0/0x40 [ 6720.556999] [<ffffffff8115b8a7>] ? notify_change+0x237/0x350 [ 6720.557013] [<ffffffff81250796>] ? security_file_permission+0x16/0x20 [ 6720.557026] [<ffffffff811428a8>] vfs_write+0xb8/0x1a0 [ 6720.557038] [<ffffffff81143141>] sys_write+0x51/0x80 [ 6720.557051] [<ffffffff81540fce>] ? do_device_not_available+0xe/0x10 [ 6720.557066] [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b [ 6840.550040] INFO: task flush-9:2:454 blocked for more than 120 seconds. [ 6840.555547] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 6840.567307] flush-9:2 D 0000000000000000 0 454 2 0x00000000 [ 6840.567317] ffff8801938578d0 0000000000000046 0000000000015b80 0000000000015b80 [ 6840.567327] ffff880193859ab0 ffff880193857fd8 0000000000015b80 ffff8801938596f0 [ 6840.567335] 0000000000015b80 ffff880193857fd8 0000000000015b80 ffff880193859ab0 [ 6840.567344] Call Trace: [ 6840.567362] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6840.567373] [<ffffffff8153e697>] io_schedule+0x47/0x70 [ 6840.567383] [<ffffffff8116c775>] sync_buffer+0x45/0x50 [ 6840.567390] [<ffffffff8153ed9a>] __wait_on_bit_lock+0x5a/0xc0 [ 6840.567397] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6840.567405] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6840.567417] [<ffffffff8153ee78>] out_of_line_wait_on_bit_lock+0x78/0x90 [ 6840.567433] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40 [ 6840.567445] [<ffffffff8116c8f6>] __lock_buffer+0x36/0x40 [ 6840.567458] [<ffffffff8116d644>] __block_write_full_page+0x374/0x3a0 [ 6840.567472] [<ffffffff810f39e7>] ? unlock_page+0x27/0x30 [ 6840.567485] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6840.567498] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6840.567511] [<ffffffff8116dfd0>] block_write_full_page_endio+0xe0/0x120 [ 6840.567524] [<ffffffff8116cb20>] ? end_buffer_async_write+0x0/0x180 [ 6840.567537] [<ffffffff8116e025>] block_write_full_page+0x15/0x20 [ 6840.567551] [<ffffffff811b636d>] ext3_ordered_writepage+0x1dd/0x200 [ 6840.567565] [<ffffffff810fb907>] __writepage+0x17/0x40 [ 6840.567578] [<ffffffff810fcac7>] write_cache_pages+0x227/0x4d0 [ 6840.567591] [<ffffffff810fb8f0>] ? __writepage+0x0/0x40 [ 6840.567605] [<ffffffff810fcd94>] generic_writepages+0x24/0x30 [ 6840.567617] [<ffffffff810fcdd5>] do_writepages+0x35/0x40 [ 6840.567629] [<ffffffff81164b66>] writeback_single_inode+0xf6/0x3d0 [ 6840.567641] [<ffffffff811657d0>] writeback_inodes_wb+0x410/0x5e0 [ 6840.567653] [<ffffffff81165aaa>] wb_writeback+0x10a/0x1d0 [ 6840.567666] [<ffffffff81077895>] ? try_to_del_timer_sync+0x75/0xd0 [ 6840.567678] [<ffffffff8153eb7b>] ? schedule_timeout+0x19b/0x300 [ 6840.567690] [<ffffffff81165ddc>] wb_do_writeback+0x18c/0x1a0 [ 6840.567702] [<ffffffff81165e43>] bdi_writeback_task+0x53/0xe0 [ 6840.567715] [<ffffffff8110e726>] bdi_start_fn+0x86/0x100 [ 6840.567727] [<ffffffff8110e6a0>] ? bdi_start_fn+0x0/0x100 [ 6840.567739] [<ffffffff81084f86>] kthread+0x96/0xa0 [ 6840.567752] [<ffffffff810141ea>] child_rip+0xa/0x20 [ 6840.567763] [<ffffffff81084ef0>] ? kthread+0x0/0xa0 [ 6840.567775] [<ffffffff810141e0>] ? child_rip+0x0/0x20 [ 6840.567783] INFO: task kjournald:459 blocked for more than 120 seconds. [ 6840.574587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 6840.588501] kjournald D 00000000ffffffff 0 459 2 0x00000000 [ 6840.588510] ffff880193a2bc30 0000000000000046 0000000000015b80 0000000000015b80 [ 6840.588519] ffff8801938b9ab0 ffff880193a2bfd8 0000000000015b80 ffff8801938b96f0 [ 6840.588534] 0000000000015b80 ffff880193a2bfd8 0000000000015b80 ffff8801938b9ab0 [ 6840.588554] Call Trace: [ 6840.588565] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6840.588577] [<ffffffff8153e697>] io_schedule+0x47/0x70 [ 6840.588590] [<ffffffff8116c775>] sync_buffer+0x45/0x50 [ 6840.588601] [<ffffffff8153eeef>] __wait_on_bit+0x5f/0x90 [ 6840.588613] [<ffffffff8116c730>] ? sync_buffer+0x0/0x50 [ 6840.588626] [<ffffffff8153ef98>] out_of_line_wait_on_bit+0x78/0x90 [ 6840.588638] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40 [ 6840.588651] [<ffffffff8116c726>] __wait_on_buffer+0x26/0x30 [ 6840.588665] [<ffffffff81212a1e>] journal_commit_transaction+0x47e/0xe90 [ 6840.588678] [<ffffffff81076e0c>] ? lock_timer_base+0x3c/0x70 [ 6840.588690] [<ffffffff81077895>] ? try_to_del_timer_sync+0x75/0xd0 [ 6840.588704] [<ffffffff812167dd>] kjournald+0xed/0x250 [ 6840.588717] [<ffffffff81085300>] ? autoremove_wake_function+0x0/0x40 [ 6840.588729] [<ffffffff812166f0>] ? kjournald+0x0/0x250 [ 6840.588741] [<ffffffff81084f86>] kthread+0x96/0xa0 [ 6840.588753] [<ffffffff810141ea>] child_rip+0xa/0x20 [ 6840.588765] [<ffffffff81084ef0>] ? kthread+0x0/0xa0 [ 6840.588776] [<ffffffff810141e0>] ? child_rip+0x0/0x20 [ 7045.284267] md: md3: resync done. [ 7045.339937] RAID1 conf printout: [ 7045.339941] --- wd:2 rd:2 [ 7045.339948] disk 0, wo:0, o:1, dev:sda4 [ 7045.339952] disk 1, wo:0, o:1, dev:sdb4 [ 7281.917047] kvm: emulating exchange as write [ 7317.183996] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7318.101071] Netfilter messages via NETLINK v0.30. [ 7318.135548] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 7318.136318] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use [ 7318.136323] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or [ 7318.136327] sysctl net.netfilter.nf_conntrack_acct=1 to enable it. [ 7318.400018] ctnetlink v0.93: registering with nfnetlink. [ 7318.563756] ClusterIP Version 0.8 loaded successfully [ 7319.132532] xt_time: kernel timezone is +0200 [ 7319.668678] u32 classifier [ 7319.668684] Actions configured Whatever that causes trouble. I do not knw what is the source for it. Software raid, lvm, kvm, scheduler, no idea. If you need more info, please let me know. And by the way: Happy easter days folks :-) -- Impossible to start KVM from LVM volumes https://bugs.launchpad.net/bugs/555067 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
