>>> Greg Woods <[email protected]> schrieb am 23.05.2013 um 21:45 in Nachricht
<[email protected]>:
> On Thu, 2013-05-23 at 15:00 -0400, David Vossel wrote:
>> Migration time, depending on network speed and hardware, is much longer
> than the shared storage option (minutes vs. seconds).
>
>
> This is just one data point (of course), but for the vast majority of
> services that I run, if the live migration time is as long as it takes
> to shut down a VM and boot it on another server, then there isn't much
> of an advantage to doing the live migration. Especially if we're talking
> about an option that is a long way from being battle-tested, and
> critical services such as DNS and authentication. Most of these critical
> services do not use long-lived connections.
I tend to disagree: The time it takes to complete live migration is less
important than the time of the virtual stand-still. Agreed, of the virtual
stand-still exceeds 30 Seconds, most busy TCP connections will be dropped
anyway. There are services out in the field that take minutes to shutdown or to
start up. Added to the reboot time the total downtime can be significant. Here
a few seconds of virtual downtime may be preferred.
Unfortunately in Xen paravirtualization the kernel has a problem if time is
stopped virtuelly for some seconds:
---
May 18 12:28:06 v02 kernel: [2494407.076783] PM: suspend of devices complete
after 0.206 msecs
May 18 12:28:06 v02 kernel: [2494407.076923] PM: late suspend of devices
complete after 0.111 msecs
May 18 12:28:06 v02 kernel: [2494407.076949] PM: early resume of devices
complete after 0.044 msecs
May 18 12:28:06 v02 kernel: [2494407.088911] Unable to read sysrq code in
control/sysrq
May 18 12:28:06 v02 kernel: [2494407.106669] Setting capacity to 6564928
May 18 12:28:06 v02 kernel: [2494407.128504] Setting capacity to 62914416
May 18 12:28:06 v02 kernel: [2494407.151298] Setting capacity to 62914560
May 18 12:28:06 v02 kernel: [2494407.186533] Setting capacity to 62914560
May 18 12:28:06 v02 kernel: [2494407.194052] Setting capacity to 1031798784
May 18 12:28:06 v02 kernel: [2494407.207848] Setting capacity to 1031798784
May 18 12:28:06 v02 kernel: [2494407.240368] Setting capacity to 20971520
May 18 12:28:06 v02 kernel: [2494407.240483] Setting capacity to 20971520
May 18 12:28:07 v02 kernel: [2494407.254029] PM: resume of devices complete
after 0.062 msecs
May 18 12:28:07 v02 kernel: [2494407.254051] suspend: event channel 22
May 18 12:28:07 v02 kernel: [2494407.277073] Setting capacity to 41943040
May 18 12:28:07 v02 kernel: [2494407.279750] Setting capacity to 41943040
May 18 12:28:07 v02 kernel: [2494407.285265] Setting capacity to 1048576000
May 18 12:29:06 v02 kernel: [2494467.088064] INFO: rcu_sched_state detected
stalls on CPUs/tasks: { 1} (detected by 0, t=15002 jiffies)
May 18 12:29:06 v02 kernel: [2494467.088072] sending NMI to all CPUs:
May 18 12:29:06 v02 kernel: [2494467.088078] NMI backtrace for cpu 0
May 18 12:29:06 v02 kernel: [2494467.088080] CPU 0
May 18 12:29:06 v02 kernel: [2494467.088082] Modules linked in: binfmt_misc
ip6t_LOG xt_tcpudp xt_pkttype ipt_LOG xt_limit nfs lockd fscache auth_rpcgss
nfs_acl sunrpc ip6t_REJECT nf_conntrack_i
pv6 nf_defrag_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT xt_state iptable_raw
iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4
ip_tables ip6table_filter ip6_tables x_tables fuse xfs loop raid1 ipv6 ipv6_lib
dm_mirror dm_region_hash dm_log linear scsi_dh_alua scsi_dh_emc scsi_dh_rdac
scsi_dh_hp_sw scsi_dh scsi_mod dm_snapshot
dm_mod xennet ext3 mbcache jbd processor thermal_sys hwmon xenblk cdrom
May 18 12:29:06 v02 kernel: [2494467.088133] Supported: Yes
May 18 12:29:06 v02 kernel: [2494467.088136]
May 18 12:29:06 v02 kernel: [2494467.088140] Pid: 0, comm: swapper Not tainted
3.0.58-0.6.6-xen #1
[...]
May 18 12:29:07 v02 kernel: [2494467.088274] Call Trace:
May 18 12:29:07 v02 kernel: [2494467.088296] [<ffffffff802dea9c>]
notify_remote_via_ipi+0x4c/0xa0
May 18 12:29:07 v02 kernel: [2494467.088306] [<ffffffff8001811a>]
xen_send_IPI_mask+0x3a/0x70
May 18 12:29:07 v02 kernel: [2494467.088314] [<ffffffff8001828a>]
arch_trigger_all_cpu_backtrace+0x8a/0xd0
May 18 12:29:07 v02 kernel: [2494467.088322] [<ffffffff800ac5de>]
print_other_cpu_stall+0x13e/0x180
May 18 12:29:07 v02 kernel: [2494467.088329] [<ffffffff800ac68f>]
__rcu_pending+0x6f/0x260
May 18 12:29:07 v02 kernel: [2494467.088334] [<ffffffff800ac8df>]
rcu_check_callbacks+0x5f/0x110
May 18 12:29:07 v02 kernel: [2494467.088343] [<ffffffff80054f5f>]
update_process_times+0x3f/0x80
May 18 12:29:07 v02 kernel: [2494467.088352] [<ffffffff80078b6b>]
tick_sched_timer+0x5b/0xc0
May 18 12:29:07 v02 kernel: [2494467.088360] [<ffffffff8006c6ee>]
__run_hrtimer+0xde/0x1d0
May 18 12:29:07 v02 kernel: [2494467.088366] [<ffffffff8006ca57>]
hrtimer_interrupt+0xe7/0x270
May 18 12:29:07 v02 kernel: [2494467.088374] [<ffffffff802e5617>]
timer_interrupt+0x27/0x100
May 18 12:29:07 v02 kernel: [2494467.088382] [<ffffffff800a5c32>]
handle_irq_event_percpu+0x62/0x1c0
May 18 12:29:07 v02 kernel: [2494467.088389] [<ffffffff800a88d2>]
handle_percpu_irq+0x42/0x80
May 18 12:29:07 v02 kernel: [2494467.088399] [<ffffffff8000840a>]
handle_irq+0x1a/0x30
May 18 12:29:07 v02 kernel: [2494467.088405] [<ffffffff802def97>]
evtchn_do_upcall+0x1c7/0x300
May 18 12:29:07 v02 kernel: [2494467.088414] [<ffffffff80402e3e>]
do_hypervisor_callback+0x1e/0x30
May 18 12:29:07 v02 kernel: [2494467.088421] [<ffffffff800033aa>]
0xffffffff800033a9
May 18 12:29:07 v02 kernel: [2494467.088434] [<ffffffff8000dba7>]
xen_idle+0xb7/0x180
May 18 12:29:07 v02 kernel: [2494467.088440] [<ffffffff80006537>]
cpu_idle+0x67/0xc0
May 18 12:29:07 v02 kernel: [2494467.088450] [<ffffffff80728baf>]
start_kernel+0x350/0x41f
May 18 12:29:07 v02 kernel: [2494467.088454] Code: cc 51 41 53 50 b8 17 00 00
00 0f 05 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41
53 b8 18 00 00 00 0f 05
May 18 12:29:07 v02 kernel: <41>[2494467.088505] 5b 59 c3 cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc
May 18 12:29:07 v02 kernel: [2494467.088518] Call Trace:
May 18 12:29:07 v02 kernel: [2494467.088527] [<ffffffff802dea9c>]
notify_remote_via_ipi+0x4c/0xa0
May 18 12:29:07 v02 kernel: [2494467.088533] [<ffffffff8001811a>]
xen_send_IPI_mask+0x3a/0x70
May 18 12:29:07 v02 kernel: [2494467.088539] [<ffffffff8001828a>]
arch_trigger_all_cpu_backtrace+0x8a/0xd0
May 18 12:29:07 v02 kernel: [2494467.088544] [<ffffffff800ac5de>]
print_other_cpu_stall+0x13e/0x180
May 18 12:29:07 v02 kernel: [2494467.088550] [<ffffffff800ac68f>]
__rcu_pending+0x6f/0x260
May 18 12:29:07 v02 kernel: [2494467.088556] [<ffffffff800ac8df>]
rcu_check_callbacks+0x5f/0x110
May 18 12:29:07 v02 kernel: [2494467.088562] [<ffffffff80054f5f>]
update_process_times+0x3f/0x80
May 18 12:29:07 v02 kernel: [2494467.088568] [<ffffffff80078b6b>]
tick_sched_timer+0x5b/0xc0
May 18 12:29:07 v02 kernel: [2494467.088574] [<ffffffff8006c6ee>]
__run_hrtimer+0xde/0x1d0
May 18 12:29:07 v02 kernel: [2494467.088580] [<ffffffff8006ca57>]
hrtimer_interrupt+0xe7/0x270
May 18 12:29:07 v02 kernel: [2494467.088586] [<ffffffff802e5617>]
timer_interrupt+0x27/0x100
May 18 12:29:07 v02 kernel: [2494467.088592] [<ffffffff800a5c32>]
handle_irq_event_percpu+0x62/0x1c0
May 18 12:29:07 v02 kernel: [2494467.088598] [<ffffffff800a88d2>]
handle_percpu_irq+0x42/0x80
May 18 12:29:07 v02 kernel: [2494467.088604] [<ffffffff8000840a>]
handle_irq+0x1a/0x30
May 18 12:29:07 v02 kernel: [2494467.088609] [<ffffffff802def97>]
evtchn_do_upcall+0x1c7/0x300
May 18 12:29:07 v02 kernel: [2494467.088615] [<ffffffff80402e3e>]
do_hypervisor_callback+0x1e/0x30
May 18 12:29:07 v02 kernel: [2494467.088622] [<ffffffff800033aa>]
0xffffffff800033a9
May 18 12:29:07 v02 kernel: [2494467.088633] [<ffffffff8000dba7>]
xen_idle+0xb7/0x180
May 18 12:29:07 v02 kernel: [2494467.088638] [<ffffffff80006537>]
cpu_idle+0x67/0xc0
May 18 12:29:07 v02 kernel: [2494467.088644] [<ffffffff80728baf>]
start_kernel+0x350/0x41f
[...]
---
>
> I can see a few VMs that exist to provide ssh logins where a
> minutes-long live migration would be clearly preferable to a shut down
> and reboot, but in most cases, if it's as slow as rebooting, it isn't
> going to be any advantage to me.
You are still mixing total migration time (which may be minutes) with virtual
stand-still time (which is a few seconds).
>
> It will be interesting though to see how many applications people come
> up with where a minutes-long live migration is preferable to shutdown
> and reboot.
The SAP Java Stack takes more than 5 minutes for a "clean" shutdown (Note: I'm
not saying it MUST take that long; I'm just stating that is DOES). If you
perform that procedure during normal host shutdown, the host shutdown alone
will take more than 6 minutes. If you add a minute for rebooting the OS plus a
few minutes to start the application again, you easily get 10 minutes of
downtime (not to talk about dropped connections). Now if your system is just
standing still for say 20 seconds, thats an improvement.
Regards,
Ulrich
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
