Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
On 10/25/2013 08:24 PM, Lei Li wrote: On 10/25/2013 03:30 PM, Paolo Bonzini wrote: Il 25/10/2013 06:58, Lei Li ha scritto: Right now just has inaccurate numbers without the new vmsplice, which based on the result from info migrate, as the guest ram size increases, although the 'total time' is number of times less compared with the current live migration, but the 'downtime' performs badly. Of course. For a 1GB ram guest, total time: 702 milliseconds downtime: 692 milliseconds And when the ram size of guest increasesexponentially, those numbers are proportional to it. I will make a list of the performance with the new vmsplice later, I am sure it'd be much better than this at least. Yes, please. Is the memory usage is still 2x without vmsplice? I think you have a nice proof of concept, but on the other hand this probably needs to be coupled with some kind of postcopy live migration, that is: * the source starts sending data * but the destination starts running immediately * if the machine needs a page that is missing, the destination asks the source to send it * as soon as it arrives, the destination can restart Using postcopy is problematic for reliability: if the destination fails, the virtual machine is lost because the source doesn't have the latest content of memory. However, this is a much, much smaller problem for live QEMU upgrade where the network cannot fail. If you do this, you can achieve pretty much instantaneous live upgrade, well within your original 200 ms goals. But the flipping code with vmsplice should be needed anyway to avoid doubling memory usage, and Yes, I have read the postcopy migration patches, it does perform very good on downtime, as just send the vmstates then switch the execution to destination host. And as you pointed out, it can not avoid doubling memory usage. The numbers list above are based on the old vmsplice as I have not yet worked on the benchmark for performance, it actually copys data rather than moving. As the feedback for this version is positive, now I am trying to get a real result out with the new vmsplice. BTW, kernel side is looking for huge page solution for the improvement of performance. The recently patches from kernel as link, http://article.gmane.org/gmane.linux.kernel/1574277 Hi Paolo, I have been working on the benchmark of the performance, I am afraid that it may take a bit more time as there has some problems on the new vmsplice which kernel side is working on right now. I will post a v3 of the series with your comments in previous version fixed soon. it's looking pretty good in this version already! I'm relieved that the RDMA code was designed right! I am happy with it too. :) Those RDMA hooks really make thingsmore flexible! Paolo -- Lei
Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
Il 25/10/2013 06:58, Lei Li ha scritto: Right now just has inaccurate numbers without the new vmsplice, which based on the result from info migrate, as the guest ram size increases, although the 'total time' is number of times less compared with the current live migration, but the 'downtime' performs badly. Of course. For a 1GB ram guest, total time: 702 milliseconds downtime: 692 milliseconds And when the ram size of guest increasesexponentially, those numbers are proportional to it. I will make a list of the performance with the new vmsplice later, I am sure it'd be much better than this at least. Yes, please. Is the memory usage is still 2x without vmsplice? I think you have a nice proof of concept, but on the other hand this probably needs to be coupled with some kind of postcopy live migration, that is: * the source starts sending data * but the destination starts running immediately * if the machine needs a page that is missing, the destination asks the source to send it * as soon as it arrives, the destination can restart Using postcopy is problematic for reliability: if the destination fails, the virtual machine is lost because the source doesn't have the latest content of memory. However, this is a much, much smaller problem for live QEMU upgrade where the network cannot fail. If you do this, you can achieve pretty much instantaneous live upgrade, well within your original 200 ms goals. But the flipping code with vmsplice should be needed anyway to avoid doubling memory usage, and it's looking pretty good in this version already! I'm relieved that the RDMA code was designed right! Paolo
Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
On Oct 25, 2013 8:30 AM, Paolo Bonzini pbonz...@redhat.com wrote: Il 25/10/2013 06:58, Lei Li ha scritto: Right now just has inaccurate numbers without the new vmsplice, which based on the result from info migrate, as the guest ram size increases, although the 'total time' is number of times less compared with the current live migration, but the 'downtime' performs badly. Of course. For a 1GB ram guest, total time: 702 milliseconds downtime: 692 milliseconds And when the ram size of guest increasesexponentially, those numbers are proportional to it. I will make a list of the performance with the new vmsplice later, I am sure it'd be much better than this at least. Yes, please. Is the memory usage is still 2x without vmsplice? I think you have a nice proof of concept, but on the other hand this probably needs to be coupled with some kind of postcopy live migration, that is: * the source starts sending data * but the destination starts running immediately * if the machine needs a page that is missing, the destination asks the source to send it * as soon as it arrives, the destination can restart Using postcopy is problematic for reliability: if the destination fails, the virtual machine is lost because the source doesn't have the latest content of memory. However, this is a much, much smaller problem for live QEMU upgrade where the network cannot fail. If you do this, you can achieve pretty much instantaneous live upgrade, well within your original 200 ms goals. This is actually a very nice justification for post copy. Regards, Anthony Liguori But the flipping code with vmsplice should be needed anyway to avoid doubling memory usage, and it's looking pretty good in this version already! I'm relieved that the RDMA code was designed right! Paolo
Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
On 10/25/2013 03:30 PM, Paolo Bonzini wrote: Il 25/10/2013 06:58, Lei Li ha scritto: Right now just has inaccurate numbers without the new vmsplice, which based on the result from info migrate, as the guest ram size increases, although the 'total time' is number of times less compared with the current live migration, but the 'downtime' performs badly. Of course. For a 1GB ram guest, total time: 702 milliseconds downtime: 692 milliseconds And when the ram size of guest increasesexponentially, those numbers are proportional to it. I will make a list of the performance with the new vmsplice later, I am sure it'd be much better than this at least. Yes, please. Is the memory usage is still 2x without vmsplice? I think you have a nice proof of concept, but on the other hand this probably needs to be coupled with some kind of postcopy live migration, that is: * the source starts sending data * but the destination starts running immediately * if the machine needs a page that is missing, the destination asks the source to send it * as soon as it arrives, the destination can restart Using postcopy is problematic for reliability: if the destination fails, the virtual machine is lost because the source doesn't have the latest content of memory. However, this is a much, much smaller problem for live QEMU upgrade where the network cannot fail. If you do this, you can achieve pretty much instantaneous live upgrade, well within your original 200 ms goals. But the flipping code with vmsplice should be needed anyway to avoid doubling memory usage, and Yes, I have read the postcopy migration patches, it does perform very good on downtime, as just send the vmstates then switch the execution to destination host. And as you pointed out, it can not avoid doubling memory usage. The numbers list above are based on the old vmsplice as I have not yet worked on the benchmark for performance, it actually copys data rather than moving. As the feedback for this version is positive, now I am trying to get a real result out with the new vmsplice. BTW, kernel side is looking for huge page solution for the improvement of performance. The recently patches from kernel as link, http://article.gmane.org/gmane.linux.kernel/1574277 it's looking pretty good in this version already! I'm relieved that the RDMA code was designed right! I am happy with it too. :) Those RDMA hooks really make thingsmore flexible! Paolo -- Lei
Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
Il 22/10/2013 04:25, Lei Li ha scritto: This patch series tries to introduce a mechanism using side channel pipe for RAM via SCM_RIGHTS with unix domain socket protocol migration. This side channel is used for the page flipping by vmsplice, which is the internal mechanism for localhost migration that we are trying to add to QEMU. The backgroud info and previous patch series for reference, Localhost migration http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html migration: Introduce side channel for RAM http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04043.html I have picked patches from the localhost migration series and rebased it on the series of side channel, now it is a complete series that passed the basic test. Please let me know if there is anything needs to be fixed or improved. Your suggestions and comments are very welcome, and thanks for Paolo for his review and useful suggestions. Thanks to you for listening. :) What is performance like? How much downtime do you get as the guest size increases? Paolo
Re: [Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
On 10/24/2013 10:17 PM, Paolo Bonzini wrote: Il 22/10/2013 04:25, Lei Li ha scritto: This patch series tries to introduce a mechanism using side channel pipe for RAM via SCM_RIGHTS with unix domain socket protocol migration. This side channel is used for the page flipping by vmsplice, which is the internal mechanism for localhost migration that we are trying to add to QEMU. The backgroud info and previous patch series for reference, Localhost migration http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html migration: Introduce side channel for RAM http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04043.html I have picked patches from the localhost migration series and rebased it on the series of side channel, now it is a complete series that passed the basic test. Please let me know if there is anything needs to be fixed or improved. Your suggestions and comments are very welcome, and thanks for Paolo for his review and useful suggestions. Thanks to you for listening. :) What is performance like? How much downtime do you get as the guest size increases? Hi Paolo, Thanks very much for your continued reviews, as well as those nice suggestions and comments! For the performance, we have not yet benchmarked it from QEMU layer, as now it's just an initial implementation, and Robert Jennings has been worked on the improvement of performance on kernel side. Right now just has inaccurate numbers without the new vmsplice, which based on the result from info migrate, as the guest ram size increases, although the 'total time' is number of times less compared with the current live migration, but the 'downtime' performs badly. For a 1GB ram guest, total time: 702 milliseconds downtime: 692 milliseconds And when the ram size of guest increasesexponentially, those numbers are proportional to it. I will make a list of the performance with the new vmsplice later, I am sure it'd be much better than this at least. Paolo -- Lei
[Qemu-devel] [PATCH 0/17 v2] Localhost migration with side channel for ram
This patch series tries to introduce a mechanism using side channel pipe for RAM via SCM_RIGHTS with unix domain socket protocol migration. This side channel is used for the page flipping by vmsplice, which is the internal mechanism for localhost migration that we are trying to add to QEMU. The backgroud info and previous patch series for reference, Localhost migration http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html migration: Introduce side channel for RAM http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04043.html I have picked patches from the localhost migration series and rebased it on the series of side channel, now it is a complete series that passed the basic test. Please let me know if there is anything needs to be fixed or improved. Your suggestions and comments are very welcome, and thanks for Paolo for his review and useful suggestions. Changes since V1: Address suggestions from Paolo Bonzini including: - Use Unix socket QEMUFile as basis of code and adjust the way of overriding RDMA hooks. - Involve the vmsplice for page flipping. - Add new RunState RUN_STATE_FLIPPING_MIGRATE and add it to runstate_needs_reset() for the adjustment of the current migration process with page flipping. Lei Li (17): rename is_active to is_block_active QAPI: introduce magration capability unix_page_flipping migration: add migrate_unix_page_flipping() qmp-command.hx: add missing docs for migration capabilites migration-local: add QEMUFileLocal with socket based QEMUFile migration-local: introduce qemu_fopen_socket_local() migration-local: add send_pipefd() migration-local: add recv_pipefd() migration-local: override before_ram_iterate to send pipefd savevm: adjust ram_control_save_page with page flipping migration-local: override save_page for page transmit migration-local: override hook_ram_load migration-unix: replace qemu_fopen_socket with qemu_fopen_socket_local add new RanState RAN_STATE_FLIPPING_MIGRATE migration-unix: page flipping support on unix outgoing migration: adjust migration_thread() process for unix_page_flipping hmp: better fomat for info migrate_capabilities Makefile.target | 1 + block-migration.c | 2 +- migration-local.c | 512 ++ hmp.c | 5 +- include/migration/migration.h | 3 + include/migration/qemu-file.h | 2 + include/migration/vmstate.h | 2 +- migration-unix.c | 27 ++- migration.c | 18 +- qapi-schema.json | 18 +- qmp-commands.hx | 8 + savevm.c | 21 +- vl.c | 12 +- 13 files changed, 624 insertions(+), 27 deletions(-) create mode 100644 migration-local.c
