Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
Hi, I would like to add some more points to Gerd's explanation: On 06/05/2012 04:15 PM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Actually, this is not the first time we address you with this issues. For example: http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg01805.html (The first part of the above discussion is not directly related to the current one). I'll try to explain in more details: As Gerd mentioned, migrating the spice connection smoothly requires the src server to keep running and send/receive data to/from the client, after migration has already completed, till the client completely transfers to the target. The suggested patch series only delays the migration state change from ACTIVE to COMPLETED/ERROR/CANCELED, till spice signals it has completed its part in migration. As I see it, if spice connection does exists, its migration should be treated as a non separate part of the whole migration process, and thus, the migration state shouldn't change from ACTIVE, till spice has completed its part. Hence, I don't think we should have a qmp event for signaling libvirt about spice migration. The second challenge we are facing, which I addressed in the plans part of the cover-letter, and on which I think you (anthony) actually replied, is how to tackle migrating spice data from the src server to the target server. Such data can be usb/smartcard packets sent from a device connected on the client, to the server, and that haven't reached the device. Or partial data that has been read from a guest character device and that haven't been sent to the client. Other data can be internal server-client state data we would wish to keep on the server in order to avoid establishing the connection to the target from scratch, and possibly also suffer from a slower responsiveness at start. In the cover-letter I suggested to transfer spice migration data via the vmstate infrastructure. The other alternative which we also discussed in the link above, is to transfer the data via the client. The latter also requires holding the src process alive after migration completion, in order to manage to complete transferring the data from the src to the client. The vmstate option has the advantages of faster data transfer (src-dst, instead of src-client-dst), and in addition employing an already existing reliable mechanism for data migration. The disadvantage is that in order to have an updated vmstate we need to communicate with spice client and get all in-flight data before saving the vmstate. So, we can either busy wait on the relevant fds during the pre_save of the vmstates, or have async pre_save, so that the main loop will be active (but I think that it can be risky once the non-live phase started), or have an async notifier for changing from live to non-live phase, (spice will be able to update the vmstates during this notification handler). Of course, we would in any case use a timeout in order to prevent too long delay. To summarize, since we can still use the client to transfer data from the src to the target (instead of using vmstate), the major requirement of spice, is to keep the src running after migration has completed. Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt.
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 05:10 PM, Yonit Halperin wrote: Hi, I would like to add some more points to Gerd's explanation: On 06/05/2012 04:15 PM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Actually, this is not the first time we address you with this issues. For example: http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg01805.html (The first part of the above discussion is not directly related to the current one). I'll try to explain in more details: As Gerd mentioned, migrating the spice connection smoothly requires the src server to keep running and send/receive data to/from the client, after migration has already completed, till the client completely transfers to the target. The suggested patch series only delays the migration state change from ACTIVE to COMPLETED/ERROR/CANCELED, till spice signals it has completed its part in migration. As I see it, if spice connection does exists, its migration should be treated as a non separate part of the whole migration process, and thus, the migration state shouldn't change from ACTIVE, till spice has completed its part. Hence, I don't think we should have a qmp event for signaling libvirt about spice migration. Spice client migration has nothing to do with guest migration. Trying to abuse QEMU to support it is not acceptable. The second challenge we are facing, which I addressed in the plans part of the cover-letter, and on which I think you (anthony) actually replied, is how to tackle migrating spice data from the src server to the target server. Such data can be usb/smartcard packets sent from a device connected on the client, to the server, and that haven't reached the device. Or partial data that has been read from a guest character device and that haven't been sent to the client. Other data can be internal server-client state data we would wish to keep on the server in order to avoid establishing the connection to the target from scratch, and possibly also suffer from a slower responsiveness at start. In the cover-letter I suggested to transfer spice migration data via the vmstate infrastructure. The other alternative which we also discussed in the link above, is to transfer the data via the client. The latter also requires holding the src process alive after migration completion, in order to manage to complete transferring the data from the src to the client. -- To summarize, since we can still use the client to transfer data from the src to the target (instead of using vmstate), the major requirement of spice, is to keep the src running after migration has completed. So send a QMP event and call it a day. Regards, Anthony Liguori Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On Wed, Jun 06, 2012 at 05:22:21PM +0800, Anthony Liguori wrote: On 06/06/2012 05:10 PM, Yonit Halperin wrote: Hi, I would like to add some more points to Gerd's explanation: On 06/05/2012 04:15 PM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Actually, this is not the first time we address you with this issues. For example: http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg01805.html (The first part of the above discussion is not directly related to the current one). I'll try to explain in more details: As Gerd mentioned, migrating the spice connection smoothly requires the src server to keep running and send/receive data to/from the client, after migration has already completed, till the client completely transfers to the target. The suggested patch series only delays the migration state change from ACTIVE to COMPLETED/ERROR/CANCELED, till spice signals it has completed its part in migration. As I see it, if spice connection does exists, its migration should be treated as a non separate part of the whole migration process, and thus, the migration state shouldn't change from ACTIVE, till spice has completed its part. Hence, I don't think we should have a qmp event for signaling libvirt about spice migration. Spice client migration has nothing to do with guest migration. Trying to I don't understand this POV. If it were a VNC connection instead of a Spice one would it make a difference? If there is an active VNC client then it is there as a result of a user choosing to use it, so it should be treated as part of the user experience and not as something external. The experience from ignoring this and choosing to treat the remote console as an unrelated part is bound to be suboptimal. abuse QEMU to support it is not acceptable. The second challenge we are facing, which I addressed in the plans part of the cover-letter, and on which I think you (anthony) actually replied, is how to tackle migrating spice data from the src server to the target server. Such data can be usb/smartcard packets sent from a device connected on the client, to the server, and that haven't reached the device. Or partial data that has been read from a guest character device and that haven't been sent to the client. Other data can be internal server-client state data we would wish to keep on the server in order to avoid establishing the connection to the target from scratch, and possibly also suffer from a slower responsiveness at start. In the cover-letter I suggested to transfer spice migration data via the vmstate infrastructure. The other alternative which we also discussed in the link above, is to transfer the data via the client. The latter also requires holding the src process alive after migration completion, in order to manage to complete transferring the data from the src to the client. -- To summarize, since we can still use the client to transfer data from the src to the target (instead of using vmstate), the major requirement of spice, is to keep the src running after migration has completed. So send a QMP event and call it a day. Regards, Anthony Liguori Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 06:54 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 05:22:21PM +0800, Anthony Liguori wrote: On 06/06/2012 05:10 PM, Yonit Halperin wrote: Spice client migration has nothing to do with guest migration. Trying to I don't understand this POV. If it were a VNC connection instead of a Spice one would it make a difference? Of course, I would say yes if it was VNC. Because the only possibly way I could disagree with something Spice related is because I'm biased against it. Give me the benefit of the doubt at least. More importantly, try to stop and think about what I'm saying before you assume the anti-Spice brigade is coming in to rain on your parade. If there is an active VNC client then it is there as a result of a user choosing to use it, so it should be treated as part of the user experience and not as something external. The experience from ignoring this and choosing to treat the remote console as an unrelated part is bound to be suboptimal. Guest migration affects correctness! If the Spice client is slow (even due to network lag) in responding to your flush message, you will disrupt the guest and potentially drop network connections and/or cause lockup detectors to trigger. Migrating the Spice client is a UI feature. It has absolutely no affect no the workloads that are running in the guest. Impacting migration *correctness* in order to support a UI feature is unacceptable especially when there are ways to achieve the same results without having any impact on correctness. We have had a simple rule with migration in QEMU. Nothing gets to impact downtime with migration. No device gets magic hooks or anything like that. Go read the TPM threads if you want to see another example of this. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On Wed, Jun 06, 2012 at 07:05:44PM +0800, Anthony Liguori wrote: On 06/06/2012 06:54 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 05:22:21PM +0800, Anthony Liguori wrote: On 06/06/2012 05:10 PM, Yonit Halperin wrote: Spice client migration has nothing to do with guest migration. Trying to I don't understand this POV. If it were a VNC connection instead of a Spice one would it make a difference? Of course, I would say yes if it was VNC. Because the only possibly way I could disagree with something Spice related is because I'm biased against it. Give me the benefit of the doubt at least. More importantly, try to stop and think about what I'm saying before you assume the anti-Spice brigade is coming in to rain on your parade. I stand corrected. If there is an active VNC client then it is there as a result of a user choosing to use it, so it should be treated as part of the user experience and not as something external. The experience from ignoring this and choosing to treat the remote console as an unrelated part is bound to be suboptimal. Guest migration affects correctness! If the Spice client is slow (even due to network lag) in responding to your flush message, you will disrupt the guest and potentially drop network connections and/or cause lockup detectors to trigger. OK, you think any timeout here would be too large. Migrating the Spice client is a UI feature. It has absolutely no affect no the workloads that are running in the guest. Impacting migration *correctness* in order to support a UI feature is unacceptable especially when there are ways to achieve the same results without having any impact on correctness. We have had a simple rule with migration in QEMU. Nothing gets to impact downtime with migration. No device gets magic hooks or anything like that. Go read the TPM threads if you want to see another example of this. OK. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 07:27 PM, Alon Levy wrote: If there is an active VNC client then it is there as a result of a user choosing to use it, so it should be treated as part of the user experience and not as something external. The experience from ignoring this and choosing to treat the remote console as an unrelated part is bound to be suboptimal. Guest migration affects correctness! If the Spice client is slow (even due to network lag) in responding to your flush message, you will disrupt the guest and potentially drop network connections and/or cause lockup detectors to trigger. OK, you think any timeout here would be too large. What would it's value be? Migration is convergent and our downtime estimate is just that--an estimate. It's literally always a crap-shoot as to whether the actual migration will complete fast enough. What do you propose the timeout to be? 1us? Can you even do a round trip to a client in 1us? 50us? I still question whether a round trip is feasible in that time period and you've blown away the default 30us time anyway. Even 1us would be too much though. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 12:22 PM, Anthony Liguori wrote: On 06/06/2012 05:10 PM, Yonit Halperin wrote: Hi, I would like to add some more points to Gerd's explanation: On 06/05/2012 04:15 PM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Actually, this is not the first time we address you with this issues. For example: http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg01805.html (The first part of the above discussion is not directly related to the current one). I'll try to explain in more details: As Gerd mentioned, migrating the spice connection smoothly requires the src server to keep running and send/receive data to/from the client, after migration has already completed, till the client completely transfers to the target. The suggested patch series only delays the migration state change from ACTIVE to COMPLETED/ERROR/CANCELED, till spice signals it has completed its part in migration. As I see it, if spice connection does exists, its migration should be treated as a non separate part of the whole migration process, and thus, the migration state shouldn't change from ACTIVE, till spice has completed its part. Hence, I don't think we should have a qmp event for signaling libvirt about spice migration. Spice client migration has nothing to do with guest migration. Trying to abuse QEMU to support it is not acceptable. The second challenge we are facing, which I addressed in the plans part of the cover-letter, and on which I think you (anthony) actually replied, is how to tackle migrating spice data from the src server to the target server. Such data can be usb/smartcard packets sent from a device connected on the client, to the server, and that haven't reached the device. Or partial data that has been read from a guest character device and that haven't been sent to the client. Other data can be internal server-client state data we would wish to keep on the server in order to avoid establishing the connection to the target from scratch, and possibly also suffer from a slower responsiveness at start. In the cover-letter I suggested to transfer spice migration data via the vmstate infrastructure. The other alternative which we also discussed in the link above, is to transfer the data via the client. The latter also requires holding the src process alive after migration completion, in order to manage to complete transferring the data from the src to the client. -- To summarize, since we can still use the client to transfer data from the src to the target (instead of using vmstate), the major requirement of spice, is to keep the src running after migration has completed. So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. Delaying the status change to migration completed, (1) doesn't affect qemu migration time, the migration has already completed, and (2) will allow spice to seamlessly migrate, no matter which libvirt version is used. Yonit. Regards, Anthony Liguori Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 08:01 PM, Yonit Halperin wrote: On 06/06/2012 12:22 PM, Anthony Liguori wrote: So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. That is not an acceptable justification. Delaying the status change to migration completed, (1) doesn't affect qemu migration time, the migration has already completed, and (2) will allow spice to seamlessly migrate, no matter which libvirt version is used. (1) libvirt starts the destination with -S and starts it manually IIUC. It waits for the migration completed event to do this. Seriously, just add the event. Async notifiers are not an option. Regards, Anthony Liguori Yonit. Regards, Anthony Liguori Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go with a very short timeout there. Option (2): Add a new QMP event which is emmitted when spice-server is done, then make libvirt wait for it before killing qemu. Obvious disadvantage is that it requires libvirt changes. Option (3): Your suggestion? thanks, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On Wed, Jun 06, 2012 at 08:08:40PM +0800, Anthony Liguori wrote: On 06/06/2012 08:01 PM, Yonit Halperin wrote: On 06/06/2012 12:22 PM, Anthony Liguori wrote: So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. That is not an acceptable justification. To let spice know that libvirt doesn't support the new event would require libvirt capabilities advertisement to qemu. Is that acceptable? Delaying the status change to migration completed, (1) doesn't affect qemu migration time, the migration has already completed, and (2) will allow spice to seamlessly migrate, no matter which libvirt version is used. (1) libvirt starts the destination with -S and starts it manually IIUC. It waits for the migration completed event to do this. Seriously, just add the event. Async notifiers are not an option. Regards, Anthony Liguori Yonit. Regards, Anthony Liguori Yonit. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go with a very short timeout there. Option (2): Add a new QMP event which is emmitted when spice-server is done, then make libvirt wait for it before killing qemu. Obvious disadvantage is that it requires libvirt changes. Option (3): Your suggestion? thanks, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 08:15 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 08:08:40PM +0800, Anthony Liguori wrote: On 06/06/2012 08:01 PM, Yonit Halperin wrote: On 06/06/2012 12:22 PM, Anthony Liguori wrote: So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. That is not an acceptable justification. To let spice know that libvirt doesn't support the new event would require libvirt capabilities advertisement to qemu. Is that acceptable? I literally have danpb's event introspection patches from Luiz's PULL request testing on my system right now to be pushed. So this is already a solved problem. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On Wed, Jun 06, 2012 at 08:17:29PM +0800, Anthony Liguori wrote: On 06/06/2012 08:15 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 08:08:40PM +0800, Anthony Liguori wrote: On 06/06/2012 08:01 PM, Yonit Halperin wrote: On 06/06/2012 12:22 PM, Anthony Liguori wrote: So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. That is not an acceptable justification. To let spice know that libvirt doesn't support the new event would require libvirt capabilities advertisement to qemu. Is that acceptable? I literally have danpb's event introspection patches from Luiz's PULL request testing on my system right now to be pushed. this?: [PATCH 29/29] Add 'query-events' command to QMP to query async events This is about libvirt getting qemu's event list. I am talking about qemu getting libvirt to say we support event SPICE_MIGRATE_DONE. i.e. Yet another capability negotiation, during the handshake QMP phase. So this is already a solved problem. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/06/2012 08:30 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 08:17:29PM +0800, Anthony Liguori wrote: On 06/06/2012 08:15 PM, Alon Levy wrote: On Wed, Jun 06, 2012 at 08:08:40PM +0800, Anthony Liguori wrote: On 06/06/2012 08:01 PM, Yonit Halperin wrote: On 06/06/2012 12:22 PM, Anthony Liguori wrote: So send a QMP event and call it a day. Using a QMP event is making spice seamless migration dependent on libvirt version. That is not an acceptable justification. To let spice know that libvirt doesn't support the new event would require libvirt capabilities advertisement to qemu. Is that acceptable? I literally have danpb's event introspection patches from Luiz's PULL request testing on my system right now to be pushed. this?: [PATCH 29/29] Add 'query-events' command to QMP to query async events This is about libvirt getting qemu's event list. I am talking about qemu getting libvirt to say we support event SPICE_MIGRATE_DONE. i.e. Yet another capability negotiation, during the handshake QMP phase. qemu -spice foo,seamless-migration=on Doesn't seem that hard to me... Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
Hi, I literally have danpb's event introspection patches from Luiz's PULL request testing on my system right now to be pushed. Good. this?: [PATCH 29/29] Add 'query-events' command to QMP to query async events This is about libvirt getting qemu's event list. I am talking about qemu getting libvirt to say we support event SPICE_MIGRATE_DONE. Why do you think we need this? Other way around (libvirt detecting whenever qemu supports SPICE_MIGRATE_DONE event) should be good enougth, no? i.e. Yet another capability negotiation, during the handshake QMP phase. I'm sure libvirt will use query-events for other reasons anyway, so there is no extra overhead for this. cheers, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On Wed, Jun 06, 2012 at 03:03:37PM +0200, Gerd Hoffmann wrote: Hi, I literally have danpb's event introspection patches from Luiz's PULL request testing on my system right now to be pushed. Good. this?: [PATCH 29/29] Add 'query-events' command to QMP to query async events This is about libvirt getting qemu's event list. I am talking about qemu getting libvirt to say we support event SPICE_MIGRATE_DONE. Why do you think we need this? Other way around (libvirt detecting whenever qemu supports SPICE_MIGRATE_DONE event) should be good enougth, no? i.e. Yet another capability negotiation, during the handshake QMP phase. I'm sure libvirt will use query-events for other reasons anyway, so there is no extra overhead for this. What Anthony suggested, using a command line switch, would work fine for the problem I am talking about. The problem is how do we know that libvirt will support our new event. Libvirt using query-events doesn't help - unless you suggest we intercept it and record libvirt is aware of our new event, it probably supports it, but that's obviously wrong. If libvirt doesn't support this event we want to fall back to semi-seamless migration, if it does we want to do seamless migration by waiting for the source to complete guest migration, send the spice client a notification that the target is alive, then send this event, and libvirt will then close the source vm - migration downtime is unaffected since we only delay the closing of the vm after it is dead, i.e. stopped and migrated. cheers, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
Hi, If libvirt doesn't support this event we want to fall back to semi-seamless migration, Ah, ok. Yes, a new -spice option will work here. cheers, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/05/2012 01:49 PM, Yonit Halperin wrote: Hi, I'm sending this patch series again. This time with an additional patch for setting a migrate_end notifier completion callback for spice migration interface. I've also added more detailed commit messages. This patch series introduces async handlers for notifiers, and integrates them with migration state change notifications. Asynchronous migration completion notifier is essential for allowing spice to cleanly complete the src server connection to the client, and transfer the connection to the target. Currently, as soon as the migration completes, the src qemu can be closed by the management, and spice cannot complete the spice-connection migration. In order to support spice seamless migration, next to these patches, I plan to add: (1) notifier for switching from the live phase of the migration to the non-live phase, before completing savevm. Spice will use this notification to finalize the connection to the client: send and receive all in-flight data. Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. I'd like to be more constructive in my response, but you aren't explaining the problem well enough for me to offer an alternative solution. You need to find another way to solve this problem. Very short version: The requirement is simply to not kill qemu on the source side until the source spice-server has finished session handover to the target spice-server. Long version: spice-client connects automatically to the target machine, so the user ideally doesn't notice that his virtual machine was just migrated over to another host. Today this happens via switch-host, which is a simple message asking the spice client to connect to the new host. We want move to seamless migration model where we don't start over from scratch, but hand over the session from the source to the target. Advantage is that various state cached in spice-client will stay valid and doesn't need to be retransmitted. It also requires a handshake between spice-servers on source and target. libvirt killing qemu on the source host before the handshake is done isn't exactly helpful. [ Side note: In theory this issue exists even today: in case the data pipe to the client is full spice-server will queue up the switch-host message and qemu might be killed before it is sent out. In practice it doesn't happen though because it goes through the low-traffic main channel so the socket buffers usually have enougth space. ] So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go with a very short timeout there. Option (2): Add a new QMP event which is emmitted when spice-server is done, then make libvirt wait for it before killing qemu. Obvious disadvantage is that it requires libvirt changes. Option (3): Your suggestion? thanks, Gerd
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/05/2012 07:15 AM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go with a very short timeout there. Option (2): Add a new QMP event which is emmitted when spice-server is done, then make libvirt wait for it before killing qemu. Obvious disadvantage is that it requires libvirt changes. But there was recently a proposal for a new monitor command that will let libvirt query which events a given qemu supports, and therefore libvirt can at least know in advance whether to expect and wait for the event, or to fall back to some other option. Just because libvirt would require a change doesn't necessarily rule out this option. Option (3): Your suggestion? thanks, Gerd -- Eric Blake ebl...@redhat.com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
On 06/05/2012 09:38 PM, Eric Blake wrote: On 06/05/2012 07:15 AM, Gerd Hoffmann wrote: Hi, Absolutely not. This is hideously ugly and affects a bunch of code. Spice is *not* getting a hook in migration where it gets to add arbitrary amounts of downtime to the migration traffic. That's a terrible idea. So, the big question is how to tackle the issue? Option (1): Wait until spice-server is done before signaling completion to libvirt. This is what this patch series implements. Advantage is that it is completely transparent for libvirt, thats why I like it. Disadvantage is that it indeed adds a small delay for the spice-server handshake. The target qemu doesn't process main loop events while the incoming migration is running, and because of that the spice-server handshake doesn't run in parallel with the final stage of vm migration, which it could in theory. BTW: There will be no arbitrary amounts of downtime. Seamless spice client migration is pretty pointless if it doesn't finish within a fraction of a second, so we can go with a very short timeout there. Option (2): Add a new QMP event which is emmitted when spice-server is done, then make libvirt wait for it before killing qemu. Obvious disadvantage is that it requires libvirt changes. But there was recently a proposal for a new monitor command that will let libvirt query which events a given qemu supports, and therefore libvirt can at least know in advance whether to expect and wait for the event, or to fall back to some other option. Just because libvirt would require a change doesn't necessarily rule out this option. Right, this approach sounds much, much better to me too because it doesn't affect migration downtime. Regards, Anthony Liguori Option (3): Your suggestion? thanks, Gerd
[Qemu-devel] [RFC PATCH 0/5] asynchronous migration state change handlers
Hi, I'm sending this patch series again. This time with an additional patch for setting a migrate_end notifier completion callback for spice migration interface. I've also added more detailed commit messages. This patch series introduces async handlers for notifiers, and integrates them with migration state change notifications. Asynchronous migration completion notifier is essential for allowing spice to cleanly complete the src server connection to the client, and transfer the connection to the target. Currently, as soon as the migration completes, the src qemu can be closed by the management, and spice cannot complete the spice-connection migration. In order to support spice seamless migration, next to these patches, I plan to add: (1) notifier for switching from the live phase of the migration to the non-live phase, before completing savevm. Spice will use this notification to finalize the connection to the client: send and receive all in-flight data. (2) add vmstates for spice data that need to be migrated, e.g., usb/agent/smartcard buffers that were sent from the client and haven't been written to device yet. We would also want to migrate data that will allow us to continue the new spice connection from the same point the old one stopped. Without requiring special treatment in the client side. Regards, Yonit. Yonit Halperin (5): notifiers: add support for async notifiers handlers migration: moving migration start code to a separated routine migration: moving migration completion code to a separated routine migration: replace migration state change notifier with async notifiers spice: turn spice migration end handler to be async input.c |2 +- migration.c | 154 --- migration.h | 11 +++- notify.c| 78 ++-- notify.h| 55 ++-- qemu-timer.c|2 +- ui/spice-core.c | 58 +++-- vl.c|2 +- 8 files changed, 290 insertions(+), 72 deletions(-) -- 1.7.7.6
