Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
(2010/11/30 1:41), Dor Laor wrote: Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. You mean there are few, potential, users of live migration? We are planning to user live migration. So if there are known issues about live migration, apart from Kemari, it will be really helpful to have the lists. Thanks, Takuya -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Sat, Nov 27, 2010 at 1:11 PM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. I actually think the other way. Tracing just instruments and stashes away values. It does not change inputs or outputs, it does not change control flow, it does not affect state. Going down the route of side-effects mixes two different things: hooking into a subsystem and instrumentation. For hooking into a subsystem we should define proper interfaces. That interface can explicitly support modifying inputs/outputs or changing control flow. Tracing is much more ad-hoc and not a clean interface. It's also based on a layer of indirection via the tracetool code generator. That's okay because it doesn't affect the code it is called from and you don't need to debug trace events (they are simple and have almost no behavior). Hooking via tracing is just taking advantage of the cheap layer of indirection in order to get at interesting events in a subsystem. It's easy to hook up and quick to develop, but it's not a proper interface and will be hard to understand for other developers. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Whether Kemari adds constraints such as you mentioned, yes. If the devices (including existing ones) don't call Kemari code, they would certainly break Kemari. Altough using proxies looks explicit, to make it unaware from people writing device emulation, it's possible to remove proxies and put changes only into the block/net layer as Blue suggested. Anything that makes it hard to violate the constraints is good. Otherwise Kemari might get broken in the future and no one will know until a failover behaves incorrectly. Blue and Paul prefer to put it into block/net layer, and you think it's better to provide API. Sorry, I wasn't clear. I agree that event tap behavior should be in generic block and net layer code. That way we're guaranteeing that all net and block I/O goes through event tap. Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Here's a different question: what requirements must an emulated device meet in order to be added to the Kemari supported whitelist? That's what I want to know so that I don't break existing devices and can add new devices that work with Kemari :). Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Here's a different question: what requirements must an emulated device meet in order to be added to the Kemari supported whitelist? That's what I want to know so that I don't break existing devices and can add new devices that work with Kemari :). Why isn't it completely device agnostic? i.e. if a device has to care about Kemari at all (of vice-versa) then IMO you're doing it wrong. The whole point of the internal block/net APIs is that they isolate the host implementation details from the device emulation. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/29 Paul Brook p...@codesourcery.com: Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Here's a different question: what requirements must an emulated device meet in order to be added to the Kemari supported whitelist? That's what I want to know so that I don't break existing devices and can add new devices that work with Kemari :). Why isn't it completely device agnostic? i.e. if a device has to care about Kemari at all (of vice-versa) then IMO you're doing it wrong. The whole point of the internal block/net APIs is that they isolate the host implementation details from the device emulation. You're right theoretically. But what I've learned so far, there are cases like virtio-net and e1000 woks but virtio-blk doesn't. Theoretically, any emulated device should be able to get into the whitelist if the event-tap is properly implemented but sometimes it doesn't seem to be that simple. To answer Stefan's question, there shouldn't be any requirement for a device, but must be tested with Kemari. If it doesn't work correctly, the problems must be fixed before adding to the list. Yoshi Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/29 Paul Brook p...@codesourcery.com: Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Here's a different question: what requirements must an emulated device meet in order to be added to the Kemari supported whitelist? That's what I want to know so that I don't break existing devices and can add new devices that work with Kemari :). Why isn't it completely device agnostic? i.e. if a device has to care about Kemari at all (of vice-versa) then IMO you're doing it wrong. The whole point of the internal block/net APIs is that they isolate the host implementation details from the device emulation. You're right theoretically. But what I've learned so far, there are cases like virtio-net and e1000 woks but virtio-blk doesn't. Theoretically, any emulated device should be able to get into the whitelist if the event-tap is properly implemented but sometimes it doesn't seem to be that simple. To answer Stefan's question, there shouldn't be any requirement for a device, but must be tested with Kemari. If it doesn't work correctly, the problems must be fixed before adding to the list. What exactly are the problems? Is this a device bus of a Kemari bug? If it's the former then that implies you're imposing additional requirements that weren't previously part of the API. If the latter, then it's a bug like any other. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/29 Paul Brook p...@codesourcery.com: 2010/11/29 Paul Brook p...@codesourcery.com: Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Here's a different question: what requirements must an emulated device meet in order to be added to the Kemari supported whitelist? That's what I want to know so that I don't break existing devices and can add new devices that work with Kemari :). Why isn't it completely device agnostic? i.e. if a device has to care about Kemari at all (of vice-versa) then IMO you're doing it wrong. The whole point of the internal block/net APIs is that they isolate the host implementation details from the device emulation. You're right theoretically. But what I've learned so far, there are cases like virtio-net and e1000 woks but virtio-blk doesn't. Theoretically, any emulated device should be able to get into the whitelist if the event-tap is properly implemented but sometimes it doesn't seem to be that simple. To answer Stefan's question, there shouldn't be any requirement for a device, but must be tested with Kemari. If it doesn't work correctly, the problems must be fixed before adding to the list. What exactly are the problems? Is this a device bus of a Kemari bug? If it's the former then that implies you're imposing additional requirements that weren't previously part of the API. If the latter, then it's a bug like any other. It's a problem if devices don't continue correctly upon failover. I would say it's a bug of live migration (not all of course) because Kemari is just live migrating at specific points. Yoshi Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
To answer Stefan's question, there shouldn't be any requirement for a device, but must be tested with Kemari. If it doesn't work correctly, the problems must be fixed before adding to the list. What exactly are the problems? Is this a device bus of a Kemari bug? If it's the former then that implies you're imposing additional requirements that weren't previously part of the API. If the latter, then it's a bug like any other. It's a problem if devices don't continue correctly upon failover. I would say it's a bug of live migration (not all of course) because Kemari is just live migrating at specific points. Ah, now we're getting somewhere. So you're saying that these devices are broken anyway, and Kemari happens to trigger that brokenness more frequently? If the requirement is that a device must support live migration, then that should be the criteria for enabling Kemari, not some arbitrary whitelist. If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/29 Paul Brook p...@codesourcery.com: To answer Stefan's question, there shouldn't be any requirement for a device, but must be tested with Kemari. If it doesn't work correctly, the problems must be fixed before adding to the list. What exactly are the problems? Is this a device bus of a Kemari bug? If it's the former then that implies you're imposing additional requirements that weren't previously part of the API. If the latter, then it's a bug like any other. It's a problem if devices don't continue correctly upon failover. I would say it's a bug of live migration (not all of course) because Kemari is just live migrating at specific points. Ah, now we're getting somewhere. So you're saying that these devices are broken anyway, and Kemari happens to trigger that brokenness more frequently? If the requirement is that a device must support live migration, then that should be the criteria for enabling Kemari, not some arbitrary whitelist. Sorry, I though that criteria to be obvious one and didn't think to clarify. The whitelist is a guard not to let users get into trouble with arbitrary devices. If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? Yoshi Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/29 Paul Brook p...@codesourcery.com: If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Alright! Then if a user encounters a problem with Kemari, we'll fix Kemari or the devices or both. Correct? Yoshi Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Alright! Then if a user encounters a problem with Kemari, we'll fix Kemari or the devices or both. Correct? Correct. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Mon, Nov 29, 2010 at 3:00 PM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/29 Paul Brook p...@codesourcery.com: If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Alright! Then if a user encounters a problem with Kemari, we'll fix Kemari or the devices or both. Correct? Is this a fair summary: any device that supports live migration workw under Kemari? (If such a device does not work under Kemari then this is a bug that needs to be fixed in live migration, Kemari, or the device.) Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On 11/29/2010 06:23 PM, Stefan Hajnoczi wrote: On Mon, Nov 29, 2010 at 3:00 PM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/29 Paul Brookp...@codesourcery.com: If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Alright! Then if a user encounters a problem with Kemari, we'll fix Kemari or the devices or both. Correct? Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. (If such a device does not work under Kemari then this is a bug that needs to be fixed in live migration, Kemari, or the device.) Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. That's not what I've been hearing earlier in this thread. The responses from Yoshi indicate that Stefan's summary is correct. i.e. the current Kemari implementation may require per-device hooks, but that's a bug and should be fixed before merging. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On 11/29/2010 10:53 AM, Paul Brook wrote: Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. That's not what I've been hearing earlier in this thread. The responses from Yoshi indicate that Stefan's summary is correct. i.e. the current Kemari implementation may require per-device hooks, but that's a bug and should be fixed before merging. It's actually really important that Kemari make use of an intermediate layer such that the hooks can distinguish between a device access and a recursive access. You could s/bdrv_aio_multiwrite/bdrv_aio_multiwrite_internal/g and then within kemari, s/bdrv_aio_multiwrite_proxy/bdrv_aio_multiwrite/ but I don't think that results in a cleaner interface. I don't like the _proxy naming and I think it has led to some confusion. I think having a dev_aio_multiwrite interface is a better naming scheme and ultimately provides a clearer idea of why a separate interface is needed--to distinguish between device accesses and internal accesses. BTW, dev_aio_multiwrite should take a DeviceState * and a BlockDriverState. Regards, Anthony Liguori Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On 11/29/2010 10:53 AM, Paul Brook wrote: Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. That's not what I've been hearing earlier in this thread. The responses from Yoshi indicate that Stefan's summary is correct. i.e. the current Kemari implementation may require per-device hooks, but that's a bug and should be fixed before merging. It's actually really important that Kemari make use of an intermediate layer such that the hooks can distinguish between a device access and a recursive access. I'm failing to understand how this is anything other than running sed over block/*.c (or hw/*.c, depending whether you choose to rename the internal or external API). Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On 11/29/2010 11:18 AM, Paul Brook wrote: On 11/29/2010 10:53 AM, Paul Brook wrote: Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. That's not what I've been hearing earlier in this thread. The responses from Yoshi indicate that Stefan's summary is correct. i.e. the current Kemari implementation may require per-device hooks, but that's a bug and should be fixed before merging. It's actually really important that Kemari make use of an intermediate layer such that the hooks can distinguish between a device access and a recursive access. I'm failing to understand how this is anything other than running sed over block/*.c (or hw/*.c, depending whether you choose to rename the internal or external API). You're right, it's not a big deal, and requiring everything in hw use the new interface is not a bad idea. If a device doesn't work with Kemari, that's okay as long as the non-Kemari case is essentially a nop. Regards, Anthony Liguori Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/30 Dor Laor dl...@redhat.com: On 11/29/2010 06:23 PM, Stefan Hajnoczi wrote: On Mon, Nov 29, 2010 at 3:00 PM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/29 Paul Brookp...@codesourcery.com: If devices incorrectly claim support for live migration, then that should also be fixed, either by removing the broken code or by making it work. I totally agree with you. AFAICT your current proposal is just feeding back the results of some fairly specific QA testing. I'd rather not get into that game. The correct response in the context of upstream development is to file a bug and/or fix the code. We already have config files that allow third party packagers to remove devices they don't want to support. Sorry, I didn't get what you're trying to tell me. My plan would be to initially start from a subset of devices, and gradually grow the number of devices that Kemari works with. While this process, it'll include what you said above, file a but and/or fix the code. Am I missing what you're saying? My point is that the whitelist shouldn't exist at all. Devices either support migration or they don't. Having some sort of separate whitelist is the wrong way to determine which devices support migration. Alright! Then if a user encounters a problem with Kemari, we'll fix Kemari or the devices or both. Correct? Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. IIUC, the additional hook you're mentioning is the hack for virtio. Michael has commented on it, I hope his patch make the hack unnecessary. Yoshi (If such a device does not work under Kemari then this is a bug that needs to be fixed in live migration, Kemari, or the device.) Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/30 Anthony Liguori anth...@codemonkey.ws: On 11/29/2010 10:53 AM, Paul Brook wrote: Is this a fair summary: any device that supports live migration workw under Kemari? It might be fair summary but practically we barely have live migration working w/o Kemari. In addition, last I checked Kemari needs additional hooks and it will be too hard to keep that out of tree until all devices get it. That's not what I've been hearing earlier in this thread. The responses from Yoshi indicate that Stefan's summary is correct. i.e. the current Kemari implementation may require per-device hooks, but that's a bug and should be fixed before merging. It's actually really important that Kemari make use of an intermediate layer such that the hooks can distinguish between a device access and a recursive access. You could s/bdrv_aio_multiwrite/bdrv_aio_multiwrite_internal/g and then within kemari, s/bdrv_aio_multiwrite_proxy/bdrv_aio_multiwrite/ but I don't think that results in a cleaner interface. I don't like the _proxy naming and I think it has led to some confusion. I think having a dev_aio_multiwrite interface is a better naming scheme and ultimately provides a clearer idea of why a separate interface is needed--to distinguish between device accesses and internal accesses. Sorry about the naming. But from the discussion so far, adding an intermediate layer and exporting it to some/all approach needs a strong reason. Kemari itself can be implemented w/ or w/o the intermediate layer, and this makes the discussion toward folding the layer into block/net to be appropriate. I think there are two perspectives to decide which way to go: - What is clean interfaces for upper/lower layer? - If we introduce the intermediate layer, is there anyone who may use now or in the future? If not, it may not be worth to add. Yoshi BTW, dev_aio_multiwrite should take a DeviceState * and a BlockDriverState. Regards, Anthony Liguori Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. I understand the benefit of your suggestion. However it seems a bit tricky. It's because event-tap uses functions of emulators and net, but block.c is also linked for utilities like qemu-img that doesn't need emulators or net. In the previous version, I added function pointers to get around. http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html I wasn't confident of this approach and discussed it at KVM Forum, and decided to give a try to replace emulator functions with proxies. Suggestions are welcomed of course. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Whether Kemari adds constraints such as you mentioned, yes. If the devices (including existing ones) don't call Kemari code, they would certainly break Kemari. Altough using proxies looks explicit, to make it unaware from people writing device emulation, it's possible to remove proxies and put changes only into the block/net layer as Blue suggested. Yoshi Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. I understand the benefit of your suggestion. However it seems a bit tricky. It's because event-tap uses functions of emulators and net, but block.c is also linked for utilities like qemu-img that doesn't need emulators or net. In the previous version, I added function pointers to get around. http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html I wasn't confident of this approach and discussed it at KVM Forum, and decided to give a try to replace emulator functions with proxies. Suggestions are welcomed of course. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. This all depends on how generic we want the trace points become. One possible extension to the event injection or instrumentation could be fault injection: based on some rule, make the instrumented function return error. That would be interesting for testing how guest handles failure cases. Maybe it should be also possible to handle event injection in a generic way. Split the instrumented function to two, before and after the tracepoint. The tracepoint registers the tail function in addition to the parameters. This may require a lot of refactoring though. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Whether Kemari adds constraints such as
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? IMO anything that requires devices to act differently is wrong. All external IO already goes though a common API (e.g. qemu_send_packet). You should be putting your transaction code there, not hacking individual devices. Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. I actually think the other way. Tracing just instruments and stashes away values. It does not change inputs or outputs, it does not change control flow, it does not affect state. Going down the route of side-effects mixes two different things: hooking into a subsystem and instrumentation. For hooking into a subsystem we should define proper interfaces. That interface can explicitly support modifying inputs/outputs or changing control flow. Tracing is much more ad-hoc and not a clean interface. It's also based on a layer of indirection via the tracetool code generator. That's okay because it doesn't affect the code it is called from and you don't need to debug trace events (they are simple and have almost no behavior). Hooking via tracing is just taking advantage of the cheap layer of indirection in order to get at interesting events in a subsystem. It's easy to hook up and quick to develop, but it's not a proper interface and will be hard to understand for other developers. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Whether Kemari adds constraints such as you mentioned, yes. If the devices (including existing ones) don't call Kemari code, they would certainly break Kemari. Altough using proxies looks explicit, to make it unaware from people writing device emulation, it's possible to remove proxies and put changes only into the block/net layer as Blue suggested. Anything that makes it hard to violate the constraints is good. Otherwise Kemari might get broken in the future and no one will know until a failover behaves incorrectly. Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/27 Blue Swirl blauwir...@gmail.com: On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. I understand the benefit of your suggestion. However it seems a bit tricky. It's because event-tap uses functions of emulators and net, but block.c is also linked for utilities like qemu-img that doesn't need emulators or net. In the previous version, I added function pointers to get around. http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html I wasn't confident of this approach and discussed it at KVM Forum, and decided to give a try to replace emulator functions with proxies. Suggestions are welcomed of course. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. This all depends on how generic we want the trace points become. One possible extension to the event injection or instrumentation could be fault injection: based on some rule, make the instrumented function return error. That would be interesting for testing how guest handles failure cases. Maybe it should be also possible to handle event injection in a generic way. Split the instrumented function to two, before and after the tracepoint. The tracepoint registers the tail function in addition to the parameters. This may require a lot of refactoring though. The idea looks cool but it's a bit out of the range I can handle now:-) Let's keep the idea of binding with trace points for now, and focus on how to insert net/block tap points. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/27 Paul Brook p...@codesourcery.com: One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? IMO anything that requires devices to act differently is wrong. All external IO already goes though a common API (e.g. qemu_send_packet). You should be putting your transaction code there, not hacking individual devices. So you're with Blue's idea to put them in block/net layer. Yoshi Paul -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 8:53 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Stefan Hajnoczi stefa...@gmail.com: On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. The connection is that it may be possible to remove Kemari specific hook point like in ioport.c and exec.c, and let tracing notify Kemari instead. I actually think the other way. Tracing just instruments and stashes away values. It does not change inputs or outputs, it does not change control flow, it does not affect state. Going down the route of side-effects mixes two different things: hooking into a subsystem and instrumentation. For hooking into a subsystem we should define proper interfaces. That interface can explicitly support modifying inputs/outputs or changing control flow. Tracing is much more ad-hoc and not a clean interface. It's also based on a layer of indirection via the tracetool code generator. That's okay because it doesn't affect the code it is called from and you don't need to debug trace events (they are simple and have almost no behavior). Hooking via tracing is just taking advantage of the cheap layer of indirection in order to get at interesting events in a subsystem. It's easy to hook up and quick to develop, but it's not a proper interface and will be hard to understand for other developers. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Whether Kemari adds constraints such as you mentioned, yes. If the devices (including existing ones) don't call Kemari code, they would certainly break Kemari. Altough using proxies looks explicit, to make it unaware from people writing device emulation, it's possible to remove proxies and put changes only into the block/net layer as Blue suggested. Anything that makes it hard to violate the constraints is good. Otherwise Kemari might get broken in the future and no one will know until a failover behaves incorrectly. Blue and Paul prefer to put it into block/net layer, and you think it's better to provide API. I have an idea which may fit into both, which is to put the look into block/net layer, and make a list of devices that Kemari supports. Before turning on, we can check whether the devices tapped are those on the list. It's Kemari's responsibility to keep checking which devices can be supported. At this point, devices with proxies are on the list. Could you formulate the constraints so developers are aware of them in the future and can protect the codebase. How about expanding the Kemari wiki pages? If you like the idea above, I'm happy to make the list also on the wiki page. Yoshi Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. I understand the benefit of your suggestion. However it seems a bit tricky. It's because event-tap uses functions of emulators and net, but block.c is also linked for utilities like qemu-img that doesn't need emulators or net. In the previous version, I added function pointers to get around. http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html I wasn't confident of this approach and discussed it at KVM Forum, and decided to give a try to replace emulator functions with proxies. Suggestions are welcomed of course. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Qemu-devel] [PATCH 00/21] Kemari for KVM 0.2
On Sat, Nov 27, 2010 at 4:29 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: 2010/11/27 Blue Swirl blauwir...@gmail.com: On Thu, Nov 25, 2010 at 6:06 AM, Yoshiaki Tamura tamura.yoshi...@lab.ntt.co.jp wrote: Hi, This patch series is a revised version of Kemari for KVM, which applied comments for the previous post and KVM Forum 2010. The current code is based on qemu.git f711df67d611e4762966a249742a5f7499e19f99. For general information about Kemari, I've made a wiki page at qemu.org. http://wiki.qemu.org/Features/FaultTolerance The changes from v0.1.1 - v0.2 are: - Introduce a queue in event-tap to make VM sync live. - Change transaction receiver to a state machine for async receiving. - Replace net/block layer functions with event-tap proxy functions. - Remove dirty bitmap optimization for now. - convert DPRINTF() in ft_trans_file to trace functions. - convert fprintf() in ft_trans_file to error_report(). - improved error handling in ft_trans_file. - add a tmp pointer to qemu_del_vm_change_state_handler. The changes from v0.1 - v0.1.1 are: - events are tapped in net/block layer instead of device emulation layer. - Introduce a new option for -incoming to accept FT transaction. - Removed writev() support to QEMUFile and FdMigrationState for now. I would post this work in a different series. - Modified virtio-blk save/load handler to send inuse variable to correctly replay. - Removed configure --enable-ft-mode. - Removed unnecessary check for qemu_realloc(). The first 6 patches modify several functions of qemu to prepare introducing Kemari specific components. The next 6 patches are the components of Kemari. They introduce event-tap and the FT transaction protocol file based on buffered file. The design document of FT transaction protocol can be found at, http://wiki.qemu.org/images/b/b1/Kemari_sender_receiver_0.5a.pdf Then the following 4 patches modifies dma-helpers, virtio-blk virtio-net and e1000 to replace net/block layer functions with event-tap proxy functions. Please note that if Kemari is off, event-tap will just passthrough, and there is most no intrusion to exisiting functions including normal live migration. Would it be possible to make the changes only in the block/net layer, so that the devices are not modified at all? That is, the proxy function would always replaces the unproxied version. I understand the benefit of your suggestion. However it seems a bit tricky. It's because event-tap uses functions of emulators and net, but block.c is also linked for utilities like qemu-img that doesn't need emulators or net. In the previous version, I added function pointers to get around. http://lists.nongnu.org/archive/html/qemu-devel/2010-05/msg02378.html I wasn't confident of this approach and discussed it at KVM Forum, and decided to give a try to replace emulator functions with proxies. Suggestions are welcomed of course. Somehow I find some similarities to instrumentation patches. Perhaps the instrumentation framework could be used (maybe with some changes) for Kemari as well? That could be beneficial to both. Yes. I had the same idea but I'm not sure how tracing works. I think Stefan Hajnoczi knows it better. Stefan, is it possible to call arbitrary functions from the trace points? Yes, if you add code to ./tracetool. I'm not sure I see the connection between Kemari and tracing though. One question I have about Kemari is whether it adds new constraints to the QEMU codebase? Fault tolerance seems like a cross-cutting concern - everyone writing device emulation or core QEMU code may need to be aware of new constraints. For example, you are not allowed to release I/O operations to the outside world directly, instead you need to go through Kemari code which makes I/O transactional and communicates with the passive host. You have converted e1000, virtio-net, and virtio-blk. How do we make sure new devices that are merged into qemu.git don't break Kemari? How do we go about supporting the existing hw/* devices? Stefan -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
