Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On 01/14/2011 06:00 AM, Juan Quintela wrote: Marcelo Tosattimtosa...@redhat.com wrote: On Fri, Jan 07, 2011 at 10:44:20AM -1000, Zachary Amsden wrote: On 01/07/2011 12:48 AM, Marcelo Tosatti wrote: On Thu, Jan 06, 2011 at 12:10:45AM -1000, Zachary Amsden wrote: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. Migration to hosts which do not support the feature can be achieved by saving/restoring the TSC rate + flags in a subsection. A subsection seems more appropriate than an MSR for this. Yes, I looked at that, but it looked to me like a subsection was intended for an optional feature which MUST be present on the destination if the source is using the feature. This way, newer hosts without the feature enabled can migrate to older hosts which do not support the feature. Right. But you can use a subsection to achieve the same effect. Just consider that the source is not using the feature if you want to migrate to an older host without support for it. Juan, is there a problem to use subsections in this fashion? With the MSR scheme, there is no way for management to enforce support of the feature on the destination (at least not that i can see). And you create an MSR that does not exist on real hardware. The TSC rate migration is slightly different; we may wish to migrate from a host with the TSC rate feature enabled to a host which does not support the TSC rate feature. This is exactly the current behavior, the TSC rate will change on that migration, and I wanted to preserve that behavior. I don't advise that mode of usage, but there may be use cases for it and it should be decided by policy, not dictated by our feature set. That said, I'm happy to remove the MSR if we truly don't want to support that mode of usage. Ok, I chime it late. We are adding a new MSR to the comunication with userspace. So far so good, but this new field, need to be transmited to the other end of the migration. This means a new field for migration (notice that this is independtly if this is an MSR or not). VMSTATE_UINT64_V(system_time_msr, CPUState, 11), VMSTATE_UINT64_V(wall_clock_msr, CPUState, 11), Oh, wow. I thought the MSRs were sent automatically by qemu based on what MSRs the kvm module told it were available. It looks like EFER and STAR and friends are all special cased as part of CPUstate. So my approach has been doomed from the beginning. This are the values that we are sending now. We are getting now, a new value, the problem is how to migrate it. Solutions: - create a new field in a new field in CPUState, and up the version. That would make backward migration impossible. - create a new field, and sent it only it is has been used with a subsection. This makes migration backwards if this was not used. - but, it appears that there if this features is not known on destination, we can use the old way to migrate information. BIG WARNING HERE: I don't claim to understand how clocks work at all There is not a way to convince old qemu/kernels to ignore new fields for good reason. So the only solution here is to encode this new vcpu-kvm-arch.virtual_tsc_khz in the two previous fields, in a way that is understable for old qemu/new qemu. old qemu will use old method, new qemu will use a new method. If there is not a common encoding that will work for both old/new method, I can't really think of a way to make things work here :( And as per the warning, I can't think of a way to encode virtual_tsc_khz into system_time_msr and wall_clock_msr off-hand. To make things clearer about optional features, after lots of discussions, it was decided that target of migration will never ignore anything sent, that means that the only one that can decide not to sent a feature/value is the source of the migration. There is no way to express: - try this method, if you don't know - try this other second best, ... That decides it then, the feature is migrated in the state it is set if it is enabled. This makes things much simpler all around. Cheers, Zach -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
Marcelo Tosatti mtosa...@redhat.com wrote: On Fri, Jan 07, 2011 at 10:44:20AM -1000, Zachary Amsden wrote: On 01/07/2011 12:48 AM, Marcelo Tosatti wrote: On Thu, Jan 06, 2011 at 12:10:45AM -1000, Zachary Amsden wrote: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. Migration to hosts which do not support the feature can be achieved by saving/restoring the TSC rate + flags in a subsection. A subsection seems more appropriate than an MSR for this. Yes, I looked at that, but it looked to me like a subsection was intended for an optional feature which MUST be present on the destination if the source is using the feature. This way, newer hosts without the feature enabled can migrate to older hosts which do not support the feature. Right. But you can use a subsection to achieve the same effect. Just consider that the source is not using the feature if you want to migrate to an older host without support for it. Juan, is there a problem to use subsections in this fashion? With the MSR scheme, there is no way for management to enforce support of the feature on the destination (at least not that i can see). And you create an MSR that does not exist on real hardware. The TSC rate migration is slightly different; we may wish to migrate from a host with the TSC rate feature enabled to a host which does not support the TSC rate feature. This is exactly the current behavior, the TSC rate will change on that migration, and I wanted to preserve that behavior. I don't advise that mode of usage, but there may be use cases for it and it should be decided by policy, not dictated by our feature set. That said, I'm happy to remove the MSR if we truly don't want to support that mode of usage. Ok, I chime it late. We are adding a new MSR to the comunication with userspace. So far so good, but this new field, need to be transmited to the other end of the migration. This means a new field for migration (notice that this is independtly if this is an MSR or not). VMSTATE_UINT64_V(system_time_msr, CPUState, 11), VMSTATE_UINT64_V(wall_clock_msr, CPUState, 11), This are the values that we are sending now. We are getting now, a new value, the problem is how to migrate it. Solutions: - create a new field in a new field in CPUState, and up the version. That would make backward migration impossible. - create a new field, and sent it only it is has been used with a subsection. This makes migration backwards if this was not used. - but, it appears that there if this features is not known on destination, we can use the old way to migrate information. BIG WARNING HERE: I don't claim to understand how clocks work at all There is not a way to convince old qemu/kernels to ignore new fields for good reason. So the only solution here is to encode this new vcpu-kvm-arch.virtual_tsc_khz in the two previous fields, in a way that is understable for old qemu/new qemu. old qemu will use old method, new qemu will use a new method. If there is not a common encoding that will work for both old/new method, I can't really think of a way to make things work here :( And as per the warning, I can't think of a way to encode virtual_tsc_khz into system_time_msr and wall_clock_msr off-hand. To make things clearer about optional features, after lots of discussions, it was decided that target of migration will never ignore anything sent, that means that the only one that can decide not to sent a feature/value is the source of the migration. There is no way to express: - try this method, if you don't know - try this other second best, ... and so on. Later, Juan. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On Fri, Jan 07, 2011 at 10:44:20AM -1000, Zachary Amsden wrote: On 01/07/2011 12:48 AM, Marcelo Tosatti wrote: On Thu, Jan 06, 2011 at 12:10:45AM -1000, Zachary Amsden wrote: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. Migration to hosts which do not support the feature can be achieved by saving/restoring the TSC rate + flags in a subsection. A subsection seems more appropriate than an MSR for this. Yes, I looked at that, but it looked to me like a subsection was intended for an optional feature which MUST be present on the destination if the source is using the feature. This way, newer hosts without the feature enabled can migrate to older hosts which do not support the feature. Right. But you can use a subsection to achieve the same effect. Just consider that the source is not using the feature if you want to migrate to an older host without support for it. Juan, is there a problem to use subsections in this fashion? With the MSR scheme, there is no way for management to enforce support of the feature on the destination (at least not that i can see). And you create an MSR that does not exist on real hardware. The TSC rate migration is slightly different; we may wish to migrate from a host with the TSC rate feature enabled to a host which does not support the TSC rate feature. This is exactly the current behavior, the TSC rate will change on that migration, and I wanted to preserve that behavior. I don't advise that mode of usage, but there may be use cases for it and it should be decided by policy, not dictated by our feature set. That said, I'm happy to remove the MSR if we truly don't want to support that mode of usage. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On Thu, Jan 06, 2011 at 12:10:45AM -1000, Zachary Amsden wrote: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. Migration to hosts which do not support the feature can be achieved by saving/restoring the TSC rate + flags in a subsection. A subsection seems more appropriate than an MSR for this. This allows administrative tools to configure migration policy as they see appropriate. Rather than dictate this policy with the KVM implementation, we properly allow migration to hosts which both do and do not support setting of the TSC rate on the receiving end. If it is wished to not support migration to a host which lacks support for the TSC rate feature, that can be coordinated externally. Signed-off-by: Zachary Amsden zams...@redhat.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On 01/07/2011 12:48 AM, Marcelo Tosatti wrote: On Thu, Jan 06, 2011 at 12:10:45AM -1000, Zachary Amsden wrote: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. Migration to hosts which do not support the feature can be achieved by saving/restoring the TSC rate + flags in a subsection. A subsection seems more appropriate than an MSR for this. Yes, I looked at that, but it looked to me like a subsection was intended for an optional feature which MUST be present on the destination if the source is using the feature. This way, newer hosts without the feature enabled can migrate to older hosts which do not support the feature. The TSC rate migration is slightly different; we may wish to migrate from a host with the TSC rate feature enabled to a host which does not support the TSC rate feature. This is exactly the current behavior, the TSC rate will change on that migration, and I wanted to preserve that behavior. I don't advise that mode of usage, but there may be use cases for it and it should be decided by policy, not dictated by our feature set. That said, I'm happy to remove the MSR if we truly don't want to support that mode of usage. Zach -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
Am 06.01.2011 um 11:10 schrieb Zachary Amsden zams...@redhat.com: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. This allows administrative tools to configure migration policy as they see appropriate. Rather than dictate this policy with the KVM implementation, we properly allow migration to hosts which both do and do not support setting of the TSC rate on the receiving end. If it is wished to not support migration to a host which lacks support for the TSC rate feature, that can be coordinated externally. Isn't there a real hw equivalent of such a register? It might make more sense to just implement that then. Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On 01/06/2011 12:34 AM, Alexander Graf wrote: Am 06.01.2011 um 11:10 schrieb Zachary Amsdenzams...@redhat.com: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. This allows administrative tools to configure migration policy as they see appropriate. Rather than dictate this policy with the KVM implementation, we properly allow migration to hosts which both do and do not support setting of the TSC rate on the receiving end. If it is wished to not support migration to a host which lacks support for the TSC rate feature, that can be coordinated externally. Isn't there a real hw equivalent of such a register? It might make more sense to just implement that then. Unfortunately, no. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On 06.01.2011, at 12:27, Zachary Amsden wrote: On 01/06/2011 12:34 AM, Alexander Graf wrote: Am 06.01.2011 um 11:10 schrieb Zachary Amsdenzams...@redhat.com: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. This allows administrative tools to configure migration policy as they see appropriate. Rather than dictate this policy with the KVM implementation, we properly allow migration to hosts which both do and do not support setting of the TSC rate on the receiving end. If it is wished to not support migration to a host which lacks support for the TSC rate feature, that can be coordinated externally. Isn't there a real hw equivalent of such a register? It might make more sense to just implement that then. Unfortunately, no. Bleks. I couldn't find anything in AMD documentation either. Intel documentation is usually hard to find and incomplete anyways, so maybe something's hiding there - but if it's hidden so well it's no use to implement either. Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [KVM TSC trapping / migration 2/2] Add TSC KHZ MSR
On 01/06/2011 01:40 AM, Alexander Graf wrote: On 06.01.2011, at 12:27, Zachary Amsden wrote: On 01/06/2011 12:34 AM, Alexander Graf wrote: Am 06.01.2011 um 11:10 schrieb Zachary Amsdenzams...@redhat.com: Use an MSR to allow soft migration to hosts which do not support TSC trapping. Rather than make this a required element of any migration protocol, we allow the TSC rate to be exported as a data field (useful in its own right), but we also allow a one time write of the MSR during VM creation. The result is that for the common use case, no protocol change is required to communicate TSC rate to the receiving host. This allows administrative tools to configure migration policy as they see appropriate. Rather than dictate this policy with the KVM implementation, we properly allow migration to hosts which both do and do not support setting of the TSC rate on the receiving end. If it is wished to not support migration to a host which lacks support for the TSC rate feature, that can be coordinated externally. Isn't there a real hw equivalent of such a register? It might make more sense to just implement that then. Unfortunately, no. Bleks. I couldn't find anything in AMD documentation either. Intel documentation is usually hard to find and incomplete anyways, so maybe something's hiding there - but if it's hidden so well it's no use to implement either. While it makes perfect logical sense to us software people, from a hardware perspective it is ridiculous. There is no signal input to the processor which can be counted and returned by a CPU frequency MSR. The CPU frequency is simply what the CPU is clocked at. And you can't burn it onto the die (think of overclocking), unless you have your own internal reference oscillator. No, a CPU frequency MSR is impossible to implement sensibly in hardware because 1) if you have an internal clock, you have cross-CPU drift and can't easily share a common bus. 2) if you have an external clock, you have no way to measure the frequency of it without an internal reference, which again will vary, leading to different measured frequency on each CPU. The only thing that sort of makes sense is a one-time writable MSR that is programmed by the BIOS to return the CPU frequency, as retrieved from the chipset. This is merely a convenience for software people who can't be bothered to query the chipset directly or measure against a reference clock. So as much as we'd like it, it really does make little sense from a hardware perspective. Zach -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
