Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
So, how do we address the performance regression fixed by this patch? Olaf Am Mon, 12 Mar 2018 10:15:08 +0100 schrieb Olaf Hering : > Am Tue, 6 Mar 2018 11:07:54 + > schrieb Andrew Cooper : > > > > Add a new domctl XEN_DOMCTL_set_vtsc_tolerance_khz to adjust the > > > tolerance value of a running domU that is supposed to be migrated. > > Please can we not proliferate the domctls. > > This looks like it should be part of the set_tsc_info hypercall, not a > > separate hypercall. > > How should this approach be implemented? To me it looks like set_tsc_info > should be called only once for a not-yet-active domU, but I'm not familiar > with timekeeping. > If it can be just called once, and there should be a way to adjust the > tolerance value, then XEN_DOMCTL_settscinfo would need to be changed to > have subcommands. pgpXHgJRsm8RL.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
Am Tue, 6 Mar 2018 11:07:54 + schrieb Andrew Cooper : > > Add a new domctl XEN_DOMCTL_set_vtsc_tolerance_khz to adjust the > > tolerance value of a running domU that is supposed to be migrated. > Please can we not proliferate the domctls. > This looks like it should be part of the set_tsc_info hypercall, not a > separate hypercall. How should this approach be implemented? To me it looks like set_tsc_info should be called only once for a not-yet-active domU, but I'm not familiar with timekeeping. If it can be just called once, and there should be a way to adjust the tolerance value, then XEN_DOMCTL_settscinfo would need to be changed to have subcommands. Olaf pgpexAfhCoFE1.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
>>> On 06.03.18 at 12:07, wrote: > On 06/03/18 10:41, Olaf Hering wrote: >> With this option the host admin can decide how a domU should behave when >> it is migrated across systems of the same class. Since there is always >> some jitter when Xen calibrates the cpu_khz value, all hosts of the same >> class will most likely have slightly different values. As a result vTSC >> emulation is unavoidable. Data collected during the incident which >> triggered this change showed a jitter of up to 200 KHz across systems of >> the same class. > > We should see about using better sources of information. For one, many > Intel systems actually expose the TSC frequency in the bottom of the > PLATFORM_INFO MSR, although this isn't architectural, and has been > replaced with CPUID information in Skylake. The question is how precise this information is - I can't imagine it is any better than the nominal clock, which then may be as far off the actual one as the jitter Olaf is talking about. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
Am Tue, 6 Mar 2018 11:07:54 + schrieb Andrew Cooper : > We should see about using better sources of information. For one, many > Intel systems actually expose the TSC frequency in the bottom of the > PLATFORM_INFO MSR, although this isn't architectural, and has been > replaced with CPUID information in Skylake. I'm not familiar with that. Is it what arch/x86/kernel/tsc_msr.c:cpu_khz_from_msr() gets from MSR_PLATFORM_INFO + MSR_FSB_FREQ? Olaf pgp2Wtw6K3jJf.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
Am Tue, 6 Mar 2018 11:07:54 + schrieb Andrew Cooper : > > One option to avoid the TSC option is to run domUs with tsc_mode=native. > > This has the drawback that migrating a domU from a "2.3GHz" class host > > to a "2.4GHz" class host may change the rate at wich the TSC counter > > increases, the domU may not be prepared for that. > > Given that you identify this specifically, why do you think this change > is safe or sensible in the first place? If using the options here, time > will definitely drift in the VM after migrate. This option is a variant of tsc_mode=default for different hosts, and tsc_mode=native for identical hosts. I think the tsc_mode can not be adjusted for running domUs. So it gives some control to the host admin about how to handle the drift. Olaf pgpWlm9tLuxFo.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
Am Tue, 6 Mar 2018 11:07:54 + schrieb Andrew Cooper : > This looks like it should be part of the set_tsc_info hypercall, not a > separate hypercall. How would one adjust the value at runtime with set_tsc_info of a running domU? It seems to me set_tsc_info does way more than just applying some values, it uses runtime data such as get_s_time. Olaf pgprIQW0GFfap.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3] new config option vtsc_tolerance_khz to avoid TSC emulation
On 06/03/18 10:41, Olaf Hering wrote: > Add an option to control when vTSC emulation will be activated for a > domU with tsc_mode=default. Without such option each TSC access from > domU will be emulated, which causes a significant perfomance drop for > workloads that make use of rdtsc. > > Add a new domctl XEN_DOMCTL_set_vtsc_tolerance_khz to adjust the > tolerance value of a running domU that is supposed to be migrated. Please can we not proliferate the domctls. This looks like it should be part of the set_tsc_info hypercall, not a separate hypercall. > > One option to avoid the TSC option is to run domUs with tsc_mode=native. > This has the drawback that migrating a domU from a "2.3GHz" class host > to a "2.4GHz" class host may change the rate at wich the TSC counter > increases, the domU may not be prepared for that. Given that you identify this specifically, why do you think this change is safe or sensible in the first place? If using the options here, time will definitely drift in the VM after migrate. > > With this option the host admin can decide how a domU should behave when > it is migrated across systems of the same class. Since there is always > some jitter when Xen calibrates the cpu_khz value, all hosts of the same > class will most likely have slightly different values. As a result vTSC > emulation is unavoidable. Data collected during the incident which > triggered this change showed a jitter of up to 200 KHz across systems of > the same class. We should see about using better sources of information. For one, many Intel systems actually expose the TSC frequency in the bottom of the PLATFORM_INFO MSR, although this isn't architectural, and has been replaced with CPUID information in Skylake. > > A new utility is added which allows to adjust the vtsc_tolerance_khz > value for running domUs. This is useful to avoid emulation for domUs > that are already running and which can not be restarted. > > The ordering of records sent during migration is important. The value of > vtsc_tolerance_khz must be known by the receiving host before > configuring TSC, because this is the place where the decision of vTSC > emulation is made. Therefore the existing write_tsc_info function is > modified to enforce that ordering. Right, in which case this must be part of set_tsc_info and the TSC record. We already have too many problems with co-dependent ordering, and I'm trying to remove them. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
