Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/15 10:49, Liuxiangdong wrote: On 2021/4/15 9:38, Xu, Like wrote: On 2021/4/14 22:49, Liuxiangdong wrote: Hi Like, On 2021/4/9 16:46, Like Xu wrote: Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. Sorry, in fact, I don't quite understand "My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. " For testcase: perf record -e instructions:pp ./workload We can get 2242 samples on the ICX guest, but only 17 samples or less on the Skylake guest. In my testcase on Skylake, neither the host nor the guest triggered the soft lock. Thanks for your explanation！ Could you please show your complete qemu command and qemu version used on Skylake? I hope I can test it again according to your qemu cmd and version. A new version is released and you may have a try. qemu command: "-enable-kvm -cpu host,migratable=no" qemu base commit: db55d2c9239d445cb7f1fa8ede8e42bd339058f4 kvm base commit: f96be2deac9bca3ef5a2b0b66b71fcef8bad586d diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index 63c55f45ca92..727f55400eaf 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -5618,6 +5618,7 @@ __init int intel_pmu_init(void) case INTEL_FAM6_KABYLAKE: case INTEL_FAM6_COMETLAKE_L: case INTEL_FAM6_COMETLAKE: + x86_pmu.pebs_vmx = 1; x86_add_quirk(intel_pebs_isolation_quirk); x86_pmu.late_ack = true; memcpy(hw_cache_event_ids, skl_hw_cache_event_ids, sizeof(hw_cache_event_ids)); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 100a749251b8..9e37e3dbe3ae 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -150,9 +150,8 @@ static void pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, * the accuracy of the PEBS profiling result, because the "event IP" * in the PEBS record is calibrated on the guest side. */ - attr.precise_ip = 1; - if (x86_match_cpu(vmx_icl_pebs_cpu) && pmc->idx == 32) - attr.precise_ip = 3; + attr.precise_ip = x86_match_cpu(vmx_icl_pebs_cpu) ? + ((pmc->idx == 32) ? 3 : 1) : ((pmc->idx == 1) ? 3 : 1); } event = perf_event_create_kernel_counter(, -1, current, And, I have used "-overcommit mem-lock=on" when soft lockup happens. I misunderstood the use of "mem-lock=on". It is not the same as the guest mem pin and I believe more kernel patches are needed. Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest numa nodes has 1G mem. When I use pebs(perf record -e cycles:pp) in guest, there are successful pebs samples just for a while and then I cannot get pebs samples. Host doesn't soft lockup in this process. In the worst case, no samples are expected. Are there something wrong on skylake for we can only get a few samples? IRQ? Or using hugepage is not effecitve? The few samples comes from hardware limitation. The Skylake doesn't have this "EPT-Friendly PEBS" capabilityand some PEBS records will be lost when used by guests. Thanks! On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/15 9:38, Xu, Like wrote: On 2021/4/14 22:49, Liuxiangdong wrote: Hi Like, On 2021/4/9 16:46, Like Xu wrote: Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. Sorry, in fact, I don't quite understand "My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. " For testcase: perf record -e instructions:pp ./workload We can get 2242 samples on the ICX guest, but only 17 samples or less on the Skylake guest. In my testcase on Skylake, neither the host nor the guest triggered the soft lock. Thanks for your explanation！ Could you please show your complete qemu command and qemu version used on Skylake? I hope I can test it again according to your qemu cmd and version. And, I have used "-overcommit mem-lock=on" when soft lockup happens. I misunderstood the use of "mem-lock=on". It is not the same as the guest mem pin and I believe more kernel patches are needed. Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest numa nodes has 1G mem. When I use pebs(perf record -e cycles:pp) in guest, there are successful pebs samples just for a while and then I cannot get pebs samples. Host doesn't soft lockup in this process. In the worst case, no samples are expected. Are there something wrong on skylake for we can only get a few samples? IRQ? Or using hugepage is not effecitve? The few samples comes from hardware limitation. The Skylake doesn't have this "EPT-Friendly PEBS" capabilityand some PEBS records will be lost when used by guests. Thanks! On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/14 22:49, Liuxiangdong wrote: Hi Like, On 2021/4/9 16:46, Like Xu wrote: Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. Sorry, in fact, I don't quite understand "My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. " For testcase: perf record -e instructions:pp ./workload We can get 2242 samples on the ICX guest, but only 17 samples or less on the Skylake guest. In my testcase on Skylake, neither the host nor the guest triggered the soft lock. And, I have used "-overcommit mem-lock=on" when soft lockup happens. I misunderstood the use of "mem-lock=on". It is not the same as the guest mem pin and I believe more kernel patches are needed. Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest numa nodes has 1G mem. When I use pebs(perf record -e cycles:pp) in guest, there are successful pebs samples just for a while and then I cannot get pebs samples. Host doesn't soft lockup in this process. In the worst case, no samples are expected. Are there something wrong on skylake for we can only get a few samples? IRQ? Or using hugepage is not effecitve? The few samples comes from hardware limitation. The Skylake doesn't have this "EPT-Friendly PEBS" capabilityand some PEBS records will be lost when used by guests. Thanks! On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
Hi Like, On 2021/4/9 16:46, Like Xu wrote: Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. Sorry, in fact, I don't quite understand "My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. " And, I have used "-overcommit mem-lock=on" when soft lockup happens. Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest numa nodes has 1G mem. When I use pebs(perf record -e cycles:pp) in guest, there are successful pebs samples just for a while and then I cannot get pebs samples. Host doesn't soft lockup in this process. Are there something wrong on skylake for we can only get a few samples? IRQ? Or using hugepage is not effecitve? Thanks! On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/12 23:25, Andi Kleen wrote: The reason why soft lockup happens may be the unmapped EPT pages. So, do we have a way to map all gpa before we use pebs on Skylake? Can you configure a VT-d device, that will implicitly pin all pages for the IOMMU. I *think* that should be enough for testing. -Andi Thanks! But, it doesn't seem to work because host still soft lockup when I configure a SR-IOV direct network card for vm. Besides, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest numa nodes has 1G mem. When I use pebs (perf record -e cycles:pp) in guest, there are successful pebs samples on skylake just for a while and then I cannot get pebs sample. Host doesn't soft lockup in this process. Is this method effective? Are there something wrong on skylake for we can only get a few samples ? Maybe IRQ?
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
> The reason why soft lockup happens may be the unmapped EPT pages. So, do we > have a way to map all gpa > before we use pebs on Skylake? Can you configure a VT-d device, that will implicitly pin all pages for the IOMMU. I *think* that should be enough for testing. -Andi
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/9 16:46, Like Xu wrote: Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. We have used "-overcommit mem-lock=on" for qemu when soft lockup. It seems that ept violation happens when we use pebs. [ 5199.056246] Call Trace: [ 5199.056248] _raw_spin_lock+0x1b/0x20[ 5199.056251] follow_page_pte+0xf5/0x580 [ 5199.056258] __get_user_pages+0x1d6/0x750[ 5199.056262] get_user_pages_unlocked+0xdc/0x310 [ 5199.056265] __gfn_to_pfn_memslot+0x12d/0x4d0 [kvm] [ 5199.056304] try_async_pf+0xcc/0x250 [kvm] [ 5199.056337] direct_page_fault+0x413/0xa90 [kvm] [ 5199.056367] kvm_mmu_page_fault+0x77/0x5e0 [kvm] [ 5199.056395] ? vprintk_emit+0xa2/0x240 [ 5199.056399] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 5199.056407] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 5199.056412] vmx_handle_exit+0xfe/0x640 [kvm_intel] [ 5199.056418] vcpu_enter_guest+0x904/0x1450 [kvm] [ 5199.056445] ? kvm_apic_has_interrupt+0x44/0x80 [kvm] [ 5199.056472] ? apic_has_interrupt_for_ppr+0x62/0x90 [kvm] [ 5199.056498] ? kvm_arch_vcpu_ioctl_run+0xeb/0x550 [kvm] [ 5199.056523] kvm_arch_vcpu_ioctl_run+0xeb/0x550 [kvm] [ 5199.056547] kvm_vcpu_ioctl+0x23e/0x5b0 [kvm] [ 5199.056568] __x64_sys_ioctl+0x8e/0xd0 [ 5199.056571] do_syscall_64+0x33/0x40 [ 5199.056574] entry_SYSCALL_64_after_hwframe+0x44/0xae SDM 17.4.9.2 "Setting Up the DS Save Area" says: The recording of branch records in the BTS buffer (or PEBS records in the PEBS buffer) may not operate properly if accesses to the linear addresses in any of the three DS save area sections cause page faults, VM exits, or the setting of accessed or dirty flags in the paging structures (ordinary or EPT). For that reason, system software should establish paging structures (both ordinary and EPT) to prevent such occurrences. Implications of this may be that an operating system should allocate this memory from a non-paged pool and that system software cannot do “lazy” page-table entry propagation for these pages. Some newer processor generations support “lazy” EPT page-table entry propagation for PEBS; see Section 18.3.10.1 and Section 18.9.5 for more information. A virtual-machine monitor may choose to allow use of PEBS by guest software only if EPT maps all guest-physical memory as present and read/write. The reason why soft lockup happens may be the unmapped EPT pages. So, do we have a way to map all gpa before we use pebs on Skylake? On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
Hi Liuxiangdong, On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ My expectation is that there may be many fewer PEBS samples on Skylake without any soft lockup. You may need to confirm the statement "All that matters is that the EPT pages don't get unmapped ever while PEBS is active" is true in the kernel level. Try "-overcommit mem-lock=on" for your qemu. On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
Do you have any comments or ideas about it ? https://lore.kernel.org/kvm/606e5ef6.2060...@huawei.com/ On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/6 13:14, Xu, Like wrote: Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Yes. In fact, I have tried using this patch set in Skylake after these modifications: 1. Expose x86_pmu.pebs_vmx for Skylake. 2. Use PMC0-PMC3 for pebs 2.1 Replace "INTEL_PMC_IDX_FIXED + x86_pmu.num_counters_fixed" with "x86_pmu.max_pebs_events" in "x86_pmu_handle_guest_pebs" 2.2 Unmask other unsupported bits in the pmu->pebs_enable_mask. IA32_PERF_CAPABILITIES.PEBS_BASELINE [bit 14] is always 0 in Skylake, so pmu->pebs_enable_mask equals `((1ull << pmu->nr_arch_gp_counters)-1). 2.3 Replace "pmc->idx == 32 " with "pmc->idx == 1" because the PDIR counter on the Skylake is the gp counter 1. 3. Shield patch-09 because Skylake does not support adaptive pebs. 4. Shield all cpu check code in this patch set just for test. But, unfortunately, guest will record only a few seconds and then host will certainly soft lockup . Is there anything wrong? Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On Wed, Apr 07, 2021 at 11:05:20AM +0800, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: > > > On 2021/4/6 20:47, Andi Kleen wrote: > > > AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake > > > doesn't. > > > But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose > > > counter in Skylake. > > > Is there anything else that only Icelake supports in this patches set? > > Only Icelake server has the support for recovering from a EPT violation > > on the PEBS data structures. To use it on Skylake server you would > > need to pin the whole guest, but that is currently not done. > Sorry. Some questions about "Pin the whole guest". Do you mean VmPin equals > VmSize > in "/proc/$(pidof qemu-kvm)/status"? Or just VmLck equals VmSize? Or > something else? Either would be sufficient. All that matters is that the EPT pages don't get unmapped ever while PEBS is active. -Andi
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
On 2021/4/6 20:47, Andi Kleen wrote: AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. Is there anything else that only Icelake supports in this patches set? Only Icelake server has the support for recovering from a EPT violation on the PEBS data structures. To use it on Skylake server you would need to pin the whole guest, but that is currently not done. Sorry. Some questions about "Pin the whole guest". Do you mean VmPin equals VmSize in "/proc/$(pidof qemu-kvm)/status"? Or just VmLck equals VmSize? Or something else? Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? You would need a guest kernel that supports Icelake server PEBS. 4.18 would need backports for tht. -Andi
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
> AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake > doesn't. > But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose > counter in Skylake. > Is there anything else that only Icelake supports in this patches set? Only Icelake server has the support for recovering from a EPT violation on the PEBS data structures. To use it on Skylake server you would need to pin the whole guest, but that is currently not done. > Besides, we have tried this patches set in Icelake. We can use pebs(eg: > "perf record -e cycles:pp") > when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum > guest kernel version requirement? You would need a guest kernel that supports Icelake server PEBS. 4.18 would need backports for tht. -Andi
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
Hi Xiangdong, On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service Product Dept.) wrote: Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. Yes, we have documented this "EPT-friendly PEBS" capability in the SDM 18.3.10.1 Processor Event Based Sampling (PEBS) Facility And again, this patch set doesn't officially support guest PEBS on the Skylake. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. For Skylake, only the PMC0-PMC3 are valid for PEBS and you may mask the other unsupported bits in the pmu->pebs_enable_mask. Is there anything else that only Icelake supports in this patches set? The PDIR counter on the Ice Lake is the fixed counter 0 while the PDIR counter on the Sky Lake is the gp counter 1. You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? The Ice Lake CPU model has been added since v5.4. You may double check whether the stable tree(s) code has INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. Thanks, Xiangdong Liu
Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers
Hi，like. Some questions about this new pebs patches set： https://lore.kernel.org/kvm/20210329054137.120994-2-like...@linux.intel.com/ The new hardware facility supporting guest PEBS is only available on Intel Ice Lake Server platforms for now. AFAIK， Icelake supports adaptive PEBS and extended PEBS which Skylake doesn't. But we can still use IA32_PEBS_ENABLE MSR to indicate general-purpose counter in Skylake. Is there anything else that only Icelake supports in this patches set? Besides, we have tried this patches set in Icelake. We can use pebs(eg: "perf record -e cycles:pp") when guest is kernel-5.11, but can't when kernel-4.18. Is there a minimum guest kernel version requirement? Thanks, Xiangdong Liu
