Re: Xens handling of MCE
On Thu, Sep 07, 2023 at 08:56:51AM +0200, Jan Beulich wrote: > On 06.09.2023 23:38, Elliott Mitchell wrote: > > On Thu, Aug 31, 2023 at 07:52:05PM +, Development wrote: > >> > >> However, in 2009-02, "cegger" wrote MCA/MCE_in_Xen, a proposal for > >> having xen start checking the information > >> Xen started accessing the EDAC information (now called "MCE") at some > >> point after that, which blocks the linux kernel in dom0 from accessing it. > >> (I also found what appears to be related sides from a presentation > >> from 2012 at: > >> https://lkml.iu.edu/hypermail/linux/kernel/1206.3/01304/xen_vMCE_design_%28v0_2%29.pdf > >> ) > >> > > > > I hadn't seen that before. Clearly shows someone who had no idea what > > they were doing designed it. The author was thinking "virtualize > > everything!", whereas MCE is a perfect situation for paravirtualization. > > Let Dom0 process MCE events (which allows use of Linux's more up to date > > MCE drivers), then let Dom0 notify Xen if action is needed (a page was > > corrupted, tell the effected domain). > > > > There was a recent proposal to simply import Linux's rather more recent > > MCE/EDAC source. This hasn't happened yet. For people using Xen this > > has been a very concerning issue for some time. > > I'm unaware of such a proposal; do you have a reference? EDAC drivers > typically are vendor- or even chipset-specific aiui. At least the latter > wouldn't make them a good fit to import into Xen. Along of what you say > earlier, they instead want to become Xen-aware (to deal with address > translation as necessary). That'll also have better chances of things > staying up-to-date. I don't recall who wrote the message, I think it was less than 6 months ago though. I read it as $person had been pondering the idea of simply ripping out Xen's MCE implementation and replacing it with minimally adjusted Linux MCE implementation. What you describe matches my thinking. Even though the EDAC hardware is fully attached to processors now, it doesn't need virtualization similar to page tables. Instead EDAC should be handled similar to most hardware devices and go through Domain 0. The approach for Xen should also differ. Instead of first telling the OS, it might be better to immediately unmap the page and trigger a page fault if it is accessed. Then notify the OS a page has disappeared. Mainly immitate how Linux handles MCE events for a userspace process, rather than the usual paravirtualization. I'm not on sufficiently intimate terms with the drivers or hardware to try this right now. Yet the number of complaints about this is rather substantial (okay, I'm aware since this is no small concern for me too). -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
Re: Xens handling of MCE
On 06.09.2023 23:38, Elliott Mitchell wrote: > On Thu, Aug 31, 2023 at 07:52:05PM +, Development wrote: >> >> However, in 2009-02, "cegger" wrote MCA/MCE_in_Xen, a proposal for >> having xen start checking the information >> Xen started accessing the EDAC information (now called "MCE") at some >> point after that, which blocks the linux kernel in dom0 from accessing it. >> (I also found what appears to be related sides from a presentation from >> 2012 at: >> https://lkml.iu.edu/hypermail/linux/kernel/1206.3/01304/xen_vMCE_design_%28v0_2%29.pdf >> ) >> > > I hadn't seen that before. Clearly shows someone who had no idea what > they were doing designed it. The author was thinking "virtualize > everything!", whereas MCE is a perfect situation for paravirtualization. > Let Dom0 process MCE events (which allows use of Linux's more up to date > MCE drivers), then let Dom0 notify Xen if action is needed (a page was > corrupted, tell the effected domain). > > There was a recent proposal to simply import Linux's rather more recent > MCE/EDAC source. This hasn't happened yet. For people using Xen this > has been a very concerning issue for some time. I'm unaware of such a proposal; do you have a reference? EDAC drivers typically are vendor- or even chipset-specific aiui. At least the latter wouldn't make them a good fit to import into Xen. Along of what you say earlier, they instead want to become Xen-aware (to deal with address translation as necessary). That'll also have better chances of things staying up-to-date. Jan
Re: Xens handling of MCE
On Thu, Aug 31, 2023 at 07:52:05PM +, Development wrote: > > However, in 2009-02, "cegger" wrote MCA/MCE_in_Xen, a proposal for having > xen start checking the information > Xen started accessing the EDAC information (now called "MCE") at some > point after that, which blocks the linux kernel in dom0 from accessing it. > (I also found what appears to be related sides from a presentation from > 2012 at: > https://lkml.iu.edu/hypermail/linux/kernel/1206.3/01304/xen_vMCE_design_%28v0_2%29.pdf > ) > I hadn't seen that before. Clearly shows someone who had no idea what they were doing designed it. The author was thinking "virtualize everything!", whereas MCE is a perfect situation for paravirtualization. Let Dom0 process MCE events (which allows use of Linux's more up to date MCE drivers), then let Dom0 notify Xen if action is needed (a page was corrupted, tell the effected domain). There was a recent proposal to simply import Linux's rather more recent MCE/EDAC source. This hasn't happened yet. For people using Xen this has been a very concerning issue for some time. This seems a combination of not enough people using Xen not yet having gotten quite noisy enough (I think the threshold is being approached), plus the people with the right development skills being out of touch. Commit 767f4b620edadac579c9b8b6660761d4285fa6f9 to the Linux kernel yet again shows someone *REALLY* out of touch! -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
Xens handling of MCE
We have been trying to find documentation on how to tell Xen to forward MCE information to the linux kernel in Dom0 in order to let a system administrator be able to get notified when his system has bad memory. However from what I can tell this has not been documented anywhere. If anyone knows of documentation (or knows the answer) of what someone is supposed to do in order to monitor the corrected errors and monitor the uncorrected errors when they are running modern xen, it would be appreciated. To clarify, (and for people not familiar): When running old xen ( example: Xen 4.1) on a system, linux in dom0 would load the edac modules. example: amd64_edac_mod , edac_mce_amd , and edac_core Once the modules loaded, the error counts for ECC memory, and PCI, could be found in these "files": /sys/devices/system/edac/mc/mc0/ce_count /sys/devices/system/edac/mc/mc0/ue_count /sys/devices/system/edac/pci/pci0/npe_count /sys/devices/system/edac/pci/pci0/pe_count However, in 2009-02, "cegger" wrote MCA/MCE_in_Xen, a proposal for having xen start checking the information Xen started accessing the EDAC information (now called "MCE") at some point after that, which blocks the linux kernel in dom0 from accessing it. (I also found what appears to be related sides from a presentation from 2012 at: https://lkml.iu.edu/hypermail/linux/kernel/1206.3/01304/xen_vMCE_design_%28v0_2%29.pdf ) Now, The linux kernel compile option: CONFIG_XEN_MCE_LOG=y is documented as: "Allow kernel fetching MCE error from Xen platform and converting it into Linux mcelog format for mcelog tools". I imagine there must be some way on the xen side for this to work for CONFIG_XEN_MCE_LOG to have gotten into the linux kernel and be enabled by default in distributions. (notes: mcelog seems to have been replaced with "ras daemon", but I believe that it pulls information using the same kernel APT as "mcelog") (so I believe the final output of if you are having memory errors is pulled by doing "ras-mc-ctl --errors" now instead of looking in /sys/devices/system/edac/mc and /sys/devices/system/edac/pci) I suspect that to check if it was working on a modern system, one would do "ras-mc-ctl --status" and get something implying that the xen mce interface is working instead of just saying "ras-mc-ctl: drivers not loaded." Somewhere it was said that adding the xen boot parameter "mce=1" to grub would cause xen to forward the info to the linux kernel, but that conflicts with recent changes to the documentation. Also, tested by setting "mce=1" and nothing appears to change. Any help is appreciated.
