Bug#1063270: The "64bits time_t transition" in Debian/Xen
On 12/02/2024 5:27 pm, zithro wrote:
> Hey all,
>
> the Debian project is focused on the "2038 time_t" switch.
> So the maintainers of the Debian Xen package must ensure that all
> imported Xen code conforms to the new Debian standards.
>
> I was asked by Andrew Cooper to post here about this, I'll quote him :
> "So I had been idly wondering whether Xen would match up to Debian's new
> policy, and it appears not
> this topic really needs to be brought up on the xen-devel mailing list
> do you have any more details as to what has gone wrong?
> this is something we ought to arrange to happen in CI by default
> but it sounds like there's some work needed first"
>
> (Not answering the question because I'm just a messenger).
xen.git/xen$ git grep -w time_t -- :/
../tools/console/client/main.c:106: time_t start, now;
../tools/console/daemon/io.c:272: time_t now = time(NULL);
../tools/libs/light/libxl_qmp.c:116: time_t timeout;
../tools/libs/light/libxl_qmp.c:585:
time_t ask_timeout)
../tools/libs/light/libxl_x86.c:516: time_t t;
../tools/libs/toollog/xtl_logger_stdio.c:61: time_t now = time(0);
../tools/tests/xenstore/test-xenstore.c:453: time_t stop;
../tools/xenmon/xenbaked.c:98:time_t start_time;
../tools/xenstored/core.c:109: time_t now;
../tools/xenstored/core.h:150: time_t ta_start_time;
../tools/xenstored/domain.c:143: time_t mem_last_msg;
../tools/xenstored/domain.c:188:static time_t wrl_log_last_warning; /*
0: no previous warning */
../tools/xenstored/domain.c:1584: time_t now;
../tools/xenstored/lu.c:160: time_t now = time(NULL);
../tools/xenstored/lu.c:185: time_t now = time(NULL);
../tools/xenstored/lu.c:292: time_t now = time(NULL);
../tools/xenstored/lu.h:32: time_t started_at;
../tools/xentop/xentop.c:947: time_t curt;
../tools/xl/xl_info.c:742:static char *current_time_to_string(time_t now)
../tools/xl/xl_info.c:759:static void print_dom0_uptime(int short_mode,
time_t now)
../tools/xl/xl_info.c:810:static void print_domU_uptime(uint32_t domuid,
int short_mode, time_t now)
../tools/xl/xl_info.c:847: time_t now;
../tools/xl/xl_vmcontrol.c:336: time_t start;
../tools/xl/xl_vmcontrol.c:495: time_t now;
../tools/xl/xl_vmcontrol.c:504: if (now == ((time_t) -1)) {
../tools/xs-clients/xenstore_control.c:33: time_t time_start;
arch/x86/cpu/mcheck/mce.h:224: uint64_t time; /* wall time_t when
error was detected */
arch/x86/time.c:1129: * machines were long is 32-bit! (However, as
time_t is signed, we
I don't see any ABI problems from using a 64bit time_t. The only header
file with a time_t is xenstored/lu.h which is a private header and not a
public ABI.
I guess we fell into the "could not be analysed via
abi-compliance-checker" case?
~Andrew
Bug#850425: mpt3sas "swiotlb buffer is full" problem only under Xen
On 10/03/2019 23:12, Hans van Kranenburg wrote: > On 3/10/19 11:03 PM, Andrew Cooper wrote: >> On 10/03/2019 21:35, Hans van Kranenburg wrote: >>> [...] >>> >>> Using dom0_mem=2GiB,max:4GiB instead of dom0_mem=2GiB,max:2GiB (which I >>> started with) makes the errors go away, so workaround confirmed. > It's actually dom0_mem=2G,max:4G, I typed something before which was not > identical to what I started that machine with. > >>> [...] >>> I think I'm fine with this workaround. >> I think >> https://lists.xen.org/archives/html/xen-devel/2014-12/msg00699.html is >> the last attempt David made to upstream the fixes. >> >> Linux is still broken, and these fixes are still necessary. > FMI, is this max:4G DTRT for me now in the meantime, or am I still > facing more hidden problems while using this workaround? I believe that is good enough to work around the problem. The issue is that the these drivers are using dom0's max ram (well - max pfn to be specific) to decide whether it reports itself as 32bit or 64bit DMA capable, which is nonsense. Therefore, if dom0 thinks it has less that 4G of potential RAM, the driver reports the device as only 32-bit capable, and bounce-buffers all of its DMA (as dom0 is generally allocated above the 4G boundary in physical address space). ~Andrew
Bug#850425: mpt3sas "swiotlb buffer is full" problem only under Xen
On 10/03/2019 21:35, Hans van Kranenburg wrote: > found -1 4.19.20-1 > thanks > > Hi, > > Reviving a thing from Jan 2017 here. I don't have this thread in my > mailbox, so no inline quotes. > > I just installed some HP z820 workstation and rebooted it into Xen > 4.11.1+26-g87f51bf366-3 with linux 4.19.20-1 as dom0 kernel. > > During boot I'm greeted by a long list of... > > [ 14.518793] mpt3sas :02:00.0: swiotlb buffer is full (sz: 65536 > bytes) > [ 14.518899] mpt3sas :02:00.0: swiotlb buffer is full > [ 14.518956] mpt3sas :02:00.0: swiotlb buffer is full (sz: 65536 > bytes) > [ 14.518988] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes! > [ 14.519081] mpt3sas :02:00.0: swiotlb buffer is full > [ 14.519309] sd 6:0:1:0: pci_map_sg failed: request for 1310720 bytes! > [ 14.524611] mpt3sas :02:00.0: swiotlb buffer is full (sz: 65536 > bytes) > [ 14.527309] mpt3sas :02:00.0: swiotlb buffer is full > [ 14.527405] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes! > [...] > > ...and some hangs here and there. This indeed did not happen when > booting just Linux, without Xen. > > Some searching brought me to this Debian bug. So, thanks for writing > down all kinds of research here already. Even if it's not fixed upstream > yet, this helps a lot. :-) > > Using dom0_mem=2GiB,max:4GiB instead of dom0_mem=2GiB,max:2GiB (which I > started with) makes the errors go away, so workaround confirmed. > > I can try any of the linked patches, but I see that in message 54, > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850425#54 > Andrew says: "IIRC, they were essentially rejected,". Next message, Ian > asks "Do you have a reference ?", but I don't see any fup on that. > > I think I'm fine with this workaround. > > If someone will ever work on the upstream patches, then this is just to > let know that I might be able to help testing. However, I only have one > of this type of box and it's gonna be installed as server at some > non-profit organization without OOB access, replacing even older donated > hardware, so, it will be kinda limited... :) I think https://lists.xen.org/archives/html/xen-devel/2014-12/msg00699.html is the last attempt David made to upstream the fixes. Linux is still broken, and these fixes are still necessary. Boris/Juergen: Any chance you could look into these patches? I have no idea what they they're in against master, but its also liable its now more complicated with the host max mfn calculations which have gone in more recently. ~Andrew
Bug#810964: [Xen-devel] [BUG] EDAC infomation partially missing
On 16/05/17 10:54, Jan Beulich wrote: On 16.05.17 at 05:47, wrote: >> On Mon, May 15, 2017 at 02:02:53AM -0600, Jan Beulich wrote: >> On 14.05.17 at 00:36, wrote: I haven't yet done as much experimentation as Andreas Pflug has, but I can confirm I'm also running into this bug with Xen 4.4.1. I've only tried Linux kernel 3.16.43, but as Dom0: EDAC MC: Ver: 3.0.0 AMD64 EDAC driver v3.4.0 EDAC amd64: DRAM ECC enabled. EDAC amd64: NB MCE bank disabled, set MSR 0x017b[4] on node 0 to enable. EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. AMD64 EDAC driver v3.4.0 EDAC amd64: DRAM ECC enabled. EDAC amd64: NB MCE bank disabled, set MSR 0x017b[4] on node 0 to enable. EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. >>> Afaict the driver as is simply can't work in a Xen Dom0; it needs >>> enabling (read: para-virtualizing). I'm actually glad to see it doesn't >>> load (the worse alternative would be for it to load and then do the >>> wrong thing or give you a false sense of safety of your data). >> I'm unsure of how to evaluate the situation. Since ECC is enabled in the >> BIOS, data should be safe whether or not the EDAC driver loads. I >> /suspect/ the EDAC driver failing to load merely means reportting of ECC >> errors won't happen. > "Merely" being relative here: The missing reports mean a false feeling > of safety, as they may be early indications of later double-bit errors. > >> I suspect the only paravirtualization needed is to >> map the physical address of the soft|hard errors to which VM's memory >> range was effected. What this effects is which VM should panic in case >> of hard errors. > Which in turn obviously requires hypervisor interaction. It's not really > clear to me whether perhaps the driver would better live in the > hypervisor in the first place for that reason. The driver should probably live directly in Xen; it needs to program a number of nothbridge and CPU registers including interrupt information. For the reporting side of things, it looks like it would require vMCE to pass on fault information to guests. ~Andrew
Bug#852324: [Xen-devel] Bug#852324: x86/mm: Found insecure W+X mapping
On 17/03/17 03:05, Boris Ostrovsky wrote: > > > On 03/16/2017 08:18 PM, Ben Hutchings wrote: >> On Thu, 2017-03-16 at 21:49 +, Andrew Cooper wrote: >>> On 16/03/2017 21:05, Ben Hutchings wrote: >>>> On Thu, 2017-03-16 at 00:50 +, Ben Hutchings wrote: >>>>> On Wed, 2017-03-15 at 22:24 +, Ben Hutchings wrote: >>>>>> Control: retitle -1 [xen] x86/mm: Found insecure W+X mapping >>>>>> Control: tag -1 upstream confirmed >>>>>> Control: found -1 4.9.13-1 >>>>>> >>>>>> I can reproduce this with a current Debian kernel on top of Xen 4.4. >>>>>> It doesn't happen with the same hardware booting the kernel >>>>>> directly. >>>>> >>>>> With CONFIG_X86_PTDUMP enabled, I can see that the first 16 MiB of >>>>> the >>>>> low kernel mapping is mapped with W+X permissions, with a few >>>>> exceptions: >>>>> >>>>> 0x8800-0x88099000 612K USR >>>>> RW x pte >>>>> 0x88099000-0x8809a000 4K USR >>>>> ro NX pte >>>>> 0x8809a000-0x8809b000 4K USR >>>>> ro x pte >>>>> 0x8809b000-0x8809f000 16K USR >>>>> RW NX pte >>>>> 0x8809f000-0x8810 388K USR RW PWT >>>>> PCD x pte >>>>> 0x8810-0x88102000 8K USR >>>>> RW x pte >>>>> 0x88102000-0x88000100 15352K USR >>>>> RW x pte >>>>> >>>>> This accounts for all the 4090 pages reported at boot. >>>> >>>> I see this same mapping when running Linux 4.9 under either Xen 4.4 or >>>> 4.8 (from Debian stable or unstable). >>>> >>>> I don't really understand how the PV MMU page tables are set up. I >>>> did >>>> try setting the NX flag in make_lowmem_page_readwrite() and that >>>> didn't >>>> make any difference to the number of W+X pages. >>> >>> 64bit PV guests have some rather special pagetable set up. For one, >>> the >>> USR bit will be unconditionally set and Xen doesn't tolerate some >>> mappings being writeable, >> >> I understood that much. >> >>> but these RWX areas are clearly ok in Xens eyes. >> >> So that proves these pages aren't mapped to native page tables or other >> sensitive structures, right? >> >>> Everything from 0x8800 upwards belongs to the guest kernel >>> per the PV ABI. The initial layout will be constructed by the domain >>> builder on behalf of the kernel, but it is Linux's to arbitrarily alter >>> later on. >> >> But it seems to avoid doing that in generic code - see phys_pte_init() >> in arch/x86/mm/init_64.c. > > > I believe that's because we are reusing pre-built tables which don't > have NX bit set, see xen_setup_kernel_pagetable(). Right, in which case the W+X warning is entirely correct and pointing out a security weakness. The domain builder can't usefully know exactly how Linux wants its NX and RO mappings at boot, particularly if the init code wants to rely on things being RW to start with. xen_setup_kernel_pagetable() should be altered to make suitable permission alterations to the provided initial pagetables. ~Andrew
Bug#852324: [Xen-devel] Bug#852324: x86/mm: Found insecure W+X mapping
On 16/03/2017 21:05, Ben Hutchings wrote: > On Thu, 2017-03-16 at 00:50 +, Ben Hutchings wrote: >> On Wed, 2017-03-15 at 22:24 +, Ben Hutchings wrote: >>> Control: retitle -1 [xen] x86/mm: Found insecure W+X mapping >>> Control: tag -1 upstream confirmed >>> Control: found -1 4.9.13-1 >>> >>> I can reproduce this with a current Debian kernel on top of Xen 4.4. >>> It doesn't happen with the same hardware booting the kernel directly. >> With CONFIG_X86_PTDUMP enabled, I can see that the first 16 MiB of the >> low kernel mapping is mapped with W+X permissions, with a few >> exceptions: >> >> 0x8800-0x88099000 612K USR RW >> x pte >> 0x88099000-0x8809a000 4K USR ro >> NX pte >> 0x8809a000-0x8809b000 4K USR ro >> x pte >> 0x8809b000-0x8809f000 16K USR RW >> NX pte >> 0x8809f000-0x8810 388K USR RW PWT PCD >> x pte >> 0x8810-0x88102000 8K USR RW >> x pte >> 0x88102000-0x88000100 15352K USR RW >> x pte >> >> This accounts for all the 4090 pages reported at boot. > I see this same mapping when running Linux 4.9 under either Xen 4.4 or > 4.8 (from Debian stable or unstable). > > I don't really understand how the PV MMU page tables are set up. I did > try setting the NX flag in make_lowmem_page_readwrite() and that didn't > make any difference to the number of W+X pages. 64bit PV guests have some rather special pagetable set up. For one, the USR bit will be unconditionally set and Xen doesn't tolerate some mappings being writeable, but these RWX areas are clearly ok in Xens eyes. Everything from 0x8800 upwards belongs to the guest kernel per the PV ABI. The initial layout will be constructed by the domain builder on behalf of the kernel, but it is Linux's to arbitrarily alter later on. Can you boot a debug hypervisor and look at `xl dmesg` starting at the "*** LOADING DOMAIN 0 ***" line? This should dump the state that the domain builder leaves dom0 in, which would help identify whether those regions are leftovers from the domain builder, or something constructed by the Linux PV early boot code. ~Andrew
Bug#850425: Debian bug #850425 - mpt3sas "swiotlb buffer is full" problem only under Xen
On 16/01/17 13:00, Ian Jackson wrote:
> Andy Smith writes ("Re: Bug#850425: Debian bug #850425 - mpt3sas "swiotlb
> buffer is full" problem only under Xen"):
>> On Mon, Jan 16, 2017 at 11:47:54AM +, Ian Jackson wrote:
>>> Right. Well, you need to post to linux-kernel saying "this patch
>>> fixes such-and-such". You should:
> ...
>> I see. This is not something I've done before, but I'd be willing to
>> give it a go.
>>
>> But, I am not the author of these two patches. That's David Vrabel,
>> and they're from something Andrew Cooper referred to as "the
>> XenServer Patch queue", here:
> Indeed.
>
>> At the present time I don't actually know in detail what these
>> patches do, nor how they have been tested, etc.; I only know that
>> they fix the problem I was seeing.
>>
>> Is it appropriate for me to be the one presenting them to
>> linux-kernel and the maintainers and so on?
> Yes, it is normal to be passing on other people's patches. Although
> of course if there are questions about them then you may not be able
> to answer.
>
> You should CC the patch authors (generally, everyone named with a
> Signed-off-by or a CC) on your emails.
>
> FYI David Vrabel doesn't work for Citrix any more but I'm hoping if
> there are questions Andrew Cooper will be able to help.
I am not sure how best to proceed. IIRC, they were essentially
rejected, without any suitable alternative proposed by the maintainers.
~Andrew
Bug#812166: [PATCH] x86/mce: fix misleading indentation in init_nonfatal_mce_checker().
On 22/01/16 14:38, Ian Campbell wrote:
> Debian bug 812166[0] reported this build failure due to
> Wmisleading-indentation with gcc-6:
>
> non-fatal.c: In function 'init_nonfatal_mce_checker':
> non-fatal.c:103:2: error: statement is indented as if it were guarded by...
> [-Werror=misleading-indentation]
> switch (c->x86_vendor) {
> ^~
>
> non-fatal.c:97:5: note: ...this 'if' clause, but it is not
> if ( __get_cpu_var(poll_bankmask) == NULL )
> ^~
>
> I was unable to reproduce (xen builds cleanly for me with "6.0.0 20160117
> (experimental) [trunk revision 232481]") but looking at the code the issue
> above is clearly real.
>
> Correctly reindent the if statement.
>
> This file uses Linux coding style (infact the use of Xen style for
> this line is the root cause of the wanring) so use tabs and while
> there remove the whitespace inside the if as Linux does.
>
> [0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812166
>
> Signed-off-by: Ian Campbell
> Cc: Christoph Egger
> Cc: Liu Jinsong
Reviewed-by: Andrew Cooper
Bug#759018: [Xen-devel] Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use
On 04/09/14 15:25, Ian Campbell wrote:
> On Thu, 2014-09-04 at 15:15 +0100, Ian Jackson wrote:
>> Colin Watson writes ("Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen
>> binaries for host (dom0) use"):
>> ...
>>> There is also the question of whether the guest-side name should mention
>>> GRUB. One might argue that it shouldn't because all that matters is
>>> that it uses the Multiboot protocol. Then there is the question of who
>>> gets to own the architecture names ...
>> The general scheme seems sound.
>>
>>
>> Ian Campbell:
>>
I'm not sure what the best way to promulgate the spec is -- I
think a patch to add xen.git/docs/misc/pvgrub2.markdown would be
sufficient (it would end up under http://xenbits.xen.org/docs/).
>> Since this path is in /boot/xen and contains an executable which
>> expects to run in the Xen PV environment, it could also use Xen names
>> for the architectures. I don't know whether a GNU config triplet arch
>> name as you suggest is easier or harder than that.
> About the same. Colin suggested the GNU config triplet names in his
> strawman and I couldn't think of a reason to change.
>
>> I have a question about the spec's wording about bitness:
>>
>> +It is not in general possible under Xen for a bootloader to boot a
>> +kernel of a different width from itself, and this extends to
>> +chainloading from a stage one. Therefore it is permissible to have
>> +both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf`
>> +present in a guest to be used by the appropriate stage 1 (e.g. for
>> +systems with 32-bit userspace and an optional 64-bit kernel).
>>
>> Is it therefore expected that the host admin will be told out of band
>> what bitness the guest would prefer ? And that then the host
>> toolstack will set up that bitness of guest, load its pvgrub for that
>> bitness, and hope that the guest has an appropriate-bitness core image
>> load in the canonical place ?
> Yes. Essentially you write kernel = /path/too/pvgrub-<32bit-name> or
> -<64bit-name> in your guest config. Yes, this sucks.
>
> AIUI in cloud interfaces etc you generally have to ask for a 32- or
> 64-bit domain explicitly too.
>
>> I wonder if we might, in the future, want this to be more automatic.
> I suppose that Would Be Nice(tm).
>
> I'm not sure but it might be that for a PVH guest (once grub is ported
> to that) we might be able to switch bitness at runtime and avoid this
> whole mess (which comes largely from the size of the p2m entries and the
> difficulties in switching, which is hidden from pvh guests)
>
>> I guess we could have a feature in the host's 64-bit pvgrub which
>> would look for and load 64-bit guest pvgrub if it exists, and
>> alternatively check for 32-bit guest pvgrub and if found exit
>> signalling somehow to the host toolstack to restart the domain with
>> the other bitness.
>>
>> But what if the host has both 64- and 32-bit pvgrub but in fact only
>> has one bitness of kernel ? Signalling this back to the host by
>> somehow hiding or renaming one of the bitnesses of guest pvgrub seems
>> unpleasant.
> You mean if all guests happen to only use one bitness? You waste a
> roundtrip through a stunt domain which will do the exiting trick and
> restart, or you supply the right dom0 grub to start with I guess.
There is no real technical obstacle to prevent PV domains switching
width after starting.
In the past, the toolstack has directly loaded the target running
kernel, but that behaviour/assumption is no longer correct given these
chainloading plans.
As we no longer support 2-level PV guests, allowing a PV domain to
switch between 3 and 4 levels becomes easier to manage from Xens point
of view.
>From the top of my head, it would involve relaxing the restriction that
3 level PV guests can't pin L4 tables (but still enforce that a 3 level
PV guest can't load an L4 pagetable), and provide a new hypercall which
takes a desired with, new cr3 (referring to a pinned pagetable of the
appropriate new width) and a new eip to jump to.
~Andrew
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#748052: [Xen-devel] dom0 USB failing with "ehci-pci: probe of 0000:00:1d.0 failed with error -110"
On 16/05/14 11:08, Jan Beulich wrote: On 16.05.14 at 10:58, wrote: >> So it seems like dom0 is unable to (correctly) bind to some hardware >> interrupts. I wonder if these messages from Xen's dmesg are relevant. >> (XEN) Not enabling x2APIC: depends on iommu_supports_eim. >> (XEN) I/O virtualisation disabled >> (XEN) Enabled directed EOI with ioapic_ack_old on! > The last one certainly isn't, and the first two shouldn't (albeit the > non-Xen kernel is running in x2APIC mode). That difference is likely > because the Xen and non-Xen boots are with differing BIOS > configurations, or on different machines: The non-Xen boot shows > a DMAR ACPI table, while the Xen one doesn't. Or wait, no, the > hypervisor and kernel-under-Xen logs differ in that respect too. We > clearly need a consistent set of logs. > > The one clearly odd thing in the hypervisor logs are these two lines > > (XEN) traps.c:3061: GPF (): 82c4c0186a91 -> 82c4c0218daa > (XEN) traps.c:3061: GPF (): 82c4c0186a91 -> 82c4c0218daa > > Can at least the left side address please be associated back with a > symbol (with the help of xen-syms perhaps)? As this is a debug message anyway, it would be sensible for the extable handler to print %pS of the left hand address. It would avoid needing to refer back to xen-syms in cases like this. ~Andrew -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#665433: hypervisor fault in move_masked_irq
On 14/07/12 21:46, Ian Campbell wrote: > tags 665433 +upstream > thanks > > Hi Andrew, > > This [0] Debian bug report (against 4.0) looks like the sort of thing > you might have fixed (or perhaps worked around) in one of your many > fixes to the IRQ stuff in 4.1/unstable. Does it look at all familiar? Unfortunately it doesn't look too familiar. Judging by the fact that Xen has jumped outside of its code space, I would say that Xen has made a function call off an invalid function pointer. Given that desc->handler->set_affinity() is the only function pointer call in the function, this is possibly a race condition between dom0 dying (which the upper stack trace indicates), Xen cleaning up after dom0, and Xen receiving an interrupt which was midway through being migrated. Furthermore, it appears that unstable might be vulnerable to the same race condition. ~Andrew > > Cheers, > Ian. > > [0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665433 > -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
