Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Thu, Feb 23, 2017 at 06:45:50PM +0530, Bhupinder Thakur wrote: > Hi, > > Currently, xenconsoled supports only PV console. To support one > additional pl011 console, some of the data structures in struct domain > and certain functions such as buffer_append, ring_free_bytes, etc. > will have to be be scaled to support now two types of consoles. > > Some changes are required in xenconsole client (client/maiun.c) and > tools/libxl/xl_cmdimpl.c to support one additional pl011 console. > Note that we're in the process of splitting xl out of libxl directory. Please keep an eye on xen.git staging branch. > I have done the above-mentioned changes and can now connect to both PV > and pl011 consoles. > > Command for connecting to pl011 console: > > xl console -t vpl011 connects to the pl011 console. > > Currently, the console type definitions such as LIBXL_CONSOLE_TYPE_PV > are defined in /usr/include/_libxl_types.h, which is not part of the > xen tree. How should I add a new console type in this header file? > See libxl_types.idl. _libxl_types.h is autogenerated from that file. Wei. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, Currently, xenconsoled supports only PV console. To support one additional pl011 console, some of the data structures in struct domain and certain functions such as buffer_append, ring_free_bytes, etc. will have to be be scaled to support now two types of consoles. Some changes are required in xenconsole client (client/maiun.c) and tools/libxl/xl_cmdimpl.c to support one additional pl011 console. I have done the above-mentioned changes and can now connect to both PV and pl011 consoles. Command for connecting to pl011 console: xl console -t vpl011 connects to the pl011 console. Currently, the console type definitions such as LIBXL_CONSOLE_TYPE_PV are defined in /usr/include/_libxl_types.h, which is not part of the xen tree. How should I add a new console type in this header file? Thanks. Regards, Bhupinder On 16 February 2017 at 02:51, Stefano Stabellini wrote: > On Wed, 15 Feb 2017, Bhupinder Thakur wrote: >> On 15 February 2017 at 13:45, Bhupinder Thakur >> wrote: >> > Hi Stefano, >> > >> > On 14 February 2017 at 03:14, Stefano Stabellini >> > wrote: >> >> On Mon, 13 Feb 2017, Bhupinder Thakur wrote: >> >>> Hi Stefano, >> >>> >> >>> On 9 February 2017 at 05:40, Stefano Stabellini >> >>> wrote: >> >>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: >> >>> >> Hi Julien, >> >>> >> >> >>> >> On 3 February 2017 at 19:38, Julien Grall >> >>> >> wrote: >> >>> >> > So if I understand correctly, you don't receive anymore output. >> >>> >> > Correct? >> >>> >> > Have you tried to see whether the pl011 driver is receiving >> >>> >> > interrupt or >> >>> >> > even Linux calling it. >> >>> >> >> >>> >> The pl011 driver is not receiving the interrupts (I put a >> >>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not >> >>> >> see the print on the xen console) . When the console is in this state, >> >>> >> if I keep typing the characters on the console then Xen keeps raising >> >>> >> the interrupts (for each character typed) but all those interrupts are >> >>> >> marked as inflight. So it seems that interrupt is not getting cleared >> >>> >> in GIC. Which GIC function can I look at to further debug what is the >> >>> >> issue? >> >>> >> >> >>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then >> >>> >> I used to receive the interrupts all fine. >> >>> > >> >>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you >> >>> > don't >> >>> > mark the interrupt as a real hardware interrupt by mistake >> >>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 >> >>> > interrupt has been added correctly to the LRs >> >>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr). >> >>> >> >>> I verified that the PL011 interrupt is getting marked pending by Xen >> >>> but guest does not receive it. I am not sure if the local interrupt >> >>> delivery is disabled on the guest because of which it may not be >> >>> receiving the interrupts. Is there a way to verify that interrupts are >> >>> indeed disabled at that point. Since the interrupt delivery works fine >> >>> when I manually read/write characters to /dev/ttyAMA0, I think there >> >>> is no issue from the Xen side in terms of raising an interrupt. >> >>> >> >>> One observation is that pl011_request_port () calls request_mem_region >> >>> () for the MMIO space allocated to pl011. The function returns -EBUSY >> >>> indicating that the memory region is already in use. But when I do cat >> >>> /proc/iomem, I do not see any overlaps. I am using 0x2200 as the >> >>> start of the pl011 MMIO space. I see the same error while booting the >> >>> dom0 kernel also. >> >>> >> >>> I have attached the guest boot logs when console=ttyAMA0. I have added >> >>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c >> >>> and serial_core.c. The output from the console seems to suddenly stop >> >>> once the pl011 port is started. I have not added getty for ttyAMA0 in >> >>> the /etc/inittab file. I believe auto-serial-console should be able to >> >>> attach a getty session to ttyAMA0 automatically (i did try with adding >> >>> a getty command in the inittab file but that did not solve the issue). >> >>> >> >>> Towards the end of the log, we can see that every time a character is >> >>> typed, Xen raises an interrupt and the characters keep getting >> >>> collected in the IN RING buffer (in_prod keeps incrementing). Later >> >>> interrupts remain in inflight as the earlier interrupt is not cleared. >> >> >> >> Let me get this straight: who is printing all those messages we see on >> >> the guest console (xl console guest) up until "Freeing unused kernel >> >> memory"? Is it the early pl011 driver? Or is it the Xen PV console >> >> driver? >> >> >> >> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to >> >> issue HYPERVISOR_console_io hypercalls. You might want to check that >> >> do_console_io in Xen is letting DomU calls through. >> >> >> >> Otherwise pl011_int might act
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, 15 Feb 2017, Bhupinder Thakur wrote: > On 15 February 2017 at 13:45, Bhupinder Thakur > wrote: > > Hi Stefano, > > > > On 14 February 2017 at 03:14, Stefano Stabellini > > wrote: > >> On Mon, 13 Feb 2017, Bhupinder Thakur wrote: > >>> Hi Stefano, > >>> > >>> On 9 February 2017 at 05:40, Stefano Stabellini > >>> wrote: > >>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: > >>> >> Hi Julien, > >>> >> > >>> >> On 3 February 2017 at 19:38, Julien Grall wrote: > >>> >> > So if I understand correctly, you don't receive anymore output. > >>> >> > Correct? > >>> >> > Have you tried to see whether the pl011 driver is receiving > >>> >> > interrupt or > >>> >> > even Linux calling it. > >>> >> > >>> >> The pl011 driver is not receiving the interrupts (I put a > >>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not > >>> >> see the print on the xen console) . When the console is in this state, > >>> >> if I keep typing the characters on the console then Xen keeps raising > >>> >> the interrupts (for each character typed) but all those interrupts are > >>> >> marked as inflight. So it seems that interrupt is not getting cleared > >>> >> in GIC. Which GIC function can I look at to further debug what is the > >>> >> issue? > >>> >> > >>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then > >>> >> I used to receive the interrupts all fine. > >>> > > >>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't > >>> > mark the interrupt as a real hardware interrupt by mistake > >>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 > >>> > interrupt has been added correctly to the LRs > >>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr). > >>> > >>> I verified that the PL011 interrupt is getting marked pending by Xen > >>> but guest does not receive it. I am not sure if the local interrupt > >>> delivery is disabled on the guest because of which it may not be > >>> receiving the interrupts. Is there a way to verify that interrupts are > >>> indeed disabled at that point. Since the interrupt delivery works fine > >>> when I manually read/write characters to /dev/ttyAMA0, I think there > >>> is no issue from the Xen side in terms of raising an interrupt. > >>> > >>> One observation is that pl011_request_port () calls request_mem_region > >>> () for the MMIO space allocated to pl011. The function returns -EBUSY > >>> indicating that the memory region is already in use. But when I do cat > >>> /proc/iomem, I do not see any overlaps. I am using 0x2200 as the > >>> start of the pl011 MMIO space. I see the same error while booting the > >>> dom0 kernel also. > >>> > >>> I have attached the guest boot logs when console=ttyAMA0. I have added > >>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c > >>> and serial_core.c. The output from the console seems to suddenly stop > >>> once the pl011 port is started. I have not added getty for ttyAMA0 in > >>> the /etc/inittab file. I believe auto-serial-console should be able to > >>> attach a getty session to ttyAMA0 automatically (i did try with adding > >>> a getty command in the inittab file but that did not solve the issue). > >>> > >>> Towards the end of the log, we can see that every time a character is > >>> typed, Xen raises an interrupt and the characters keep getting > >>> collected in the IN RING buffer (in_prod keeps incrementing). Later > >>> interrupts remain in inflight as the earlier interrupt is not cleared. > >> > >> Let me get this straight: who is printing all those messages we see on > >> the guest console (xl console guest) up until "Freeing unused kernel > >> memory"? Is it the early pl011 driver? Or is it the Xen PV console > >> driver? > >> > >> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to > >> issue HYPERVISOR_console_io hypercalls. You might want to check that > >> do_console_io in Xen is letting DomU calls through. > >> > >> Otherwise pl011_int might actually be called but the printk might not > >> show up on the Xen console. > >> > >> The interrupt is enabled from Xen point of view, otherwise it would not > >> be added to the LR. However Linux might still refuse to call the > >> interrupt handler for some reason. You might want to add a printk in the > >> Linux generic interrupt handler, probably > >> kernel/irq/chip.c:handle_fasteoi_irq in your case. > > > > The pl011 console is working now. I did not fix any thing specific > > except I cleaned up the debug code which I had added. I am not sure if > > some of my debug code was causing the issue. > > > > After cleaning up the debug code and building the code afresh, I > > started receiving the interrupts and input/output on the console is > > working fine. Now I am able to execute commands on the console. > > > I believe since I was using printk in the pl011 console driver code > for debugging, that might be causing some infinite loop. Once I > replaced all printks with H
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, On 15 February 2017 at 13:45, Bhupinder Thakur wrote: > Hi Stefano, > > On 14 February 2017 at 03:14, Stefano Stabellini > wrote: >> On Mon, 13 Feb 2017, Bhupinder Thakur wrote: >>> Hi Stefano, >>> >>> On 9 February 2017 at 05:40, Stefano Stabellini >>> wrote: >>> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: >>> >> Hi Julien, >>> >> >>> >> On 3 February 2017 at 19:38, Julien Grall wrote: >>> >> > So if I understand correctly, you don't receive anymore output. >>> >> > Correct? >>> >> > Have you tried to see whether the pl011 driver is receiving interrupt >>> >> > or >>> >> > even Linux calling it. >>> >> >>> >> The pl011 driver is not receiving the interrupts (I put a >>> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not >>> >> see the print on the xen console) . When the console is in this state, >>> >> if I keep typing the characters on the console then Xen keeps raising >>> >> the interrupts (for each character typed) but all those interrupts are >>> >> marked as inflight. So it seems that interrupt is not getting cleared >>> >> in GIC. Which GIC function can I look at to further debug what is the >>> >> issue? >>> >> >>> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then >>> >> I used to receive the interrupts all fine. >>> > >>> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't >>> > mark the interrupt as a real hardware interrupt by mistake >>> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 >>> > interrupt has been added correctly to the LRs >>> > (xen/arch/arm/gic-v2.c:gicv2_update_lr). >>> >>> I verified that the PL011 interrupt is getting marked pending by Xen >>> but guest does not receive it. I am not sure if the local interrupt >>> delivery is disabled on the guest because of which it may not be >>> receiving the interrupts. Is there a way to verify that interrupts are >>> indeed disabled at that point. Since the interrupt delivery works fine >>> when I manually read/write characters to /dev/ttyAMA0, I think there >>> is no issue from the Xen side in terms of raising an interrupt. >>> >>> One observation is that pl011_request_port () calls request_mem_region >>> () for the MMIO space allocated to pl011. The function returns -EBUSY >>> indicating that the memory region is already in use. But when I do cat >>> /proc/iomem, I do not see any overlaps. I am using 0x2200 as the >>> start of the pl011 MMIO space. I see the same error while booting the >>> dom0 kernel also. >>> >>> I have attached the guest boot logs when console=ttyAMA0. I have added >>> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c >>> and serial_core.c. The output from the console seems to suddenly stop >>> once the pl011 port is started. I have not added getty for ttyAMA0 in >>> the /etc/inittab file. I believe auto-serial-console should be able to >>> attach a getty session to ttyAMA0 automatically (i did try with adding >>> a getty command in the inittab file but that did not solve the issue). >>> >>> Towards the end of the log, we can see that every time a character is >>> typed, Xen raises an interrupt and the characters keep getting >>> collected in the IN RING buffer (in_prod keeps incrementing). Later >>> interrupts remain in inflight as the earlier interrupt is not cleared. >> >> Let me get this straight: who is printing all those messages we see on >> the guest console (xl console guest) up until "Freeing unused kernel >> memory"? Is it the early pl011 driver? Or is it the Xen PV console >> driver? >> >> Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to >> issue HYPERVISOR_console_io hypercalls. You might want to check that >> do_console_io in Xen is letting DomU calls through. >> >> Otherwise pl011_int might actually be called but the printk might not >> show up on the Xen console. >> >> The interrupt is enabled from Xen point of view, otherwise it would not >> be added to the LR. However Linux might still refuse to call the >> interrupt handler for some reason. You might want to add a printk in the >> Linux generic interrupt handler, probably >> kernel/irq/chip.c:handle_fasteoi_irq in your case. > > The pl011 console is working now. I did not fix any thing specific > except I cleaned up the debug code which I had added. I am not sure if > some of my debug code was causing the issue. > > After cleaning up the debug code and building the code afresh, I > started receiving the interrupts and input/output on the console is > working fine. Now I am able to execute commands on the console. > I believe since I was using printk in the pl011 console driver code for debugging, that might be causing some infinite loop. Once I replaced all printks with HYPERVISOR_console_io(), it started working. Later I compiled out the debug code, and it was working fine. Regards, Bhupinder ___ Xen-devel mailing list Xen-devel@lists.xen.org https://
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 14 February 2017 at 03:14, Stefano Stabellini wrote: > On Mon, 13 Feb 2017, Bhupinder Thakur wrote: >> Hi Stefano, >> >> On 9 February 2017 at 05:40, Stefano Stabellini >> wrote: >> > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: >> >> Hi Julien, >> >> >> >> On 3 February 2017 at 19:38, Julien Grall wrote: >> >> > So if I understand correctly, you don't receive anymore output. Correct? >> >> > Have you tried to see whether the pl011 driver is receiving interrupt or >> >> > even Linux calling it. >> >> >> >> The pl011 driver is not receiving the interrupts (I put a >> >> HYPERVISOR_console_io() call inside the interrupt handler and I do not >> >> see the print on the xen console) . When the console is in this state, >> >> if I keep typing the characters on the console then Xen keeps raising >> >> the interrupts (for each character typed) but all those interrupts are >> >> marked as inflight. So it seems that interrupt is not getting cleared >> >> in GIC. Which GIC function can I look at to further debug what is the >> >> issue? >> >> >> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then >> >> I used to receive the interrupts all fine. >> > >> > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't >> > mark the interrupt as a real hardware interrupt by mistake >> > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 >> > interrupt has been added correctly to the LRs >> > (xen/arch/arm/gic-v2.c:gicv2_update_lr). >> >> I verified that the PL011 interrupt is getting marked pending by Xen >> but guest does not receive it. I am not sure if the local interrupt >> delivery is disabled on the guest because of which it may not be >> receiving the interrupts. Is there a way to verify that interrupts are >> indeed disabled at that point. Since the interrupt delivery works fine >> when I manually read/write characters to /dev/ttyAMA0, I think there >> is no issue from the Xen side in terms of raising an interrupt. >> >> One observation is that pl011_request_port () calls request_mem_region >> () for the MMIO space allocated to pl011. The function returns -EBUSY >> indicating that the memory region is already in use. But when I do cat >> /proc/iomem, I do not see any overlaps. I am using 0x2200 as the >> start of the pl011 MMIO space. I see the same error while booting the >> dom0 kernel also. >> >> I have attached the guest boot logs when console=ttyAMA0. I have added >> debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c >> and serial_core.c. The output from the console seems to suddenly stop >> once the pl011 port is started. I have not added getty for ttyAMA0 in >> the /etc/inittab file. I believe auto-serial-console should be able to >> attach a getty session to ttyAMA0 automatically (i did try with adding >> a getty command in the inittab file but that did not solve the issue). >> >> Towards the end of the log, we can see that every time a character is >> typed, Xen raises an interrupt and the characters keep getting >> collected in the IN RING buffer (in_prod keeps incrementing). Later >> interrupts remain in inflight as the earlier interrupt is not cleared. > > Let me get this straight: who is printing all those messages we see on > the guest console (xl console guest) up until "Freeing unused kernel > memory"? Is it the early pl011 driver? Or is it the Xen PV console > driver? > > Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to > issue HYPERVISOR_console_io hypercalls. You might want to check that > do_console_io in Xen is letting DomU calls through. > > Otherwise pl011_int might actually be called but the printk might not > show up on the Xen console. > > The interrupt is enabled from Xen point of view, otherwise it would not > be added to the LR. However Linux might still refuse to call the > interrupt handler for some reason. You might want to add a printk in the > Linux generic interrupt handler, probably > kernel/irq/chip.c:handle_fasteoi_irq in your case. The pl011 console is working now. I did not fix any thing specific except I cleaned up the debug code which I had added. I am not sure if some of my debug code was causing the issue. After cleaning up the debug code and building the code afresh, I started receiving the interrupts and input/output on the console is working fine. Now I am able to execute commands on the console. Regards, Bhupinder ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Mon, 13 Feb 2017, Bhupinder Thakur wrote: > Hi Stefano, > > On 9 February 2017 at 05:40, Stefano Stabellini > wrote: > > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: > >> Hi Julien, > >> > >> On 3 February 2017 at 19:38, Julien Grall wrote: > >> > So if I understand correctly, you don't receive anymore output. Correct? > >> > Have you tried to see whether the pl011 driver is receiving interrupt or > >> > even Linux calling it. > >> > >> The pl011 driver is not receiving the interrupts (I put a > >> HYPERVISOR_console_io() call inside the interrupt handler and I do not > >> see the print on the xen console) . When the console is in this state, > >> if I keep typing the characters on the console then Xen keeps raising > >> the interrupts (for each character typed) but all those interrupts are > >> marked as inflight. So it seems that interrupt is not getting cleared > >> in GIC. Which GIC function can I look at to further debug what is the > >> issue? > >> > >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then > >> I used to receive the interrupts all fine. > > > > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't > > mark the interrupt as a real hardware interrupt by mistake > > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 > > interrupt has been added correctly to the LRs > > (xen/arch/arm/gic-v2.c:gicv2_update_lr). > > I verified that the PL011 interrupt is getting marked pending by Xen > but guest does not receive it. I am not sure if the local interrupt > delivery is disabled on the guest because of which it may not be > receiving the interrupts. Is there a way to verify that interrupts are > indeed disabled at that point. Since the interrupt delivery works fine > when I manually read/write characters to /dev/ttyAMA0, I think there > is no issue from the Xen side in terms of raising an interrupt. > > One observation is that pl011_request_port () calls request_mem_region > () for the MMIO space allocated to pl011. The function returns -EBUSY > indicating that the memory region is already in use. But when I do cat > /proc/iomem, I do not see any overlaps. I am using 0x2200 as the > start of the pl011 MMIO space. I see the same error while booting the > dom0 kernel also. > > I have attached the guest boot logs when console=ttyAMA0. I have added > debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c > and serial_core.c. The output from the console seems to suddenly stop > once the pl011 port is started. I have not added getty for ttyAMA0 in > the /etc/inittab file. I believe auto-serial-console should be able to > attach a getty session to ttyAMA0 automatically (i did try with adding > a getty command in the inittab file but that did not solve the issue). > > Towards the end of the log, we can see that every time a character is > typed, Xen raises an interrupt and the characters keep getting > collected in the IN RING buffer (in_prod keeps incrementing). Later > interrupts remain in inflight as the earlier interrupt is not cleared. Let me get this straight: who is printing all those messages we see on the guest console (xl console guest) up until "Freeing unused kernel memory"? Is it the early pl011 driver? Or is it the Xen PV console driver? Are you using a DEBUG build of Xen? Only DEBUG builds allow DomUs to issue HYPERVISOR_console_io hypercalls. You might want to check that do_console_io in Xen is letting DomU calls through. Otherwise pl011_int might actually be called but the printk might not show up on the Xen console. The interrupt is enabled from Xen point of view, otherwise it would not be added to the LR. However Linux might still refuse to call the interrupt handler for some reason. You might want to add a printk in the Linux generic interrupt handler, probably kernel/irq/chip.c:handle_fasteoi_irq in your case. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 9 February 2017 at 05:40, Stefano Stabellini wrote: > On Wed, 8 Feb 2017, Bhupinder Thakur wrote: >> Hi Julien, >> >> On 3 February 2017 at 19:38, Julien Grall wrote: >> > So if I understand correctly, you don't receive anymore output. Correct? >> > Have you tried to see whether the pl011 driver is receiving interrupt or >> > even Linux calling it. >> >> The pl011 driver is not receiving the interrupts (I put a >> HYPERVISOR_console_io() call inside the interrupt handler and I do not >> see the print on the xen console) . When the console is in this state, >> if I keep typing the characters on the console then Xen keeps raising >> the interrupts (for each character typed) but all those interrupts are >> marked as inflight. So it seems that interrupt is not getting cleared >> in GIC. Which GIC function can I look at to further debug what is the >> issue? >> >> Earlier, when I used to manually write characters to /dev/ttyAMA0 then >> I used to receive the interrupts all fine. > > DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't > mark the interrupt as a real hardware interrupt by mistake > (GICH_V2_LR_HW). At the same time, check that the virtual PL011 > interrupt has been added correctly to the LRs > (xen/arch/arm/gic-v2.c:gicv2_update_lr). I verified that the PL011 interrupt is getting marked pending by Xen but guest does not receive it. I am not sure if the local interrupt delivery is disabled on the guest because of which it may not be receiving the interrupts. Is there a way to verify that interrupts are indeed disabled at that point. Since the interrupt delivery works fine when I manually read/write characters to /dev/ttyAMA0, I think there is no issue from the Xen side in terms of raising an interrupt. One observation is that pl011_request_port () calls request_mem_region () for the MMIO space allocated to pl011. The function returns -EBUSY indicating that the memory region is already in use. But when I do cat /proc/iomem, I do not see any overlaps. I am using 0x2200 as the start of the pl011 MMIO space. I see the same error while booting the dom0 kernel also. I have attached the guest boot logs when console=ttyAMA0. I have added debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c and serial_core.c. The output from the console seems to suddenly stop once the pl011 port is started. I have not added getty for ttyAMA0 in the /etc/inittab file. I believe auto-serial-console should be able to attach a getty session to ttyAMA0 automatically (i did try with adding a getty command in the inittab file but that did not solve the issue). Towards the end of the log, we can see that every time a character is typed, Xen raises an interrupt and the characters keep getting collected in the IN RING buffer (in_prod keeps incrementing). Later interrupts remain in inflight as the earlier interrupt is not cleared. Regards, Bhupinder (none):/root # cat domU.cfg kernel = "/root/Image" name = "guest" memory = 512 vcpus = 1 extra = "console=ttyAMA0 root=/dev/xvda ro" disk = [ 'phy:/dev/loop0,xvda,w' ] (none):/root # xl create domU.cfg Parsing config from domU.cfg (XEN) allocated xen_consumer = 1 for port = 1, remote dom=0 (XEN) searching virq in < 32, 33 > (XEN) HVM_PARAM_VPL011_VIRQ param get val=32 (XEN) HVM_PARAM_VPL011_CONSOLE_PFN = 0x39003 (XEN) HVM_PARAM_VPL011_CONSOLE_PFN param set val=0x39003 seq 1204 queued, 'add' 'xen-backend' Validate module index Check if link configuration needs reloading. seq 1204 forked new worker [1556] seq 1204 running IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15 IMPORT builtin 'hwdb' returned non-zero RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5 Execute 'load' 'xen-backend:vbd' passed device to netlink monitor 0xb38dad90 seq 1204 processed cleanup idle workers Unload module index Unloaded link configuration context. worker [1556] exited seq 1205 queued, 'add' 'xen-backend' seq 1205 forked new worker [1614] seq 1205 running IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15 IMPORT builtin 'hwdb' returned non-zero RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5 Execute 'load' 'xen-backend:console' passed device to netlink monitor 0xb38dad90 seq 1205 processed cleanup idle workers Unload module index Unloaded link configuration context. worker [1614] exited (none):/root # xl console guest [0.00] Booting Linux on physical CPU 0x0 [0.00] Linux version 4.9.0-rc3-00243-g0c183d9-dirty (btuser@blr-ubuntu-linaro) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #94 SMP PREEMPT Mon Feb 13 12:12:48 IST 2017 [0.00] Boot CPU: AArch64 Processor [410fd000] [0.00] Xen 4.9 support found [0.00] efi: Getting EFI parameters from FDT: [0.00] efi: UEFI not found. [0.00] cma: Reserved 16 MiB at 0x5f00 [0.00] psci: probing for conduit method from DT. [
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, 8 Feb 2017, Bhupinder Thakur wrote: > Hi Julien, > > On 3 February 2017 at 19:38, Julien Grall wrote: > > So if I understand correctly, you don't receive anymore output. Correct? > > Have you tried to see whether the pl011 driver is receiving interrupt or > > even Linux calling it. > > The pl011 driver is not receiving the interrupts (I put a > HYPERVISOR_console_io() call inside the interrupt handler and I do not > see the print on the xen console) . When the console is in this state, > if I keep typing the characters on the console then Xen keeps raising > the interrupts (for each character typed) but all those interrupts are > marked as inflight. So it seems that interrupt is not getting cleared > in GIC. Which GIC function can I look at to further debug what is the > issue? > > Earlier, when I used to manually write characters to /dev/ttyAMA0 then > I used to receive the interrupts all fine. DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't mark the interrupt as a real hardware interrupt by mistake (GICH_V2_LR_HW). At the same time, check that the virtual PL011 interrupt has been added correctly to the LRs (xen/arch/arm/gic-v2.c:gicv2_update_lr). ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Julien, On 3 February 2017 at 19:38, Julien Grall wrote: > So if I understand correctly, you don't receive anymore output. Correct? > Have you tried to see whether the pl011 driver is receiving interrupt or > even Linux calling it. The pl011 driver is not receiving the interrupts (I put a HYPERVISOR_console_io() call inside the interrupt handler and I do not see the print on the xen console) . When the console is in this state, if I keep typing the characters on the console then Xen keeps raising the interrupts (for each character typed) but all those interrupts are marked as inflight. So it seems that interrupt is not getting cleared in GIC. Which GIC function can I look at to further debug what is the issue? Earlier, when I used to manually write characters to /dev/ttyAMA0 then I used to receive the interrupts all fine. Regards, Bhupinder ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Fri, Feb 3, 2017 at 2:53 PM, Bhupinder Thakur wrote: > Hi, > >>> Hi, >>> >>> I have done the changes for emulating pl011 in Xen. Currently, I have >>> verified the emulation code by manually reading/writing data to >>> /dev/ttyAMA0 which is the device file for pl011 device. The data is >>> flowing fine between xenconsoled and the guest domain. >>> >>> As a next step, I wanted to use /dev/ttyAMA0 as a console. >>> >>> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the >>> "extra" directive in the domU configuration file. However, I do not >>> see the output on the console once I attached the console using "xl >>> console ". I tried using "xl console -t serial >>> " also but that shows the tty1 console and not the >>> ttyAMA0 one. >> >> I would try to verify that when Linux uses the pl011 as a console (as >> opposed to just writing some characters into /dev/ttyAMA0), that it >> doesn't error out somewhere. So like Stefano suggests, make sure you >> check the guest kernel log after boot (assuming you can SSH or something >> into the box) and look at 'dmesg' to see if the pl011 driver failed. >> >> You could also try adding some log info in you Xen pl011 emulation code >> to see if the guest kernel driver prods the device in the sequence you >> expect, based on looking at the linux driver for the pl011. > > I tested with the following changes: > > 1. Compiled out CONFIG_HVC_XEN > 2. Specified "console=ttyAMA0" in the guest configuration > > With some more changes/fixes, the guest console is working partially > over ttyAMA0. I can see the guest boot prints on the console. However, > the prints stop just towards the end when the init scripts are > executed. > > So it seems that during boot, initially ttyAMA0 is used as a > write-only port, where the console driver keeps writing the output to > the port. Till this point, only tx has been enabled on ttyAMA0 (driver > probe() is called at this point). No irq/rx has been enabled yet. Just > as the init scripts are executed, the irq/rx is also enabled on > ttyAMA0 (driver startup() is called at this time which enables rx and > registers IRQ) and then the boot prints stop. > > Another observation is that unless you compile out HVC_XEN support, > the hvc0 console also keeps receiving the data even if the > console=ttyAMA0. > What's your guest distro and file system settings? This sounds to me like the console bringup job is hard-wired to /dev/hvc0 as opposed to using the console. I think most ubuntu versions prior to using systemd needs the right files in /etc/init. Check if you have /etc/init/hvc.conf or something like that and if you have /etc/init/ttyAMA0.conf. If you're using systemd or a different distro, figure out what the equivalent settings there are. -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On 03/02/17 13:53, Bhupinder Thakur wrote: Hi, Hi Bhupinder, Hi, I have done the changes for emulating pl011 in Xen. Currently, I have verified the emulation code by manually reading/writing data to /dev/ttyAMA0 which is the device file for pl011 device. The data is flowing fine between xenconsoled and the guest domain. As a next step, I wanted to use /dev/ttyAMA0 as a console. For that I tried adding console=ttyAMA0 instead of console=hvc0 in the "extra" directive in the domU configuration file. However, I do not see the output on the console once I attached the console using "xl console ". I tried using "xl console -t serial " also but that shows the tty1 console and not the ttyAMA0 one. I would try to verify that when Linux uses the pl011 as a console (as opposed to just writing some characters into /dev/ttyAMA0), that it doesn't error out somewhere. So like Stefano suggests, make sure you check the guest kernel log after boot (assuming you can SSH or something into the box) and look at 'dmesg' to see if the pl011 driver failed. You could also try adding some log info in you Xen pl011 emulation code to see if the guest kernel driver prods the device in the sequence you expect, based on looking at the linux driver for the pl011. I tested with the following changes: 1. Compiled out CONFIG_HVC_XEN 2. Specified "console=ttyAMA0" in the guest configuration With some more changes/fixes, the guest console is working partially over ttyAMA0. I can see the guest boot prints on the console. However, the prints stop just towards the end when the init scripts are executed. So it seems that during boot, initially ttyAMA0 is used as a write-only port, where the console driver keeps writing the output to the port. Till this point, only tx has been enabled on ttyAMA0 (driver probe() is called at this point). No irq/rx has been enabled yet. Just as the init scripts are executed, the irq/rx is also enabled on ttyAMA0 (driver startup() is called at this time which enables rx and registers IRQ) and then the boot prints stop. So if I understand correctly, you don't receive anymore output. Correct? Have you tried to see whether the pl011 driver is receiving interrupt or even Linux calling it. Assuming you have Xen built with debug enabled (see CONFIG_DEBUG), you could use xen_raw_printk in the pl011 driver to see what's going on. This will issue an hypercall and the message will be directly printed on Xen console. If you compile out the HVC_XEN support, you could directly use HYPERVISOR_console_io(CONSOLEIO_write, len, str). Another observation is that unless you compile out HVC_XEN support, the hvc0 console also keeps receiving the data even if the console=ttyAMA0. HVC is the preferred console when using Xen, I think you would also need to add "earlycon=pl0111,addr" on the kernel commandline where addr is the base address of the PL011 to get Linux using the serial port as boot console. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, >> Hi, >> >> I have done the changes for emulating pl011 in Xen. Currently, I have >> verified the emulation code by manually reading/writing data to >> /dev/ttyAMA0 which is the device file for pl011 device. The data is >> flowing fine between xenconsoled and the guest domain. >> >> As a next step, I wanted to use /dev/ttyAMA0 as a console. >> >> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the >> "extra" directive in the domU configuration file. However, I do not >> see the output on the console once I attached the console using "xl >> console ". I tried using "xl console -t serial >> " also but that shows the tty1 console and not the >> ttyAMA0 one. > > I would try to verify that when Linux uses the pl011 as a console (as > opposed to just writing some characters into /dev/ttyAMA0), that it > doesn't error out somewhere. So like Stefano suggests, make sure you > check the guest kernel log after boot (assuming you can SSH or something > into the box) and look at 'dmesg' to see if the pl011 driver failed. > > You could also try adding some log info in you Xen pl011 emulation code > to see if the guest kernel driver prods the device in the sequence you > expect, based on looking at the linux driver for the pl011. I tested with the following changes: 1. Compiled out CONFIG_HVC_XEN 2. Specified "console=ttyAMA0" in the guest configuration With some more changes/fixes, the guest console is working partially over ttyAMA0. I can see the guest boot prints on the console. However, the prints stop just towards the end when the init scripts are executed. So it seems that during boot, initially ttyAMA0 is used as a write-only port, where the console driver keeps writing the output to the port. Till this point, only tx has been enabled on ttyAMA0 (driver probe() is called at this point). No irq/rx has been enabled yet. Just as the init scripts are executed, the irq/rx is also enabled on ttyAMA0 (driver startup() is called at this time which enables rx and registers IRQ) and then the boot prints stop. Another observation is that unless you compile out HVC_XEN support, the hvc0 console also keeps receiving the data even if the console=ttyAMA0. Regards, Bhupinder ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Fri, Jan 27, 2017 at 05:44:15PM +0530, Bhupinder Thakur wrote: > Hi, > > I have done the changes for emulating pl011 in Xen. Currently, I have > verified the emulation code by manually reading/writing data to > /dev/ttyAMA0 which is the device file for pl011 device. The data is > flowing fine between xenconsoled and the guest domain. > > As a next step, I wanted to use /dev/ttyAMA0 as a console. > > For that I tried adding console=ttyAMA0 instead of console=hvc0 in the > "extra" directive in the domU configuration file. However, I do not > see the output on the console once I attached the console using "xl > console ". I tried using "xl console -t serial > " also but that shows the tty1 console and not the > ttyAMA0 one. I would try to verify that when Linux uses the pl011 as a console (as opposed to just writing some characters into /dev/ttyAMA0), that it doesn't error out somewhere. So like Stefano suggests, make sure you check the guest kernel log after boot (assuming you can SSH or something into the box) and look at 'dmesg' to see if the pl011 driver failed. You could also try adding some log info in you Xen pl011 emulation code to see if the guest kernel driver prods the device in the sequence you expect, based on looking at the linux driver for the pl011. Thanks, -Christoffer > > > On 18 January 2017 at 00:57, Stefano Stabellini > wrote: > > On Tue, 17 Jan 2017, Julien Grall wrote: > >> Hi, > >> > >> Sorry for the late answer, I am just back from holidays and still > >> catching-up > >> with my e-mails. > >> > >> On 03/01/17 20:08, Stefano Stabellini wrote: > >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: > >> > > On 28 December 2016 at 23:19, Julien Grall > >> > > wrote: > >> > > > On 21/12/16 22:12, Stefano Stabellini wrote: > >> > > > > > >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote: > >> > > > > > > >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote: > >> > > > > > > > >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote: > >> > > > > > > > > >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > >> > > > > > > > > > >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall > >> > > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params > >> > > > > > > > > and > >> > > > > > > > > Xen > >> > > > > > > > > needs > >> > > > > > > > > to unmap the pfn from the guest immediately, because we > >> > > > > > > > > don't > >> > > > > > > > > want the > >> > > > > > > > > guest to have access to it. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > If you unmap the pfn, the PV backend will not be able to > >> > > > > > > > request > >> > > > > > > > the > >> > > > > > > > page > >> > > > > > > > because there will be no translation available. > >> > > > > > > > > >> > > > > > > > So what you want to do is preventing the guest to at least > >> > > > > > > > write > >> > > > > > > > into > >> > > > > > > > region > >> > > > > > > > (not sure if it is worth to restrict read) > >> > > > > > > > >> > > > > > > > >> > > > > > > That's a good idea. > >> > > > > > > > >> > > > > > > > >> > > > > > > > and unmap the page via the hypercall > >> > > > > > > > XENMEM_decrease_reservation. > >> > > > > > > > >> > > > > > > > >> > > > > > > That would be issued by the guest itself, right? To save > >> > > > > > > address > >> > > > > > > space? > >> > > > > > > >> > > > > > > >> > > > > > Correct. The main use case today is ballooning, but guest could > >> > > > > > call > >> > > > > > it > >> > > > > > on any > >> > > > > > other RAM baked page. > >> > > > > > > >> > > > > > I was thinking about more about the protection needed. > >> > > > > > Technically > >> > > > > > the > >> > > > > > data in > >> > > > > > the ring are not trusted. So if the guest is messing up with it, > >> > > > > > it > >> > > > > > would > >> > > > > > not > >> > > > > > be a big issue. Or did I miss anything here? > >> > > > > > >> > > > > > >> > > > > I understand that a guest would be smart to call > >> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but > >> > > > > it > >> > > > > cannot be a security measure, because, in fact, it needs to be > >> > > > > called > >> > > > > by > >> > > > > the guest. Of course, a malicious guest can simply not call > >> > > > > XENMEM_decrease_reservation for it. > >> > > > > >> > > > > >> > > > Sorry I was not clear. I was not suggested the guest to call > >> > > > XENMEM_decrease_reservation on ring for security but a malicious > >> > > > guest > >> > > > issuing the hypercall on the ring protected and replacing by another > >> > > > page. > >> > > > > >> > > > This is the exact same problem as the one I mentioned on the ITS > >> > > > thread. > >> > > > The > >> > > > page live in guest memory but contains data that will only be > >> > > >
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Fri, 27 Jan 2017, Bhupinder Thakur wrote: > Hi, > > I have done the changes for emulating pl011 in Xen. Currently, I have > verified the emulation code by manually reading/writing data to > /dev/ttyAMA0 which is the device file for pl011 device. The data is > flowing fine between xenconsoled and the guest domain. Good progress! > As a next step, I wanted to use /dev/ttyAMA0 as a console. Do you see, among the kernel boot messages, something about ttyAMA0 being available? Is there a getty -L ttyAMA0 in your inittab? > For that I tried adding console=ttyAMA0 instead of console=hvc0 in the > "extra" directive in the domU configuration file. However, I do not > see the output on the console once I attached the console using "xl > console ". I tried using "xl console -t serial > " also but that shows the tty1 console and not the > ttyAMA0 one. > > Note that currently for testing, I have patched the code in the > xenconsoled to read/write pl011 ring buffers and write them to the > same pty terminal as used by hvc0. Finally, once I have verified, I > will create a separate pty for pl011 and add a new console type > "pl011". If you modified xenconsoled to use the same pty as hvc0, then you should just do "xl console ", the -t serial option is useless, as there is only one console from xl console point of view. You might want to disable the PV console for these tests. I suggest to add a return at the beginning of drivers/tty/hvc/hvc_xen.c:xen_hvc_init and drivers/tty/hvc/hvc_xen.c:xen_cons_init. > Regards, > Bhupinder > > > On 18 January 2017 at 00:57, Stefano Stabellini > wrote: > > On Tue, 17 Jan 2017, Julien Grall wrote: > >> Hi, > >> > >> Sorry for the late answer, I am just back from holidays and still > >> catching-up > >> with my e-mails. > >> > >> On 03/01/17 20:08, Stefano Stabellini wrote: > >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: > >> > > On 28 December 2016 at 23:19, Julien Grall > >> > > wrote: > >> > > > On 21/12/16 22:12, Stefano Stabellini wrote: > >> > > > > > >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote: > >> > > > > > > >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote: > >> > > > > > > > >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote: > >> > > > > > > > > >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > >> > > > > > > > > > >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall > >> > > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params > >> > > > > > > > > and > >> > > > > > > > > Xen > >> > > > > > > > > needs > >> > > > > > > > > to unmap the pfn from the guest immediately, because we > >> > > > > > > > > don't > >> > > > > > > > > want the > >> > > > > > > > > guest to have access to it. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > If you unmap the pfn, the PV backend will not be able to > >> > > > > > > > request > >> > > > > > > > the > >> > > > > > > > page > >> > > > > > > > because there will be no translation available. > >> > > > > > > > > >> > > > > > > > So what you want to do is preventing the guest to at least > >> > > > > > > > write > >> > > > > > > > into > >> > > > > > > > region > >> > > > > > > > (not sure if it is worth to restrict read) > >> > > > > > > > >> > > > > > > > >> > > > > > > That's a good idea. > >> > > > > > > > >> > > > > > > > >> > > > > > > > and unmap the page via the hypercall > >> > > > > > > > XENMEM_decrease_reservation. > >> > > > > > > > >> > > > > > > > >> > > > > > > That would be issued by the guest itself, right? To save > >> > > > > > > address > >> > > > > > > space? > >> > > > > > > >> > > > > > > >> > > > > > Correct. The main use case today is ballooning, but guest could > >> > > > > > call > >> > > > > > it > >> > > > > > on any > >> > > > > > other RAM baked page. > >> > > > > > > >> > > > > > I was thinking about more about the protection needed. > >> > > > > > Technically > >> > > > > > the > >> > > > > > data in > >> > > > > > the ring are not trusted. So if the guest is messing up with it, > >> > > > > > it > >> > > > > > would > >> > > > > > not > >> > > > > > be a big issue. Or did I miss anything here? > >> > > > > > >> > > > > > >> > > > > I understand that a guest would be smart to call > >> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but > >> > > > > it > >> > > > > cannot be a security measure, because, in fact, it needs to be > >> > > > > called > >> > > > > by > >> > > > > the guest. Of course, a malicious guest can simply not call > >> > > > > XENMEM_decrease_reservation for it. > >> > > > > >> > > > > >> > > > Sorry I was not clear. I was not suggested the guest to call > >> > > > XENMEM_decrease_reservation on ring for security but a malicious > >> > > > guest > >> > > > issuing the hypercall on the ring protected and replacin
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, I have done the changes for emulating pl011 in Xen. Currently, I have verified the emulation code by manually reading/writing data to /dev/ttyAMA0 which is the device file for pl011 device. The data is flowing fine between xenconsoled and the guest domain. As a next step, I wanted to use /dev/ttyAMA0 as a console. For that I tried adding console=ttyAMA0 instead of console=hvc0 in the "extra" directive in the domU configuration file. However, I do not see the output on the console once I attached the console using "xl console ". I tried using "xl console -t serial " also but that shows the tty1 console and not the ttyAMA0 one. Note that currently for testing, I have patched the code in the xenconsoled to read/write pl011 ring buffers and write them to the same pty terminal as used by hvc0. Finally, once I have verified, I will create a separate pty for pl011 and add a new console type "pl011". Regards, Bhupinder On 18 January 2017 at 00:57, Stefano Stabellini wrote: > On Tue, 17 Jan 2017, Julien Grall wrote: >> Hi, >> >> Sorry for the late answer, I am just back from holidays and still catching-up >> with my e-mails. >> >> On 03/01/17 20:08, Stefano Stabellini wrote: >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: >> > > On 28 December 2016 at 23:19, Julien Grall wrote: >> > > > On 21/12/16 22:12, Stefano Stabellini wrote: >> > > > > >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote: >> > > > > > >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote: >> > > > > > > >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote: >> > > > > > > > >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: >> > > > > > > > > >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: >> > > > > > > > > > >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall >> > > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params and >> > > > > > > > > Xen >> > > > > > > > > needs >> > > > > > > > > to unmap the pfn from the guest immediately, because we don't >> > > > > > > > > want the >> > > > > > > > > guest to have access to it. >> > > > > > > > >> > > > > > > > >> > > > > > > > If you unmap the pfn, the PV backend will not be able to >> > > > > > > > request >> > > > > > > > the >> > > > > > > > page >> > > > > > > > because there will be no translation available. >> > > > > > > > >> > > > > > > > So what you want to do is preventing the guest to at least >> > > > > > > > write >> > > > > > > > into >> > > > > > > > region >> > > > > > > > (not sure if it is worth to restrict read) >> > > > > > > >> > > > > > > >> > > > > > > That's a good idea. >> > > > > > > >> > > > > > > >> > > > > > > > and unmap the page via the hypercall >> > > > > > > > XENMEM_decrease_reservation. >> > > > > > > >> > > > > > > >> > > > > > > That would be issued by the guest itself, right? To save address >> > > > > > > space? >> > > > > > >> > > > > > >> > > > > > Correct. The main use case today is ballooning, but guest could >> > > > > > call >> > > > > > it >> > > > > > on any >> > > > > > other RAM baked page. >> > > > > > >> > > > > > I was thinking about more about the protection needed. Technically >> > > > > > the >> > > > > > data in >> > > > > > the ring are not trusted. So if the guest is messing up with it, it >> > > > > > would >> > > > > > not >> > > > > > be a big issue. Or did I miss anything here? >> > > > > >> > > > > >> > > > > I understand that a guest would be smart to call >> > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it >> > > > > cannot be a security measure, because, in fact, it needs to be called >> > > > > by >> > > > > the guest. Of course, a malicious guest can simply not call >> > > > > XENMEM_decrease_reservation for it. >> > > > >> > > > >> > > > Sorry I was not clear. I was not suggested the guest to call >> > > > XENMEM_decrease_reservation on ring for security but a malicious guest >> > > > issuing the hypercall on the ring protected and replacing by another >> > > > page. >> > > > >> > > > This is the exact same problem as the one I mentioned on the ITS >> > > > thread. >> > > > The >> > > > page live in guest memory but contains data that will only be touched >> > > > by >> > > > Xen. >> > > > >> > > > If you remove those page from stage-2, the translation IPA -> MFN will >> > > > be >> > > > lost unless you store somewhere else. You would have to do it per-page >> > > > as >> > > > the buffer will use contiguous IPA but potentially noncontiguous MFN. >> > > > >> > > > In the case of ITS the memory is provisioned by the guest. So there are >> > > > not >> > > > much to do there except adding protection in stage-2 such as write >> > > > protection and preventing the guest to unmap it. However for the pl011 >> > > > ring, >> > > > as Andrew pointed on IRC, what we need to do is accounting this page to >> > > > the >> > > > domain memory. No mapping is necessary in s
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 17 Jan 2017, Julien Grall wrote: > Hi, > > Sorry for the late answer, I am just back from holidays and still catching-up > with my e-mails. > > On 03/01/17 20:08, Stefano Stabellini wrote: > > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: > > > On 28 December 2016 at 23:19, Julien Grall wrote: > > > > On 21/12/16 22:12, Stefano Stabellini wrote: > > > > > > > > > > On Wed, 21 Dec 2016, Julien Grall wrote: > > > > > > > > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote: > > > > > > > > > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote: > > > > > > > > > > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > > > > > > > > > > > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > If we use hvm_params for this, we need two new hvm_params and > > > > > > > > > Xen > > > > > > > > > needs > > > > > > > > > to unmap the pfn from the guest immediately, because we don't > > > > > > > > > want the > > > > > > > > > guest to have access to it. > > > > > > > > > > > > > > > > > > > > > > > > If you unmap the pfn, the PV backend will not be able to request > > > > > > > > the > > > > > > > > page > > > > > > > > because there will be no translation available. > > > > > > > > > > > > > > > > So what you want to do is preventing the guest to at least write > > > > > > > > into > > > > > > > > region > > > > > > > > (not sure if it is worth to restrict read) > > > > > > > > > > > > > > > > > > > > > That's a good idea. > > > > > > > > > > > > > > > > > > > > > > and unmap the page via the hypercall > > > > > > > > XENMEM_decrease_reservation. > > > > > > > > > > > > > > > > > > > > > That would be issued by the guest itself, right? To save address > > > > > > > space? > > > > > > > > > > > > > > > > > > Correct. The main use case today is ballooning, but guest could call > > > > > > it > > > > > > on any > > > > > > other RAM baked page. > > > > > > > > > > > > I was thinking about more about the protection needed. Technically > > > > > > the > > > > > > data in > > > > > > the ring are not trusted. So if the guest is messing up with it, it > > > > > > would > > > > > > not > > > > > > be a big issue. Or did I miss anything here? > > > > > > > > > > > > > > > I understand that a guest would be smart to call > > > > > XENMEM_decrease_reservation on the PV console page for pl011, but it > > > > > cannot be a security measure, because, in fact, it needs to be called > > > > > by > > > > > the guest. Of course, a malicious guest can simply not call > > > > > XENMEM_decrease_reservation for it. > > > > > > > > > > > > Sorry I was not clear. I was not suggested the guest to call > > > > XENMEM_decrease_reservation on ring for security but a malicious guest > > > > issuing the hypercall on the ring protected and replacing by another > > > > page. > > > > > > > > This is the exact same problem as the one I mentioned on the ITS thread. > > > > The > > > > page live in guest memory but contains data that will only be touched by > > > > Xen. > > > > > > > > If you remove those page from stage-2, the translation IPA -> MFN will > > > > be > > > > lost unless you store somewhere else. You would have to do it per-page > > > > as > > > > the buffer will use contiguous IPA but potentially noncontiguous MFN. > > > > > > > > In the case of ITS the memory is provisioned by the guest. So there are > > > > not > > > > much to do there except adding protection in stage-2 such as write > > > > protection and preventing the guest to unmap it. However for the pl011 > > > > ring, > > > > as Andrew pointed on IRC, what we need to do is accounting this page to > > > > the > > > > domain memory. No mapping is necessary in stage-2. > > > > > > Please clarify what is meant by that no stage-2 mapping is required. > > > Does it mean that no stage-2 mapping is required for the guest as it > > > never needs to access this page? > > > > That's right. > > > > > > > However, the Xen HYP will need the stage-2 mapping to find out the > > > pl011 PFN --> physical MFN mapping so that it can map the page to its > > > own address space. Currently, I am using prepare_ring_for_helper () to > > > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP > > > VA. > > > > I am not sure what Julien had in mind exactly. I like the idea of not > > mapping the page at stage-2, but it is true that many interfaces expect > > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen > > knows the mfn and could use it to map the page, instead of the pfn. > > However, the PV console backend also needs to map the same page, and it > > currently does that by calling xc_map_foreign_range, which I believe > > also expect a pfn. > > Do you agree that page such as ioreq and the pl011 PV console are only used > for communication between X
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, On 06/01/17 21:54, Stefano Stabellini wrote: On Fri, 6 Jan 2017, Bhupinder Thakur wrote: Hi, I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for a guest domain. I tried allocating a new SPI VIRQ by calling vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests is limited to 0 (vgic_num_irqs() return 32 and therefore in effect says that no SPI VIRQ can be allocated because below 32 are for PPI/SGI VIRQs). Yes, you should use vgic_allocate_virq. For that to work, you need to make sure that nr_spis for the domain is > 0. Given that nr_spis is passed from libxl by calling xc_domain_create, I would set nr_spis to 1 in libxl__arch_domain_prepare_config. Before I forgot, this would need to be documented to allow people to easily use interrupt routing (for device passthrough) when PL011 enabled. I would suggest to define the SPI used by the PL011 in include/public/arch-arm.h and the number of SPIs reserved (GUEST_NR_RESERVED_SPIS). This would avoid an hardcoded number in the code and easily extension whether we need more SPIs later on (I have the host bridge emulation in mind). Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, Sorry for the late answer, I am just back from holidays and still catching-up with my e-mails. On 03/01/17 20:08, Stefano Stabellini wrote: On Thu, 29 Dec 2016, Bhupinder Thakur wrote: On 28 December 2016 at 23:19, Julien Grall wrote: On 21/12/16 22:12, Stefano Stabellini wrote: On Wed, 21 Dec 2016, Julien Grall wrote: On 20/12/2016 20:53, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Julien Grall wrote: On 19/12/2016 21:24, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: If we use hvm_params for this, we need two new hvm_params and Xen needs to unmap the pfn from the guest immediately, because we don't want the guest to have access to it. If you unmap the pfn, the PV backend will not be able to request the page because there will be no translation available. So what you want to do is preventing the guest to at least write into region (not sure if it is worth to restrict read) That's a good idea. and unmap the page via the hypercall XENMEM_decrease_reservation. That would be issued by the guest itself, right? To save address space? Correct. The main use case today is ballooning, but guest could call it on any other RAM baked page. I was thinking about more about the protection needed. Technically the data in the ring are not trusted. So if the guest is messing up with it, it would not be a big issue. Or did I miss anything here? I understand that a guest would be smart to call XENMEM_decrease_reservation on the PV console page for pl011, but it cannot be a security measure, because, in fact, it needs to be called by the guest. Of course, a malicious guest can simply not call XENMEM_decrease_reservation for it. Sorry I was not clear. I was not suggested the guest to call XENMEM_decrease_reservation on ring for security but a malicious guest issuing the hypercall on the ring protected and replacing by another page. This is the exact same problem as the one I mentioned on the ITS thread. The page live in guest memory but contains data that will only be touched by Xen. If you remove those page from stage-2, the translation IPA -> MFN will be lost unless you store somewhere else. You would have to do it per-page as the buffer will use contiguous IPA but potentially noncontiguous MFN. In the case of ITS the memory is provisioned by the guest. So there are not much to do there except adding protection in stage-2 such as write protection and preventing the guest to unmap it. However for the pl011 ring, as Andrew pointed on IRC, what we need to do is accounting this page to the domain memory. No mapping is necessary in stage-2. Please clarify what is meant by that no stage-2 mapping is required. Does it mean that no stage-2 mapping is required for the guest as it never needs to access this page? That's right. However, the Xen HYP will need the stage-2 mapping to find out the pl011 PFN --> physical MFN mapping so that it can map the page to its own address space. Currently, I am using prepare_ring_for_helper () to map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP VA. I am not sure what Julien had in mind exactly. I like the idea of not mapping the page at stage-2, but it is true that many interfaces expect pfns. If Xen is the one to allocate the pl011 PV console page, then Xen knows the mfn and could use it to map the page, instead of the pfn. However, the PV console backend also needs to map the same page, and it currently does that by calling xc_map_foreign_range, which I believe also expect a pfn. Do you agree that page such as ioreq and the pl011 PV console are only used for communication between Xen and a backend? We don't want the guest to access the content of the pages but still be able to map. I would guess that using the gfn was a convenience but has some security impact (see guess using the gfn is mostly for convenience but has some security impact today (see XSA-197 and XSA-199). AFAIU, we also want to account those pages to a domain. For the credit, this is a suggestion made by Andrew on IRC. If I remember correctly what he suggested, it would be to have an add-to-physmap version restricted for non-domU components (e.g backend, toolstack) which would allow to map a given page (maybe via an index?). So maybe it is easier to use write-protection in stage-2 (as for ITS), unless Julien has a better idea? We need write-protection and also prevent the page to be removed. But to be fair, I would not bother to try to write-protect it for now if we are going towards keeping the pl011 console in the guest memory. I don't think letting the guest writing in the console page is a big problem. The backend console should already be protected against malicious guest. If not, we are already in trouble as it part of the PV protocol :). Regarding Xen, the console protocol is very easy and I think only bound checking is suffic
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 10 Jan 2017, Bhupinder Thakur wrote: > Hi Stefano, > > > > On 7 January 2017 at 03:24, Stefano Stabellini wrote: > > Hello Bhupinder, > > > > it is good to hear that you are making progress. > > > > > > On Fri, 6 Jan 2017, Bhupinder Thakur wrote: > >> Hi, > >> > >> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to > >> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for > >> a guest domain. I tried allocating a new SPI VIRQ by calling > >> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests > >> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect > >> says that no SPI VIRQ can be allocated because below 32 are for > >> PPI/SGI VIRQs). > > > > Yes, you should use vgic_allocate_virq. For that to work, you need to > > make sure that nr_spis for the domain is > 0. Given that nr_spis is > > passed from libxl by calling xc_domain_create, I would set nr_spis to 1 > > in libxl__arch_domain_prepare_config. > > > I increment the nr_spis by 1 in domain_vgic_init() to ensure the > vgic_allocate_virq() is able to allocate a SPI for pl011. With that > change, I am able to inject a SPI irq into guest and handle it. Great! ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 7 January 2017 at 03:24, Stefano Stabellini wrote: > Hello Bhupinder, > > it is good to hear that you are making progress. > > > On Fri, 6 Jan 2017, Bhupinder Thakur wrote: >> Hi, >> >> I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to >> the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for >> a guest domain. I tried allocating a new SPI VIRQ by calling >> vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests >> is limited to 0 (vgic_num_irqs() return 32 and therefore in effect >> says that no SPI VIRQ can be allocated because below 32 are for >> PPI/SGI VIRQs). > > Yes, you should use vgic_allocate_virq. For that to work, you need to > make sure that nr_spis for the domain is > 0. Given that nr_spis is > passed from libxl by calling xc_domain_create, I would set nr_spis to 1 > in libxl__arch_domain_prepare_config. > I increment the nr_spis by 1 in domain_vgic_init() to ensure the vgic_allocate_virq() is able to allocate a SPI for pl011. With that change, I am able to inject a SPI irq into guest and handle it. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hello Bhupinder, it is good to hear that you are making progress. On Fri, 6 Jan 2017, Bhupinder Thakur wrote: > Hi, > > I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to > the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for > a guest domain. I tried allocating a new SPI VIRQ by calling > vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests > is limited to 0 (vgic_num_irqs() return 32 and therefore in effect > says that no SPI VIRQ can be allocated because below 32 are for > PPI/SGI VIRQs). Yes, you should use vgic_allocate_virq. For that to work, you need to make sure that nr_spis for the domain is > 0. Given that nr_spis is passed from libxl by calling xc_domain_create, I would set nr_spis to 1 in libxl__arch_domain_prepare_config. > I tried modifying the guest domU configuration to specify the SPI IRQs > to be allowed for the guest but that operation fails as it is not > allowed for the unprivileged guests. Is there a way to allow a SPI > VIRQ to be allocated to a guest OS. I could not find any instance in > the Xen where it is allocating a SPI VIRQ. If you have already tried what I suggested, could you please post the error you are getting? > I was able to register a new PPI VIRQ for pl011 interrupt and could > inject that interrupt into the guest and handle the interrupt in the > guest. > > Regards, > Bhupinder > > On 4 January 2017 at 01:38, Stefano Stabellini wrote: > > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: > >> On 28 December 2016 at 23:19, Julien Grall wrote: > >> > On 21/12/16 22:12, Stefano Stabellini wrote: > >> >> > >> >> On Wed, 21 Dec 2016, Julien Grall wrote: > >> >>> > >> >>> On 20/12/2016 20:53, Stefano Stabellini wrote: > >> > >> On Tue, 20 Dec 2016, Julien Grall wrote: > >> > > >> > On 19/12/2016 21:24, Stefano Stabellini wrote: > >> >> > >> >> On Mon, 19 Dec 2016, Christoffer Dall wrote: > >> >>> > >> >>> On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > >> >> > >> >> If we use hvm_params for this, we need two new hvm_params and Xen > >> >> needs > >> >> to unmap the pfn from the guest immediately, because we don't want > >> >> the > >> >> guest to have access to it. > >> > > >> > > >> > If you unmap the pfn, the PV backend will not be able to request the > >> > page > >> > because there will be no translation available. > >> > > >> > So what you want to do is preventing the guest to at least write into > >> > region > >> > (not sure if it is worth to restrict read) > >> > >> > >> That's a good idea. > >> > >> > >> > and unmap the page via the hypercall XENMEM_decrease_reservation. > >> > >> > >> That would be issued by the guest itself, right? To save address > >> space? > >> >>> > >> >>> > >> >>> Correct. The main use case today is ballooning, but guest could call it > >> >>> on any > >> >>> other RAM baked page. > >> >>> > >> >>> I was thinking about more about the protection needed. Technically the > >> >>> data in > >> >>> the ring are not trusted. So if the guest is messing up with it, it > >> >>> would > >> >>> not > >> >>> be a big issue. Or did I miss anything here? > >> >> > >> >> > >> >> I understand that a guest would be smart to call > >> >> XENMEM_decrease_reservation on the PV console page for pl011, but it > >> >> cannot be a security measure, because, in fact, it needs to be called by > >> >> the guest. Of course, a malicious guest can simply not call > >> >> XENMEM_decrease_reservation for it. > >> > > >> > > >> > Sorry I was not clear. I was not suggested the guest to call > >> > XENMEM_decrease_reservation on ring for security but a malicious guest > >> > issuing the hypercall on the ring protected and replacing by another > >> > page. > >> > > >> > This is the exact same problem as the one I mentioned on the ITS thread. > >> > The > >> > page live in guest memory but contains data that will only be touched by > >> > Xen. > >> > > >> > If you remove those page from stage-2, the translation IPA -> MFN will be > >> > lost unless you store somewhere else. You would have to do it per-page as > >> > the buffer will use contiguous IPA but potentially noncontiguous MFN. > >> > > >> > In the case of ITS the memory is provisioned by the guest. So there are > >> > not > >> > much to do there except adding protection in stage-2 such as write > >> > protection and preventing the guest to unmap it. However for the pl011 > >> > ring, > >> > as Andrew pointed on IRC, what we need to do is accounting this page to > >> > the > >> > domain memory. No mapping is necessary in stage-2. > >> > >> Please clarify what is meant by that no stage-2 mapping is required. > >> Does it mean that no stage-2 mapping is required for the guest as it > >> never needs to access this page? > > > > That's right. > > > > > >> However, the Xen HYP will need the stage-2
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, I am trying to allocate a new SPI VIRQ for sending pl011 interrupt to the guest OS. Currently Xen does not allow to allocate a SPI VIRQ for a guest domain. I tried allocating a new SPI VIRQ by calling vgic_allocate_spi() but it failed as the SPI VIRQ range for the guests is limited to 0 (vgic_num_irqs() return 32 and therefore in effect says that no SPI VIRQ can be allocated because below 32 are for PPI/SGI VIRQs). I tried modifying the guest domU configuration to specify the SPI IRQs to be allowed for the guest but that operation fails as it is not allowed for the unprivileged guests. Is there a way to allow a SPI VIRQ to be allocated to a guest OS. I could not find any instance in the Xen where it is allocating a SPI VIRQ. I was able to register a new PPI VIRQ for pl011 interrupt and could inject that interrupt into the guest and handle the interrupt in the guest. Regards, Bhupinder On 4 January 2017 at 01:38, Stefano Stabellini wrote: > On Thu, 29 Dec 2016, Bhupinder Thakur wrote: >> On 28 December 2016 at 23:19, Julien Grall wrote: >> > On 21/12/16 22:12, Stefano Stabellini wrote: >> >> >> >> On Wed, 21 Dec 2016, Julien Grall wrote: >> >>> >> >>> On 20/12/2016 20:53, Stefano Stabellini wrote: >> >> On Tue, 20 Dec 2016, Julien Grall wrote: >> > >> > On 19/12/2016 21:24, Stefano Stabellini wrote: >> >> >> >> On Mon, 19 Dec 2016, Christoffer Dall wrote: >> >>> >> >>> On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: >> >> >> >> If we use hvm_params for this, we need two new hvm_params and Xen >> >> needs >> >> to unmap the pfn from the guest immediately, because we don't want the >> >> guest to have access to it. >> > >> > >> > If you unmap the pfn, the PV backend will not be able to request the >> > page >> > because there will be no translation available. >> > >> > So what you want to do is preventing the guest to at least write into >> > region >> > (not sure if it is worth to restrict read) >> >> >> That's a good idea. >> >> >> > and unmap the page via the hypercall XENMEM_decrease_reservation. >> >> >> That would be issued by the guest itself, right? To save address space? >> >>> >> >>> >> >>> Correct. The main use case today is ballooning, but guest could call it >> >>> on any >> >>> other RAM baked page. >> >>> >> >>> I was thinking about more about the protection needed. Technically the >> >>> data in >> >>> the ring are not trusted. So if the guest is messing up with it, it would >> >>> not >> >>> be a big issue. Or did I miss anything here? >> >> >> >> >> >> I understand that a guest would be smart to call >> >> XENMEM_decrease_reservation on the PV console page for pl011, but it >> >> cannot be a security measure, because, in fact, it needs to be called by >> >> the guest. Of course, a malicious guest can simply not call >> >> XENMEM_decrease_reservation for it. >> > >> > >> > Sorry I was not clear. I was not suggested the guest to call >> > XENMEM_decrease_reservation on ring for security but a malicious guest >> > issuing the hypercall on the ring protected and replacing by another page. >> > >> > This is the exact same problem as the one I mentioned on the ITS thread. >> > The >> > page live in guest memory but contains data that will only be touched by >> > Xen. >> > >> > If you remove those page from stage-2, the translation IPA -> MFN will be >> > lost unless you store somewhere else. You would have to do it per-page as >> > the buffer will use contiguous IPA but potentially noncontiguous MFN. >> > >> > In the case of ITS the memory is provisioned by the guest. So there are not >> > much to do there except adding protection in stage-2 such as write >> > protection and preventing the guest to unmap it. However for the pl011 >> > ring, >> > as Andrew pointed on IRC, what we need to do is accounting this page to the >> > domain memory. No mapping is necessary in stage-2. >> >> Please clarify what is meant by that no stage-2 mapping is required. >> Does it mean that no stage-2 mapping is required for the guest as it >> never needs to access this page? > > That's right. > > >> However, the Xen HYP will need the stage-2 mapping to find out the >> pl011 PFN --> physical MFN mapping so that it can map the page to its >> own address space. Currently, I am using prepare_ring_for_helper () to >> map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP >> VA. > > I am not sure what Julien had in mind exactly. I like the idea of not > mapping the page at stage-2, but it is true that many interfaces expect > pfns. If Xen is the one to allocate the pl011 PV console page, then Xen > knows the mfn and could use it to map the page, instead of the pfn. > However, the PV console backend also needs to map the same page, and it > currently does that by calling xc_map_foreign_range, which I believe > also expect a pfn. > > So maybe it is ea
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Thu, 29 Dec 2016, Bhupinder Thakur wrote: > On 28 December 2016 at 23:19, Julien Grall wrote: > > On 21/12/16 22:12, Stefano Stabellini wrote: > >> > >> On Wed, 21 Dec 2016, Julien Grall wrote: > >>> > >>> On 20/12/2016 20:53, Stefano Stabellini wrote: > > On Tue, 20 Dec 2016, Julien Grall wrote: > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > >> > >> On Mon, 19 Dec 2016, Christoffer Dall wrote: > >>> > >>> On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > >> > >> If we use hvm_params for this, we need two new hvm_params and Xen > >> needs > >> to unmap the pfn from the guest immediately, because we don't want the > >> guest to have access to it. > > > > > > If you unmap the pfn, the PV backend will not be able to request the > > page > > because there will be no translation available. > > > > So what you want to do is preventing the guest to at least write into > > region > > (not sure if it is worth to restrict read) > > > That's a good idea. > > > > and unmap the page via the hypercall XENMEM_decrease_reservation. > > > That would be issued by the guest itself, right? To save address space? > >>> > >>> > >>> Correct. The main use case today is ballooning, but guest could call it > >>> on any > >>> other RAM baked page. > >>> > >>> I was thinking about more about the protection needed. Technically the > >>> data in > >>> the ring are not trusted. So if the guest is messing up with it, it would > >>> not > >>> be a big issue. Or did I miss anything here? > >> > >> > >> I understand that a guest would be smart to call > >> XENMEM_decrease_reservation on the PV console page for pl011, but it > >> cannot be a security measure, because, in fact, it needs to be called by > >> the guest. Of course, a malicious guest can simply not call > >> XENMEM_decrease_reservation for it. > > > > > > Sorry I was not clear. I was not suggested the guest to call > > XENMEM_decrease_reservation on ring for security but a malicious guest > > issuing the hypercall on the ring protected and replacing by another page. > > > > This is the exact same problem as the one I mentioned on the ITS thread. The > > page live in guest memory but contains data that will only be touched by > > Xen. > > > > If you remove those page from stage-2, the translation IPA -> MFN will be > > lost unless you store somewhere else. You would have to do it per-page as > > the buffer will use contiguous IPA but potentially noncontiguous MFN. > > > > In the case of ITS the memory is provisioned by the guest. So there are not > > much to do there except adding protection in stage-2 such as write > > protection and preventing the guest to unmap it. However for the pl011 ring, > > as Andrew pointed on IRC, what we need to do is accounting this page to the > > domain memory. No mapping is necessary in stage-2. > > Please clarify what is meant by that no stage-2 mapping is required. > Does it mean that no stage-2 mapping is required for the guest as it > never needs to access this page? That's right. > However, the Xen HYP will need the stage-2 mapping to find out the > pl011 PFN --> physical MFN mapping so that it can map the page to its > own address space. Currently, I am using prepare_ring_for_helper () to > map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP > VA. I am not sure what Julien had in mind exactly. I like the idea of not mapping the page at stage-2, but it is true that many interfaces expect pfns. If Xen is the one to allocate the pl011 PV console page, then Xen knows the mfn and could use it to map the page, instead of the pfn. However, the PV console backend also needs to map the same page, and it currently does that by calling xc_map_foreign_range, which I believe also expect a pfn. So maybe it is easier to use write-protection in stage-2 (as for ITS), unless Julien has a better idea? ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, 28 Dec 2016, Julien Grall wrote: > Hi Stefano, > > On 21/12/16 22:12, Stefano Stabellini wrote: > > On Wed, 21 Dec 2016, Julien Grall wrote: > > > On 20/12/2016 20:53, Stefano Stabellini wrote: > > > > On Tue, 20 Dec 2016, Julien Grall wrote: > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > > > If we use hvm_params for this, we need two new hvm_params and Xen > > > > > > needs > > > > > > to unmap the pfn from the guest immediately, because we don't want > > > > > > the > > > > > > guest to have access to it. > > > > > > > > > > If you unmap the pfn, the PV backend will not be able to request the > > > > > page > > > > > because there will be no translation available. > > > > > > > > > > So what you want to do is preventing the guest to at least write into > > > > > region > > > > > (not sure if it is worth to restrict read) > > > > > > > > That's a good idea. > > > > > > > > > > > > > and unmap the page via the hypercall XENMEM_decrease_reservation. > > > > > > > > That would be issued by the guest itself, right? To save address space? > > > > > > Correct. The main use case today is ballooning, but guest could call it on > > > any > > > other RAM baked page. > > > > > > I was thinking about more about the protection needed. Technically the > > > data in > > > the ring are not trusted. So if the guest is messing up with it, it would > > > not > > > be a big issue. Or did I miss anything here? > > > > I understand that a guest would be smart to call > > XENMEM_decrease_reservation on the PV console page for pl011, but it > > cannot be a security measure, because, in fact, it needs to be called by > > the guest. Of course, a malicious guest can simply not call > > XENMEM_decrease_reservation for it. > > Sorry I was not clear. I was not suggested the guest to call > XENMEM_decrease_reservation on ring for security but a malicious guest issuing > the hypercall on the ring protected and replacing by another page. > > This is the exact same problem as the one I mentioned on the ITS thread. The > page live in guest memory but contains data that will only be touched by Xen. > > If you remove those page from stage-2, the translation IPA -> MFN will be lost > unless you store somewhere else. You would have to do it per-page as the > buffer will use contiguous IPA but potentially noncontiguous MFN. > > In the case of ITS the memory is provisioned by the guest. So there are not > much to do there except adding protection in stage-2 such as write protection > and preventing the guest to unmap it. However for the pl011 ring, as Andrew > pointed on IRC, what we need to do is accounting this page to the domain > memory. No mapping is necessary in stage-2. Thanks Julien for the explanation. I think you are right. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On 28 December 2016 at 23:19, Julien Grall wrote: > On 21/12/16 22:12, Stefano Stabellini wrote: >> >> On Wed, 21 Dec 2016, Julien Grall wrote: >>> >>> On 20/12/2016 20:53, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Julien Grall wrote: > > On 19/12/2016 21:24, Stefano Stabellini wrote: >> >> On Mon, 19 Dec 2016, Christoffer Dall wrote: >>> >>> On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: >> >> If we use hvm_params for this, we need two new hvm_params and Xen >> needs >> to unmap the pfn from the guest immediately, because we don't want the >> guest to have access to it. > > > If you unmap the pfn, the PV backend will not be able to request the > page > because there will be no translation available. > > So what you want to do is preventing the guest to at least write into > region > (not sure if it is worth to restrict read) That's a good idea. > and unmap the page via the hypercall XENMEM_decrease_reservation. That would be issued by the guest itself, right? To save address space? >>> >>> >>> Correct. The main use case today is ballooning, but guest could call it >>> on any >>> other RAM baked page. >>> >>> I was thinking about more about the protection needed. Technically the >>> data in >>> the ring are not trusted. So if the guest is messing up with it, it would >>> not >>> be a big issue. Or did I miss anything here? >> >> >> I understand that a guest would be smart to call >> XENMEM_decrease_reservation on the PV console page for pl011, but it >> cannot be a security measure, because, in fact, it needs to be called by >> the guest. Of course, a malicious guest can simply not call >> XENMEM_decrease_reservation for it. > > > Sorry I was not clear. I was not suggested the guest to call > XENMEM_decrease_reservation on ring for security but a malicious guest > issuing the hypercall on the ring protected and replacing by another page. > > This is the exact same problem as the one I mentioned on the ITS thread. The > page live in guest memory but contains data that will only be touched by > Xen. > > If you remove those page from stage-2, the translation IPA -> MFN will be > lost unless you store somewhere else. You would have to do it per-page as > the buffer will use contiguous IPA but potentially noncontiguous MFN. > > In the case of ITS the memory is provisioned by the guest. So there are not > much to do there except adding protection in stage-2 such as write > protection and preventing the guest to unmap it. However for the pl011 ring, > as Andrew pointed on IRC, what we need to do is accounting this page to the > domain memory. No mapping is necessary in stage-2. Please clarify what is meant by that no stage-2 mapping is required. Does it mean that no stage-2 mapping is required for the guest as it never needs to access this page? However, the Xen HYP will need the stage-2 mapping to find out the pl011 PFN --> physical MFN mapping so that it can map the page to its own address space. Currently, I am using prepare_ring_for_helper () to map the pl011 PFN (passed via hvm call) ---> phyiscal MFN ---> Xen HYP VA. Regards, Bhupinder ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On 28/12/16 17:49, Julien Grall wrote: On 21/12/16 22:12, Stefano Stabellini wrote: On Wed, 21 Dec 2016, Julien Grall wrote: On 20/12/2016 20:53, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Julien Grall wrote: On 19/12/2016 21:24, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: If we use hvm_params for this, we need two new hvm_params and Xen needs to unmap the pfn from the guest immediately, because we don't want the guest to have access to it. If you unmap the pfn, the PV backend will not be able to request the page because there will be no translation available. So what you want to do is preventing the guest to at least write into region (not sure if it is worth to restrict read) That's a good idea. and unmap the page via the hypercall XENMEM_decrease_reservation. That would be issued by the guest itself, right? To save address space? Correct. The main use case today is ballooning, but guest could call it on any other RAM baked page. I was thinking about more about the protection needed. Technically the data in the ring are not trusted. So if the guest is messing up with it, it would not be a big issue. Or did I miss anything here? I understand that a guest would be smart to call XENMEM_decrease_reservation on the PV console page for pl011, but it cannot be a security measure, because, in fact, it needs to be called by the guest. Of course, a malicious guest can simply not call XENMEM_decrease_reservation for it. Sorry I was not clear. I was not suggested the guest to call XENMEM_decrease_reservation on ring for security but a malicious guest issuing the hypercall on the ring protected and replacing by another page. This is the exact same problem as the one I mentioned on the ITS thread. The page live in guest memory but contains data that will only be touched by Xen. If you remove those page from stage-2, the translation IPA -> MFN will be lost unless you store somewhere else. You would have to do it per-page as the buffer will use contiguous IPA but potentially noncontiguous MFN. In the case of ITS the memory is provisioned by the guest. So there are not much to do there except adding protection in stage-2 such as write protection and preventing the guest to unmap it. However for the pl011 ring, as Andrew pointed on IRC, what we need to do is accounting this page to the domain memory. No mapping is necessary in stage-2. I forgot to mention. The that backend of PV console is already considering the data in the ring untrusted. So I think in this case, protection is not strictly necessary. The worst thing that could happen is the guest is corrupting one of its console. Not really a big deal. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 21/12/16 22:12, Stefano Stabellini wrote: On Wed, 21 Dec 2016, Julien Grall wrote: On 20/12/2016 20:53, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Julien Grall wrote: On 19/12/2016 21:24, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: If we use hvm_params for this, we need two new hvm_params and Xen needs to unmap the pfn from the guest immediately, because we don't want the guest to have access to it. If you unmap the pfn, the PV backend will not be able to request the page because there will be no translation available. So what you want to do is preventing the guest to at least write into region (not sure if it is worth to restrict read) That's a good idea. and unmap the page via the hypercall XENMEM_decrease_reservation. That would be issued by the guest itself, right? To save address space? Correct. The main use case today is ballooning, but guest could call it on any other RAM baked page. I was thinking about more about the protection needed. Technically the data in the ring are not trusted. So if the guest is messing up with it, it would not be a big issue. Or did I miss anything here? I understand that a guest would be smart to call XENMEM_decrease_reservation on the PV console page for pl011, but it cannot be a security measure, because, in fact, it needs to be called by the guest. Of course, a malicious guest can simply not call XENMEM_decrease_reservation for it. Sorry I was not clear. I was not suggested the guest to call XENMEM_decrease_reservation on ring for security but a malicious guest issuing the hypercall on the ring protected and replacing by another page. This is the exact same problem as the one I mentioned on the ITS thread. The page live in guest memory but contains data that will only be touched by Xen. If you remove those page from stage-2, the translation IPA -> MFN will be lost unless you store somewhere else. You would have to do it per-page as the buffer will use contiguous IPA but potentially noncontiguous MFN. In the case of ITS the memory is provisioned by the guest. So there are not much to do there except adding protection in stage-2 such as write protection and preventing the guest to unmap it. However for the pl011 ring, as Andrew pointed on IRC, what we need to do is accounting this page to the domain memory. No mapping is necessary in stage-2. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, 21 Dec 2016, Julien Grall wrote: > Hi Stefano, > > On 20/12/2016 20:53, Stefano Stabellini wrote: > > On Tue, 20 Dec 2016, Julien Grall wrote: > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > Finally, we cannot hijack one of the guest PV consoles, regardless of > > > > whether it's the first console or one of the others, because the guest > > > > can always try to use them at any time. We need a PV console reserved > > > > for Xen-Dom0 communications on behalf of the guest. When a VM is created > > > > with "pl011=y", the toolstack needs to allocate one more page and evtchn > > > > for the exclusive hypervisor usage. They are not going to be advertised > > > > to the guest as PV consoles; otherwise, the guest could rightfully > > > > access them. > > > > > > > > Both Xen and the PV console backend need access to the two numbers (pfn > > > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack > > > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params. > > > > > > I think it will be the other way around. Xen will allocate the event > > > channel > > > and then report to the PV backends. Very similar to what it is done for > > > ioreq > > > server on x86 today. > > > > It could work that way too. Even in the ioreq case though, the ioreq > > parameters are still exposed via hvm_params (I am looking at > > include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU). > > I am fine with exposing the event channel via hvm_params. My previous mail was > more related of who is allocating the event channel. > > From my understanding, any event channel between Xen and a domain should > currently be allocated by Xen via the function > alloc_unbound_xen_event_channel. OK > > > > If we use hvm_params for this, we need two new hvm_params and Xen needs > > > > to unmap the pfn from the guest immediately, because we don't want the > > > > guest to have access to it. > > > > > > If you unmap the pfn, the PV backend will not be able to request the page > > > because there will be no translation available. > > > > > > So what you want to do is preventing the guest to at least write into > > > region > > > (not sure if it is worth to restrict read) > > > > That's a good idea. > > > > > > > and unmap the page via the hypercall XENMEM_decrease_reservation. > > > > That would be issued by the guest itself, right? To save address space? > > Correct. The main use case today is ballooning, but guest could call it on any > other RAM baked page. > > I was thinking about more about the protection needed. Technically the data in > the ring are not trusted. So if the guest is messing up with it, it would not > be a big issue. Or did I miss anything here? I understand that a guest would be smart to call XENMEM_decrease_reservation on the PV console page for pl011, but it cannot be a security measure, because, in fact, it needs to be called by the guest. Of course, a malicious guest can simply not call XENMEM_decrease_reservation for it. > [...] > > > > IIRC, the UEFI firmware will use Xen console by default but I am not sure > > > it > > > will fallback to the PL011 if present. So we may require some change in > > > the > > > firmware to allow booting on different configuration (i.e PL011 guest or > > > PV > > > console guest). > > > > Linux has checks to see if the pfn or evtchn are 0 and skips PV console > > initialization in those cases. Tianocores probably has the same checks > > (but I haven't read the code). > > Probably, I am more concerned about Tianocore console not falling back on the > PL011. The UEFI firmware is specifically built for Xen ARM guest, so I would > be surprised to see a PL011 driver included. Good point. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 20/12/2016 22:33, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Christoffer Dall wrote: On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: hvm_param is very easy to use, but the guest has access to it too. If we used hvm_param, we would need to make sure that the guest is not able to cause any damage. On the other hand, if we introduced a new hypercall, then we wouldn't have to worry about the guest. But it would be another new hypercall. Another option we would be to introduce a set of hvm_params which are not guest-readable. Today all hvm_params are XSM_TARGET, so both "self" and Dom0 (and stubdoms) can issue hvm_params. We could restrict a few of them to XSM_DM_PRIV, which only allow Dom0 (and stubdoms) to issue them. It would be as simple as changing the xsm check for a subset of them. Obviously we would clearly document which are which. Thoughts? That would work. FWIW, x86 is already restricting the access to some HVM parameter (see hvm_allow_set_param). Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 20/12/2016 20:53, Stefano Stabellini wrote: On Tue, 20 Dec 2016, Julien Grall wrote: On 19/12/2016 21:24, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: Finally, we cannot hijack one of the guest PV consoles, regardless of whether it's the first console or one of the others, because the guest can always try to use them at any time. We need a PV console reserved for Xen-Dom0 communications on behalf of the guest. When a VM is created with "pl011=y", the toolstack needs to allocate one more page and evtchn for the exclusive hypervisor usage. They are not going to be advertised to the guest as PV consoles; otherwise, the guest could rightfully access them. Both Xen and the PV console backend need access to the two numbers (pfn and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack should use another way to tell pfn and evtchn to Xen, maybe hvm_params. I think it will be the other way around. Xen will allocate the event channel and then report to the PV backends. Very similar to what it is done for ioreq server on x86 today. It could work that way too. Even in the ioreq case though, the ioreq parameters are still exposed via hvm_params (I am looking at include/hw/xen/xen_common.h:xen_get_default_ioreq_server_info in QEMU). I am fine with exposing the event channel via hvm_params. My previous mail was more related of who is allocating the event channel. From my understanding, any event channel between Xen and a domain should currently be allocated by Xen via the function alloc_unbound_xen_event_channel. If we use hvm_params for this, we need two new hvm_params and Xen needs to unmap the pfn from the guest immediately, because we don't want the guest to have access to it. If you unmap the pfn, the PV backend will not be able to request the page because there will be no translation available. So what you want to do is preventing the guest to at least write into region (not sure if it is worth to restrict read) That's a good idea. and unmap the page via the hypercall XENMEM_decrease_reservation. That would be issued by the guest itself, right? To save address space? Correct. The main use case today is ballooning, but guest could call it on any other RAM baked page. I was thinking about more about the protection needed. Technically the data in the ring are not trusted. So if the guest is messing up with it, it would not be a big issue. Or did I miss anything here? [...] IIRC, the UEFI firmware will use Xen console by default but I am not sure it will fallback to the PL011 if present. So we may require some change in the firmware to allow booting on different configuration (i.e PL011 guest or PV console guest). Linux has checks to see if the pfn or evtchn are 0 and skips PV console initialization in those cases. Tianocores probably has the same checks (but I haven't read the code). Probably, I am more concerned about Tianocore console not falling back on the PL011. The UEFI firmware is specifically built for Xen ARM guest, so I would be surprised to see a PL011 driver included. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, Dec 20, 2016 at 01:33:39PM -0800, Stefano Stabellini wrote: > On Tue, 20 Dec 2016, Christoffer Dall wrote: > > Hi Stefano, > > > > On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote: > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > > (CC rest maintainers for event channel questions) > > > > > > > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > > > > >Hi, > > > > > > > > > > Hi Bhupinder, > > > > > > > > > > >The idea is for Xen to act as an intermediary as shown below: > > > > > > > > > > > > ring buffers > > > > > > rx/tx fifo > > > > > >dom0 <---> Xen HYP (running pl011 emulation) > > > > > ><---> domU > > > > > >event > > > > > > interrupts > > > > > > > > > > > >Xen will directly manage the in/out console ring buffers (allocated > > > > > >by > > > > > >dom0 for dom0-domU console communication) for reading/writing console > > > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > > > > >read/write data from/to domU and pass it on to/from dom0 over the > > > > > >in/out console ring buffers. There should be no change in dom0 as it > > > > > >will still use the same ring buffers. Similarly there should be no > > > > > >change in domU which would be running a standard pll011 driver. > > > > > > > > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I > > > > > >want to intercept the console events in Xen HYP which pass between > > > > > >dom0 and domU. For now, I just want to capture console data coming > > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > > > > >interface is working. > > > > > > > > > > > >Since each guest domain will have a unique event channel assigned for > > > > > >console communication, Xen HYP can find out the event channel for a > > > > > >given domU from the start_info page of that domU, which should have > > > > > > > > > > The start_info page is x86 specific. If you want to get the console > > > > > event channel for ARM, you would have to use > > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > > > > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > > > > in libxc/xc_dom_arm.c). > > > > > > > > > > >been allocated by dom0. Whenever, an event is to be dispatched via > > > > > >evtchn_send() API in Xen, it can check if the event channel is the > > > > > >console event channel for a given domU. If yes and its source domain > > > > > >is dom0 and destination domain is domU then it will write the data > > > > > >back to the console out ring buffer of the domU and raise a console > > > > > >event to dom0. > > > > > > > > > > > >Once this interface is working, Xen HYP can check the source and > > > > > >destination dom ids and decide which way the event came from and > > > > > >accordingly process the console data. To allow a mix of PV console > > > > > >guests and pl011 guests, Xen might have to maintain a flag per > > > > > >domain, > > > > > >which tells whether Xen HYP should intercept and process the data > > > > > >(for > > > > > >pl011 UART case) or let it go transparently (for PV conosle case). > > > > > > > > > > I am not very familiar with the event channel code. I will let the > > > > > others comment on this bit. > > > > > > > > > > Regardless that, how would you decide whether the hypervisor should > > > > > intercept the notification? > > > > > > > > > > I can see 2 different cases: > > > > > 1) The guest is starting to use the pl011 then move to the HVC > > > > > console (or HVC then pl011) > > > > > 2) The guest is using both the PL011 and the HVC console > > > > > > > > > > Should we consider the second case valid? I would say yes, because a > > > > > user could specify both on the command line. If we use the same > > > > > ring, the output would be a total garbage. > > > > > > > > > > So maybe we need to allocate two distinct rings and event channel? > > > > > > > > This sounds like the only sensible thing to me. I think this is really > > > > about adding a new device to the Xen virtual platform, and providing the > > > > user the option to choose which one he wants the tool in Dom0 to be > > > > presented using stdin/out. Presumably the other console/serial can be > > > > redirected to a file or socket or something? > > > > > > Let me explain how the PV console protocol and drivers work, because > > > they are a bit unusual. > > > > Thanks for this. As my detailed knowledge of Xen is relatively limited > > I can only contribute with a very high level approach to which problem > > we're trying to solve. > > > > > > > The first PV console is advertised via > > > hvm_params. The guest calls: > > > > > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > > > hvm_get_parameter(HVM_PARAM
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 20 Dec 2016, Bhupinder Thakur wrote: > Hi Stefano, > > Thanks for a detailed explanation. I have some queries. > > > Let me explain how the PV console protocol and drivers work, because > > they are a bit unusual. The first PV console is advertised via > > hvm_params. The guest calls: > > > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > > > In drivers/tty/hvc/hvc_xen.c, I see that there are two console init > functions - xen_hvm_console_init () and xen_pv_console_init () and > either of them are called based on the xen_domain_type(). Can you > clarify which init () is called in which case? Good question. x86 Xen has two different types of guests, PV guests and HVM guests. In Linux, many code paths depend on whether a guest is PV or HVM. However, on ARM we only have one kind of guest which is neither PV or HVM. In this case, initialization on ARM follows the HVM path, so xen_hvm_console_init is the one that is called. It isn't always the case though. Feel free to ask (or add a printk) if you are in doubt. > > to get the two parameters to setup the ring and evtchn. If they are 0, > > the guest considers the first console unavailable. Other PV console > > rings, from the second onward, are advertised via xenstore like any > > other Xen PV protocols. In those cases, frontend and backend access > > xenstore to setup ring and event channel. > > > > The PV console backends are unusual too. xenconsoled, available on all > > Xen systems, is one process per host and can handle only one PV console > > per domain. Specifically, it is only able to deal with the first console. > > Domains that have multiple PV consoles require QEMU (not as an emulator, > > but as a PV backends provider). The toolstack writes "type" = > > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or > > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for > > pl011 PV consoles, but it wouldn't be the end of the world if we did. > > > > Additionally, Xen cannot speak xenstore. It can neither read nor write > > to it. I don't think we should add xenstore support to the hypervisor > > for this. We need to come up with a solution that doesn't require it. > > > > Finally, we cannot hijack one of the guest PV consoles, regardless of > > whether it's the first console or one of the others, because the guest > > can always try to use them at any time. We need a PV console reserved > > I see that that there are xen events which can be allocated > (alloc_unbound_xen_event_channel ()). Should I use those kind of > events or just allocate a new event for each VM (similar to how the > console event channel is allocated) and intercept and process them in > the Xen HYP? You need to allocate a new event channel for the PV console. In fact, it should be exactly like any other event channels for consoles. > > for Xen-Dom0 communications on behalf of the guest. When a VM is created > > with "pl011=y", the toolstack needs to allocate one more page and evtchn > > for the exclusive hypervisor usage. They are not going to be advertised > > to the guest as PV consoles; otherwise, the guest could rightfully > > access them. > > > There is a function alloc_magic_pages () (defined in > tools/libxc/xc_dom_arm.c), where the console PFN and event channel are > allocated for a new domain. So can I allocate a new PFN and the event > channel here for pl011? That is what I was thinking. > > Both Xen and the PV console backend need access to the two numbers (pfn > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params. > > If we use hvm_params for this, we need two new hvm_params and Xen needs > > to unmap the pfn from the guest immediately, because we don't want the > > guest to have access to it. > > > This looks fine. We can add new hvm_params to communicate the PFN and > the event channel for Xen to use. These hvm_params can be called from > alloc_magic_pages(). That is the idea. > > However, the PV console backend can access xenstore, so in that case, it > > is fine to write the pfn and evtchn of the PV console for pl011 to > > xenstore, paying attention at using the xenstore permissions > > appropriately. There are no reasons why the guest should have access to > > them; only the console backend should be able to read them. Given that > > the console backend has dom0 privileges, it is not a problem. I also > > suggest using new xenstore nodes, different from any of the existing PV > > console nodes. For example: > > > > /local/domain/$DOMID/xen-console/$NUM/ring-ref > > /local/domain/$DOMID/xen-console/$NUM/port > > > > Where $DOMID is the guest domain id, and $NUM is the console number, > > starting from 0. If we use new hvm_parms for the pl011 PV console, we > > might get away without any xenstore stuff. > > > > For simplicity, given that xenconsoled doesn't suppo
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 20 Dec 2016, Christoffer Dall wrote: > Hi Stefano, > > On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote: > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > (CC rest maintainers for event channel questions) > > > > > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > > > >Hi, > > > > > > > > Hi Bhupinder, > > > > > > > > >The idea is for Xen to act as an intermediary as shown below: > > > > > > > > > > ring buffers > > > > > rx/tx fifo > > > > >dom0 <---> Xen HYP (running pl011 emulation) > > > > ><---> domU > > > > >event > > > > > interrupts > > > > > > > > > >Xen will directly manage the in/out console ring buffers (allocated by > > > > >dom0 for dom0-domU console communication) for reading/writing console > > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > > > >read/write data from/to domU and pass it on to/from dom0 over the > > > > >in/out console ring buffers. There should be no change in dom0 as it > > > > >will still use the same ring buffers. Similarly there should be no > > > > >change in domU which would be running a standard pll011 driver. > > > > > > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I > > > > >want to intercept the console events in Xen HYP which pass between > > > > >dom0 and domU. For now, I just want to capture console data coming > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > > > >interface is working. > > > > > > > > > >Since each guest domain will have a unique event channel assigned for > > > > >console communication, Xen HYP can find out the event channel for a > > > > >given domU from the start_info page of that domU, which should have > > > > > > > > The start_info page is x86 specific. If you want to get the console > > > > event channel for ARM, you would have to use > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > > > in libxc/xc_dom_arm.c). > > > > > > > > >been allocated by dom0. Whenever, an event is to be dispatched via > > > > >evtchn_send() API in Xen, it can check if the event channel is the > > > > >console event channel for a given domU. If yes and its source domain > > > > >is dom0 and destination domain is domU then it will write the data > > > > >back to the console out ring buffer of the domU and raise a console > > > > >event to dom0. > > > > > > > > > >Once this interface is working, Xen HYP can check the source and > > > > >destination dom ids and decide which way the event came from and > > > > >accordingly process the console data. To allow a mix of PV console > > > > >guests and pl011 guests, Xen might have to maintain a flag per domain, > > > > >which tells whether Xen HYP should intercept and process the data (for > > > > >pl011 UART case) or let it go transparently (for PV conosle case). > > > > > > > > I am not very familiar with the event channel code. I will let the > > > > others comment on this bit. > > > > > > > > Regardless that, how would you decide whether the hypervisor should > > > > intercept the notification? > > > > > > > > I can see 2 different cases: > > > > 1) The guest is starting to use the pl011 then move to the HVC > > > > console (or HVC then pl011) > > > > 2) The guest is using both the PL011 and the HVC console > > > > > > > > Should we consider the second case valid? I would say yes, because a > > > > user could specify both on the command line. If we use the same > > > > ring, the output would be a total garbage. > > > > > > > > So maybe we need to allocate two distinct rings and event channel? > > > > > > This sounds like the only sensible thing to me. I think this is really > > > about adding a new device to the Xen virtual platform, and providing the > > > user the option to choose which one he wants the tool in Dom0 to be > > > presented using stdin/out. Presumably the other console/serial can be > > > redirected to a file or socket or something? > > > > Let me explain how the PV console protocol and drivers work, because > > they are a bit unusual. > > Thanks for this. As my detailed knowledge of Xen is relatively limited > I can only contribute with a very high level approach to which problem > we're trying to solve. > > > > The first PV console is advertised via > > hvm_params. The guest calls: > > > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > > > > to get the two parameters to setup the ring and evtchn. If they are 0, > > the guest considers the first console unavailable. Other PV console > > rings, from the second onward, are advertised via xenstore like any > > other Xen PV protocols. In those cases, fronten
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 20 Dec 2016, Stefano Stabellini wrote: > On Tue, 20 Dec 2016, Julien Grall wrote: > > Hi Stefano, > > > > On 19/12/2016 21:24, Stefano Stabellini wrote: > > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > > (CC rest maintainers for event channel questions) > > > > > > > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > > > > > Hi, > > > > > > > > > > Hi Bhupinder, > > > > > > > > > > > The idea is for Xen to act as an intermediary as shown below: > > > > > > > > > > > > ring buffers > > > > > > rx/tx fifo > > > > > > dom0 <---> Xen HYP (running pl011 emulation) > > > > > > <---> domU > > > > > >event > > > > > > interrupts > > > > > > > > > > > > Xen will directly manage the in/out console ring buffers (allocated > > > > > > by > > > > > > dom0 for dom0-domU console communication) for reading/writing > > > > > > console > > > > > > data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > > > > > read/write data from/to domU and pass it on to/from dom0 over the > > > > > > in/out console ring buffers. There should be no change in dom0 as it > > > > > > will still use the same ring buffers. Similarly there should be no > > > > > > change in domU which would be running a standard pll011 driver. > > > > > > > > > > > > Currently, I am working on the interface between dom0 and Xen HYP. I > > > > > > want to intercept the console events in Xen HYP which pass between > > > > > > dom0 and domU. For now, I just want to capture console data coming > > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > > > > > interface is working. > > > > > > > > > > > > Since each guest domain will have a unique event channel assigned > > > > > > for > > > > > > console communication, Xen HYP can find out the event channel for a > > > > > > given domU from the start_info page of that domU, which should have > > > > > > > > > > The start_info page is x86 specific. If you want to get the console > > > > > event channel for ARM, you would have to use > > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > > > > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > > > > in libxc/xc_dom_arm.c). > > > > > > > > > > > been allocated by dom0. Whenever, an event is to be dispatched via > > > > > > evtchn_send() API in Xen, it can check if the event channel is the > > > > > > console event channel for a given domU. If yes and its source domain > > > > > > is dom0 and destination domain is domU then it will write the data > > > > > > back to the console out ring buffer of the domU and raise a console > > > > > > event to dom0. > > > > > > > > > > > > Once this interface is working, Xen HYP can check the source and > > > > > > destination dom ids and decide which way the event came from and > > > > > > accordingly process the console data. To allow a mix of PV console > > > > > > guests and pl011 guests, Xen might have to maintain a flag per > > > > > > domain, > > > > > > which tells whether Xen HYP should intercept and process the data > > > > > > (for > > > > > > pl011 UART case) or let it go transparently (for PV conosle case). > > > > > > > > > > I am not very familiar with the event channel code. I will let the > > > > > others comment on this bit. > > > > > > > > > > Regardless that, how would you decide whether the hypervisor should > > > > > intercept the notification? > > > > > > > > > > I can see 2 different cases: > > > > > 1) The guest is starting to use the pl011 then move to the HVC > > > > > console (or HVC then pl011) > > > > > 2) The guest is using both the PL011 and the HVC console > > > > > > > > > > Should we consider the second case valid? I would say yes, because a > > > > > user could specify both on the command line. If we use the same > > > > > ring, the output would be a total garbage. > > > > > > > > > > So maybe we need to allocate two distinct rings and event channel? > > > > > > > > This sounds like the only sensible thing to me. I think this is really > > > > about adding a new device to the Xen virtual platform, and providing the > > > > user the option to choose which one he wants the tool in Dom0 to be > > > > presented using stdin/out. Presumably the other console/serial can be > > > > redirected to a file or socket or something? > > > > > > Let me explain how the PV console protocol and drivers work, because > > > they are a bit unusual. The first PV console is advertised via > > > hvm_params. The guest calls: > > > > > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > > > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > > > > > > to get the two parameters to setup the ring and evtchn. If they are 0, > > > the guest considers the first console unavailable. Other PV console > > >
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Tue, 20 Dec 2016, Julien Grall wrote: > Hi Stefano, > > On 19/12/2016 21:24, Stefano Stabellini wrote: > > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > > (CC rest maintainers for event channel questions) > > > > > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > > > > Hi, > > > > > > > > Hi Bhupinder, > > > > > > > > > The idea is for Xen to act as an intermediary as shown below: > > > > > > > > > > ring buffers > > > > > rx/tx fifo > > > > > dom0 <---> Xen HYP (running pl011 emulation) > > > > > <---> domU > > > > >event > > > > > interrupts > > > > > > > > > > Xen will directly manage the in/out console ring buffers (allocated by > > > > > dom0 for dom0-domU console communication) for reading/writing console > > > > > data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > > > > read/write data from/to domU and pass it on to/from dom0 over the > > > > > in/out console ring buffers. There should be no change in dom0 as it > > > > > will still use the same ring buffers. Similarly there should be no > > > > > change in domU which would be running a standard pll011 driver. > > > > > > > > > > Currently, I am working on the interface between dom0 and Xen HYP. I > > > > > want to intercept the console events in Xen HYP which pass between > > > > > dom0 and domU. For now, I just want to capture console data coming > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > > > > interface is working. > > > > > > > > > > Since each guest domain will have a unique event channel assigned for > > > > > console communication, Xen HYP can find out the event channel for a > > > > > given domU from the start_info page of that domU, which should have > > > > > > > > The start_info page is x86 specific. If you want to get the console > > > > event channel for ARM, you would have to use > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > > > in libxc/xc_dom_arm.c). > > > > > > > > > been allocated by dom0. Whenever, an event is to be dispatched via > > > > > evtchn_send() API in Xen, it can check if the event channel is the > > > > > console event channel for a given domU. If yes and its source domain > > > > > is dom0 and destination domain is domU then it will write the data > > > > > back to the console out ring buffer of the domU and raise a console > > > > > event to dom0. > > > > > > > > > > Once this interface is working, Xen HYP can check the source and > > > > > destination dom ids and decide which way the event came from and > > > > > accordingly process the console data. To allow a mix of PV console > > > > > guests and pl011 guests, Xen might have to maintain a flag per domain, > > > > > which tells whether Xen HYP should intercept and process the data (for > > > > > pl011 UART case) or let it go transparently (for PV conosle case). > > > > > > > > I am not very familiar with the event channel code. I will let the > > > > others comment on this bit. > > > > > > > > Regardless that, how would you decide whether the hypervisor should > > > > intercept the notification? > > > > > > > > I can see 2 different cases: > > > > 1) The guest is starting to use the pl011 then move to the HVC > > > > console (or HVC then pl011) > > > > 2) The guest is using both the PL011 and the HVC console > > > > > > > > Should we consider the second case valid? I would say yes, because a > > > > user could specify both on the command line. If we use the same > > > > ring, the output would be a total garbage. > > > > > > > > So maybe we need to allocate two distinct rings and event channel? > > > > > > This sounds like the only sensible thing to me. I think this is really > > > about adding a new device to the Xen virtual platform, and providing the > > > user the option to choose which one he wants the tool in Dom0 to be > > > presented using stdin/out. Presumably the other console/serial can be > > > redirected to a file or socket or something? > > > > Let me explain how the PV console protocol and drivers work, because > > they are a bit unusual. The first PV console is advertised via > > hvm_params. The guest calls: > > > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > > > > to get the two parameters to setup the ring and evtchn. If they are 0, > > the guest considers the first console unavailable. Other PV console > > rings, from the second onward, are advertised via xenstore like any > > other Xen PV protocols. In those cases, frontend and backend access > > xenstore to setup ring and event channel. > > > > The PV console backends are unusual too. xenconsoled, available on all > > Xen systems, is one proce
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote: > On Mon, 19 Dec 2016, Christoffer Dall wrote: > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > > (CC rest maintainers for event channel questions) > > > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > > >Hi, > > > > > > Hi Bhupinder, > > > > > > >The idea is for Xen to act as an intermediary as shown below: > > > > > > > > ring buffers > > > > rx/tx fifo > > > >dom0 <---> Xen HYP (running pl011 emulation) > > > ><---> domU > > > >event > > > > interrupts > > > > > > > >Xen will directly manage the in/out console ring buffers (allocated by > > > >dom0 for dom0-domU console communication) for reading/writing console > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > > >read/write data from/to domU and pass it on to/from dom0 over the > > > >in/out console ring buffers. There should be no change in dom0 as it > > > >will still use the same ring buffers. Similarly there should be no > > > >change in domU which would be running a standard pll011 driver. > > > > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I > > > >want to intercept the console events in Xen HYP which pass between > > > >dom0 and domU. For now, I just want to capture console data coming > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > > >interface is working. > > > > > > > >Since each guest domain will have a unique event channel assigned for > > > >console communication, Xen HYP can find out the event channel for a > > > >given domU from the start_info page of that domU, which should have > > > > > > The start_info page is x86 specific. If you want to get the console > > > event channel for ARM, you would have to use > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > > in libxc/xc_dom_arm.c). > > > > > > >been allocated by dom0. Whenever, an event is to be dispatched via > > > >evtchn_send() API in Xen, it can check if the event channel is the > > > >console event channel for a given domU. If yes and its source domain > > > >is dom0 and destination domain is domU then it will write the data > > > >back to the console out ring buffer of the domU and raise a console > > > >event to dom0. > > > > > > > >Once this interface is working, Xen HYP can check the source and > > > >destination dom ids and decide which way the event came from and > > > >accordingly process the console data. To allow a mix of PV console > > > >guests and pl011 guests, Xen might have to maintain a flag per domain, > > > >which tells whether Xen HYP should intercept and process the data (for > > > >pl011 UART case) or let it go transparently (for PV conosle case). > > > > > > I am not very familiar with the event channel code. I will let the > > > others comment on this bit. > > > > > > Regardless that, how would you decide whether the hypervisor should > > > intercept the notification? > > > > > > I can see 2 different cases: > > > 1) The guest is starting to use the pl011 then move to the HVC > > > console (or HVC then pl011) > > > 2) The guest is using both the PL011 and the HVC console > > > > > > Should we consider the second case valid? I would say yes, because a > > > user could specify both on the command line. If we use the same > > > ring, the output would be a total garbage. > > > > > > So maybe we need to allocate two distinct rings and event channel? > > > > This sounds like the only sensible thing to me. I think this is really > > about adding a new device to the Xen virtual platform, and providing the > > user the option to choose which one he wants the tool in Dom0 to be > > presented using stdin/out. Presumably the other console/serial can be > > redirected to a file or socket or something? > > Let me explain how the PV console protocol and drivers work, because > they are a bit unusual. Thanks for this. As my detailed knowledge of Xen is relatively limited I can only contribute with a very high level approach to which problem we're trying to solve. > The first PV console is advertised via > hvm_params. The guest calls: > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > > to get the two parameters to setup the ring and evtchn. If they are 0, > the guest considers the first console unavailable. Other PV console > rings, from the second onward, are advertised via xenstore like any > other Xen PV protocols. In those cases, frontend and backend access > xenstore to setup ring and event channel. So, hvm_get_parameter does not access xenstore, because you need the console because xenstore is available, but subsequent consoles can just access xenstore, or did I get this completely wrong?
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, On 19/12/2016 21:24, Stefano Stabellini wrote: On Mon, 19 Dec 2016, Christoffer Dall wrote: On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: (CC rest maintainers for event channel questions) On 16/12/16 10:06, Bhupinder Thakur wrote: Hi, Hi Bhupinder, The idea is for Xen to act as an intermediary as shown below: ring buffers rx/tx fifo dom0 <---> Xen HYP (running pl011 emulation) <---> domU event interrupts Xen will directly manage the in/out console ring buffers (allocated by dom0 for dom0-domU console communication) for reading/writing console data from/to dom0. On the other side, Xen HYP will emulate pl011 to read/write data from/to domU and pass it on to/from dom0 over the in/out console ring buffers. There should be no change in dom0 as it will still use the same ring buffers. Similarly there should be no change in domU which would be running a standard pll011 driver. Currently, I am working on the interface between dom0 and Xen HYP. I want to intercept the console events in Xen HYP which pass between dom0 and domU. For now, I just want to capture console data coming >from dom0 at Xen HYP and loop it back to dom0, to confirm that this interface is working. Since each guest domain will have a unique event channel assigned for console communication, Xen HYP can find out the event channel for a given domU from the start_info page of that domU, which should have The start_info page is x86 specific. If you want to get the console event channel for ARM, you would have to use d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. This parameter will be setup by the toolstack (see alloc_magic_pages in libxc/xc_dom_arm.c). been allocated by dom0. Whenever, an event is to be dispatched via evtchn_send() API in Xen, it can check if the event channel is the console event channel for a given domU. If yes and its source domain is dom0 and destination domain is domU then it will write the data back to the console out ring buffer of the domU and raise a console event to dom0. Once this interface is working, Xen HYP can check the source and destination dom ids and decide which way the event came from and accordingly process the console data. To allow a mix of PV console guests and pl011 guests, Xen might have to maintain a flag per domain, which tells whether Xen HYP should intercept and process the data (for pl011 UART case) or let it go transparently (for PV conosle case). I am not very familiar with the event channel code. I will let the others comment on this bit. Regardless that, how would you decide whether the hypervisor should intercept the notification? I can see 2 different cases: 1) The guest is starting to use the pl011 then move to the HVC console (or HVC then pl011) 2) The guest is using both the PL011 and the HVC console Should we consider the second case valid? I would say yes, because a user could specify both on the command line. If we use the same ring, the output would be a total garbage. So maybe we need to allocate two distinct rings and event channel? This sounds like the only sensible thing to me. I think this is really about adding a new device to the Xen virtual platform, and providing the user the option to choose which one he wants the tool in Dom0 to be presented using stdin/out. Presumably the other console/serial can be redirected to a file or socket or something? Let me explain how the PV console protocol and drivers work, because they are a bit unusual. The first PV console is advertised via hvm_params. The guest calls: hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); to get the two parameters to setup the ring and evtchn. If they are 0, the guest considers the first console unavailable. Other PV console rings, from the second onward, are advertised via xenstore like any other Xen PV protocols. In those cases, frontend and backend access xenstore to setup ring and event channel. The PV console backends are unusual too. xenconsoled, available on all Xen systems, is one process per host and can handle only one PV console per domain. Specifically, it is only able to deal with the first console. Domains that have multiple PV consoles require QEMU (not as an emulator, but as a PV backends provider). The toolstack writes "type" = "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or QEMU are supposed to handle. Ideally, we shouldn't require QEMU for pl011 PV consoles, but it wouldn't be the end of the world if we did. Additionally, Xen cannot speak xenstore. It can neither read nor write to it. I don't think we should add xenstore support to the hypervisor for this. We need to come up with a solution that doesn't require it. Agree on this. Finally, we cannot hijack one of the guest PV consoles, regardless of whether it's the firs
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Stefano, Thanks for a detailed explanation. I have some queries. > > Let me explain how the PV console protocol and drivers work, because > they are a bit unusual. The first PV console is advertised via > hvm_params. The guest calls: > > hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); > hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); > In drivers/tty/hvc/hvc_xen.c, I see that there are two console init functions - xen_hvm_console_init () and xen_pv_console_init () and either of them are called based on the xen_domain_type(). Can you clarify which init () is called in which case? > to get the two parameters to setup the ring and evtchn. If they are 0, > the guest considers the first console unavailable. Other PV console > rings, from the second onward, are advertised via xenstore like any > other Xen PV protocols. In those cases, frontend and backend access > xenstore to setup ring and event channel. > > The PV console backends are unusual too. xenconsoled, available on all > Xen systems, is one process per host and can handle only one PV console > per domain. Specifically, it is only able to deal with the first console. > Domains that have multiple PV consoles require QEMU (not as an emulator, > but as a PV backends provider). The toolstack writes "type" = > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for > pl011 PV consoles, but it wouldn't be the end of the world if we did. > > Additionally, Xen cannot speak xenstore. It can neither read nor write > to it. I don't think we should add xenstore support to the hypervisor > for this. We need to come up with a solution that doesn't require it. > > Finally, we cannot hijack one of the guest PV consoles, regardless of > whether it's the first console or one of the others, because the guest > can always try to use them at any time. We need a PV console reserved I see that that there are xen events which can be allocated (alloc_unbound_xen_event_channel ()). Should I use those kind of events or just allocate a new event for each VM (similar to how the console event channel is allocated) and intercept and process them in the Xen HYP? > for Xen-Dom0 communications on behalf of the guest. When a VM is created > with "pl011=y", the toolstack needs to allocate one more page and evtchn > for the exclusive hypervisor usage. They are not going to be advertised > to the guest as PV consoles; otherwise, the guest could rightfully > access them. > There is a function alloc_magic_pages () (defined in tools/libxc/xc_dom_arm.c), where the console PFN and event channel are allocated for a new domain. So can I allocate a new PFN and the event channel here for pl011? > Both Xen and the PV console backend need access to the two numbers (pfn > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack > should use another way to tell pfn and evtchn to Xen, maybe hvm_params. > If we use hvm_params for this, we need two new hvm_params and Xen needs > to unmap the pfn from the guest immediately, because we don't want the > guest to have access to it. > This looks fine. We can add new hvm_params to communicate the PFN and the event channel for Xen to use. These hvm_params can be called from alloc_magic_pages(). > However, the PV console backend can access xenstore, so in that case, it > is fine to write the pfn and evtchn of the PV console for pl011 to > xenstore, paying attention at using the xenstore permissions > appropriately. There are no reasons why the guest should have access to > them; only the console backend should be able to read them. Given that > the console backend has dom0 privileges, it is not a problem. I also > suggest using new xenstore nodes, different from any of the existing PV > console nodes. For example: > > /local/domain/$DOMID/xen-console/$NUM/ring-ref > /local/domain/$DOMID/xen-console/$NUM/port > > Where $DOMID is the guest domain id, and $NUM is the console number, > starting from 0. If we use new hvm_parms for the pl011 PV console, we > might get away without any xenstore stuff. > > For simplicity, given that xenconsoled doesn't support multiple PV > consoles, we could setup the pl011 PV console *instead* of the regular > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's > safe if the toolstack doesn't provide a PV console. When pl011 is > requested, libxl could set the pfn and evtchn hvm_params to 0 for the > initial PV console. Eventually, it would be nice if xenconsoled was able > to support both consoles at the same time. Is domain_create_ring() responsible for setting up the ring and the event channel for the first PV console? This function seems to refer to xenstore for getting the shared page reference and the remote port. Is this the same shared page which was set by libxl using hvm_param for pfn (for guest to access). Regards, Bhupinder ___ Xen-devel mailing
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Mon, 19 Dec 2016, Christoffer Dall wrote: > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > > (CC rest maintainers for event channel questions) > > > > On 16/12/16 10:06, Bhupinder Thakur wrote: > > >Hi, > > > > Hi Bhupinder, > > > > >The idea is for Xen to act as an intermediary as shown below: > > > > > > ring buffers > > > rx/tx fifo > > >dom0 <---> Xen HYP (running pl011 emulation) > > ><---> domU > > >event > > > interrupts > > > > > >Xen will directly manage the in/out console ring buffers (allocated by > > >dom0 for dom0-domU console communication) for reading/writing console > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to > > >read/write data from/to domU and pass it on to/from dom0 over the > > >in/out console ring buffers. There should be no change in dom0 as it > > >will still use the same ring buffers. Similarly there should be no > > >change in domU which would be running a standard pll011 driver. > > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I > > >want to intercept the console events in Xen HYP which pass between > > >dom0 and domU. For now, I just want to capture console data coming > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > > >interface is working. > > > > > >Since each guest domain will have a unique event channel assigned for > > >console communication, Xen HYP can find out the event channel for a > > >given domU from the start_info page of that domU, which should have > > > > The start_info page is x86 specific. If you want to get the console > > event channel for ARM, you would have to use > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > > > This parameter will be setup by the toolstack (see alloc_magic_pages > > in libxc/xc_dom_arm.c). > > > > >been allocated by dom0. Whenever, an event is to be dispatched via > > >evtchn_send() API in Xen, it can check if the event channel is the > > >console event channel for a given domU. If yes and its source domain > > >is dom0 and destination domain is domU then it will write the data > > >back to the console out ring buffer of the domU and raise a console > > >event to dom0. > > > > > >Once this interface is working, Xen HYP can check the source and > > >destination dom ids and decide which way the event came from and > > >accordingly process the console data. To allow a mix of PV console > > >guests and pl011 guests, Xen might have to maintain a flag per domain, > > >which tells whether Xen HYP should intercept and process the data (for > > >pl011 UART case) or let it go transparently (for PV conosle case). > > > > I am not very familiar with the event channel code. I will let the > > others comment on this bit. > > > > Regardless that, how would you decide whether the hypervisor should > > intercept the notification? > > > > I can see 2 different cases: > > 1) The guest is starting to use the pl011 then move to the HVC > > console (or HVC then pl011) > > 2) The guest is using both the PL011 and the HVC console > > > > Should we consider the second case valid? I would say yes, because a > > user could specify both on the command line. If we use the same > > ring, the output would be a total garbage. > > > > So maybe we need to allocate two distinct rings and event channel? > > This sounds like the only sensible thing to me. I think this is really > about adding a new device to the Xen virtual platform, and providing the > user the option to choose which one he wants the tool in Dom0 to be > presented using stdin/out. Presumably the other console/serial can be > redirected to a file or socket or something? Let me explain how the PV console protocol and drivers work, because they are a bit unusual. The first PV console is advertised via hvm_params. The guest calls: hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v); hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v); to get the two parameters to setup the ring and evtchn. If they are 0, the guest considers the first console unavailable. Other PV console rings, from the second onward, are advertised via xenstore like any other Xen PV protocols. In those cases, frontend and backend access xenstore to setup ring and event channel. The PV console backends are unusual too. xenconsoled, available on all Xen systems, is one process per host and can handle only one PV console per domain. Specifically, it is only able to deal with the first console. Domains that have multiple PV consoles require QEMU (not as an emulator, but as a PV backends provider). The toolstack writes "type" = "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or QEMU are supposed to handle. Ideally, we shouldn't require QEMU for pl011 PV consoles, but it wouldn't be the end of the world if we did. Additionally, Xen cannot speak xenstore. It can neither read no
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote: > (CC rest maintainers for event channel questions) > > On 16/12/16 10:06, Bhupinder Thakur wrote: > >Hi, > > Hi Bhupinder, > > >The idea is for Xen to act as an intermediary as shown below: > > > > ring buffers > > rx/tx fifo > >dom0 <---> Xen HYP (running pl011 emulation) > ><---> domU > >event > > interrupts > > > >Xen will directly manage the in/out console ring buffers (allocated by > >dom0 for dom0-domU console communication) for reading/writing console > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to > >read/write data from/to domU and pass it on to/from dom0 over the > >in/out console ring buffers. There should be no change in dom0 as it > >will still use the same ring buffers. Similarly there should be no > >change in domU which would be running a standard pll011 driver. > > > >Currently, I am working on the interface between dom0 and Xen HYP. I > >want to intercept the console events in Xen HYP which pass between > >dom0 and domU. For now, I just want to capture console data coming > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this > >interface is working. > > > >Since each guest domain will have a unique event channel assigned for > >console communication, Xen HYP can find out the event channel for a > >given domU from the start_info page of that domU, which should have > > The start_info page is x86 specific. If you want to get the console > event channel for ARM, you would have to use > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. > > This parameter will be setup by the toolstack (see alloc_magic_pages > in libxc/xc_dom_arm.c). > > >been allocated by dom0. Whenever, an event is to be dispatched via > >evtchn_send() API in Xen, it can check if the event channel is the > >console event channel for a given domU. If yes and its source domain > >is dom0 and destination domain is domU then it will write the data > >back to the console out ring buffer of the domU and raise a console > >event to dom0. > > > >Once this interface is working, Xen HYP can check the source and > >destination dom ids and decide which way the event came from and > >accordingly process the console data. To allow a mix of PV console > >guests and pl011 guests, Xen might have to maintain a flag per domain, > >which tells whether Xen HYP should intercept and process the data (for > >pl011 UART case) or let it go transparently (for PV conosle case). > > I am not very familiar with the event channel code. I will let the > others comment on this bit. > > Regardless that, how would you decide whether the hypervisor should > intercept the notification? > > I can see 2 different cases: > 1) The guest is starting to use the pl011 then move to the HVC > console (or HVC then pl011) > 2) The guest is using both the PL011 and the HVC console > > Should we consider the second case valid? I would say yes, because a > user could specify both on the command line. If we use the same > ring, the output would be a total garbage. > > So maybe we need to allocate two distinct rings and event channel? This sounds like the only sensible thing to me. I think this is really about adding a new device to the Xen virtual platform, and providing the user the option to choose which one he wants the tool in Dom0 to be presented using stdin/out. Presumably the other console/serial can be redirected to a file or socket or something? Thanks, -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
(CC rest maintainers for event channel questions) On 16/12/16 10:06, Bhupinder Thakur wrote: Hi, Hi Bhupinder, The idea is for Xen to act as an intermediary as shown below: ring buffers rx/tx fifo dom0 <---> Xen HYP (running pl011 emulation) <---> domU event interrupts Xen will directly manage the in/out console ring buffers (allocated by dom0 for dom0-domU console communication) for reading/writing console data from/to dom0. On the other side, Xen HYP will emulate pl011 to read/write data from/to domU and pass it on to/from dom0 over the in/out console ring buffers. There should be no change in dom0 as it will still use the same ring buffers. Similarly there should be no change in domU which would be running a standard pll011 driver. Currently, I am working on the interface between dom0 and Xen HYP. I want to intercept the console events in Xen HYP which pass between dom0 and domU. For now, I just want to capture console data coming from dom0 at Xen HYP and loop it back to dom0, to confirm that this interface is working. Since each guest domain will have a unique event channel assigned for console communication, Xen HYP can find out the event channel for a given domU from the start_info page of that domU, which should have The start_info page is x86 specific. If you want to get the console event channel for ARM, you would have to use d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN]. This parameter will be setup by the toolstack (see alloc_magic_pages in libxc/xc_dom_arm.c). been allocated by dom0. Whenever, an event is to be dispatched via evtchn_send() API in Xen, it can check if the event channel is the console event channel for a given domU. If yes and its source domain is dom0 and destination domain is domU then it will write the data back to the console out ring buffer of the domU and raise a console event to dom0. Once this interface is working, Xen HYP can check the source and destination dom ids and decide which way the event came from and accordingly process the console data. To allow a mix of PV console guests and pl011 guests, Xen might have to maintain a flag per domain, which tells whether Xen HYP should intercept and process the data (for pl011 UART case) or let it go transparently (for PV conosle case). I am not very familiar with the event channel code. I will let the others comment on this bit. Regardless that, how would you decide whether the hypervisor should intercept the notification? I can see 2 different cases: 1) The guest is starting to use the pl011 then move to the HVC console (or HVC then pl011) 2) The guest is using both the PL011 and the HVC console Should we consider the second case valid? I would say yes, because a user could specify both on the command line. If we use the same ring, the output would be a total garbage. So maybe we need to allocate two distinct rings and event channel? For your testing, you could consider that the ring is only used by the pl011. I would recommend you to disable the HVC console driver in the guest (see CONFIG_HVC_XEN). I looked at the event channel code and I can only find a function for xen to create an event channel (see alloc_unbound_xen_event_channel). But in your case, you want a function to bind an event channel. Is this the correct way to find out the console event channel for a given domU or there is some other way? I could not locate in the Xen source code, where exactly the start_info page is created for a new domain, which would contain the console event channel and and machine page where the ring buffers are mapped. You can give a look to HVM_PARAM_CONSOLE_EVTCHN and HVM_PARAM_CONSOLE_PFN (see more details above). Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi, The idea is for Xen to act as an intermediary as shown below: ring buffers rx/tx fifo dom0 <---> Xen HYP (running pl011 emulation) <---> domU event interrupts Xen will directly manage the in/out console ring buffers (allocated by dom0 for dom0-domU console communication) for reading/writing console data from/to dom0. On the other side, Xen HYP will emulate pl011 to read/write data from/to domU and pass it on to/from dom0 over the in/out console ring buffers. There should be no change in dom0 as it will still use the same ring buffers. Similarly there should be no change in domU which would be running a standard pll011 driver. Currently, I am working on the interface between dom0 and Xen HYP. I want to intercept the console events in Xen HYP which pass between dom0 and domU. For now, I just want to capture console data coming from dom0 at Xen HYP and loop it back to dom0, to confirm that this interface is working. Since each guest domain will have a unique event channel assigned for console communication, Xen HYP can find out the event channel for a given domU from the start_info page of that domU, which should have been allocated by dom0. Whenever, an event is to be dispatched via evtchn_send() API in Xen, it can check if the event channel is the console event channel for a given domU. If yes and its source domain is dom0 and destination domain is domU then it will write the data back to the console out ring buffer of the domU and raise a console event to dom0. Once this interface is working, Xen HYP can check the source and destination dom ids and decide which way the event came from and accordingly process the console data. To allow a mix of PV console guests and pl011 guests, Xen might have to maintain a flag per domain, which tells whether Xen HYP should intercept and process the data (for pl011 UART case) or let it go transparently (for PV conosle case). Is this the correct way to find out the console event channel for a given domU or there is some other way? I could not locate in the Xen source code, where exactly the start_info page is created for a new domain, which would contain the console event channel and and machine page where the ring buffers are mapped. Regards, Bhupinder On 2 December 2016 at 03:25, Stefano Stabellini wrote: > On Thu, 1 Dec 2016, Christoffer Dall wrote: >> On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote: >> > On Wed, 30 Nov 2016, Julien Grall wrote: >> > > Hi all, >> > > >> > > Few months ago, Linaro has published the version 2 of the VM >> > > specification >> > > [1]. >> > > >> > > For those who don't know, the specification provides guidelines to >> > > guarantee a >> > > compliant OS images could run on various hypervisor (e.g Xen, KVM). >> > > >> > > Looking at the specification, it will require Xen to expose new devices >> > > to the >> > > guest: pl011, rtc, persistent flash (for UEFI variables). >> > > >> > > The RTC and persistent will only be used by the UEFI firwmare. The >> > > firwmare is >> > > custom made for Xen guest and be loaded by the toolstack, so we could >> > > theoretically provide PV drivers for those. >> > > >> > > This is not the case for the PL011. The guest will be shipped with a >> > > PL011/SBSA UART driver,.This means it will expect to access it through >> > > MMIO. >> > > >> > > So we have to emulate a PL011. The question is where? Before suggesting >> > > some >> > > ideas, the guest/user will expect to be able to interact with the console >> > > through the UART. This means that the UART and xenconsoled needs to >> > > communicate together. >> > > >> > > I think we can distinct two places where the PL011 could be emulated: >> > > in the hypervisor, or outside the hypervisor. >> > > >> > > Emulating the UART in the hypervisor means that we take the risk to >> > > increase >> > > to the attack surface of Xen if there is a bug in the emulation code. The >> > > attack surface could be reduced by emulating the UART in another >> > > exception >> > > level (e.g EL1, EL0) but still under the control of the hypervisor. >> > > Usually >> > > the guest is communicating between with xenconsoled using a ring. For the >> > > first console this could be discovered using hypercall HVMOP_get_param. >> > > For >> > > the second and onwards, it described in xenstore. I would not worry too >> > > much >> > > about emulating multiple PL011s, so we could implement the PV frontend >> > > in Xen. >> > > >> > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain) >> > > would require to bring the concept of ioreq server on ARM. Which left the >> > > question where do we emulate the PL011? The best place would be >> > > xenconsoled. >> > > But I am not sure how would be the security impact here. Does all guest >> > > consoles are emulated within the same daemon? >> > >> > One xen
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Thu, 1 Dec 2016, Christoffer Dall wrote: > On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote: > > On Wed, 30 Nov 2016, Julien Grall wrote: > > > Hi all, > > > > > > Few months ago, Linaro has published the version 2 of the VM specification > > > [1]. > > > > > > For those who don't know, the specification provides guidelines to > > > guarantee a > > > compliant OS images could run on various hypervisor (e.g Xen, KVM). > > > > > > Looking at the specification, it will require Xen to expose new devices > > > to the > > > guest: pl011, rtc, persistent flash (for UEFI variables). > > > > > > The RTC and persistent will only be used by the UEFI firwmare. The > > > firwmare is > > > custom made for Xen guest and be loaded by the toolstack, so we could > > > theoretically provide PV drivers for those. > > > > > > This is not the case for the PL011. The guest will be shipped with a > > > PL011/SBSA UART driver,.This means it will expect to access it through > > > MMIO. > > > > > > So we have to emulate a PL011. The question is where? Before suggesting > > > some > > > ideas, the guest/user will expect to be able to interact with the console > > > through the UART. This means that the UART and xenconsoled needs to > > > communicate together. > > > > > > I think we can distinct two places where the PL011 could be emulated: > > > in the hypervisor, or outside the hypervisor. > > > > > > Emulating the UART in the hypervisor means that we take the risk to > > > increase > > > to the attack surface of Xen if there is a bug in the emulation code. The > > > attack surface could be reduced by emulating the UART in another exception > > > level (e.g EL1, EL0) but still under the control of the hypervisor. > > > Usually > > > the guest is communicating between with xenconsoled using a ring. For the > > > first console this could be discovered using hypercall HVMOP_get_param. > > > For > > > the second and onwards, it described in xenstore. I would not worry too > > > much > > > about emulating multiple PL011s, so we could implement the PV frontend in > > > Xen. > > > > > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain) > > > would require to bring the concept of ioreq server on ARM. Which left the > > > question where do we emulate the PL011? The best place would be > > > xenconsoled. > > > But I am not sure how would be the security impact here. Does all guest > > > consoles are emulated within the same daemon? > > > > One xenconsoled instance handles all PV console frontends. However QEMU, > > not xenconsoled, handles secondary PV consoles and emulated serials. One > > QEMU instance per VM. The PV console protocol is pretty trivial and not > > easy to exploit. > > > > Instead of emulating PL011 in xenconsoled we could do it in QEMU (or > > something like it). But I think we should do something else instead, see > > below. > > > > > > > I would lean towards the first solution if we implement all the security > > > safety I mentioned. Although, the second solution would be a good move if > > > we > > > decide to implement more devices (e.g RTC, pflash) in the future. > > > > > > Do you have any opinions? > > > > As I have just written in this other email: > > > > http://marc.info/?l=xen-devel&m=148054285829397 > > > > I don't think we should introduce userspace emulators in Dom0 on ARM. > > The reason is that they are bad both for security and performance. In > > general I think that the best solution is to introduce emulators in EL1 > > in Xen. > > Slightly hijacking the discussion: > > Why would you run stuff in EL1 as opposed to EL0? > > My understanding of the HCR_EL2.TGE bit is exactly designed to run > bare-metal applications on top of a hypervisor running in EL2 and has > the added benefit that I think you can switch to your bare-metal > application without having to context-switch any EL1 tate, because TGE > just disabled the whole stage 1 MMU. Obviously this doesn't give you > virtual memory, unless you map that using stage 2 tables, but that might > actually be preferred, and since the whole point is to separate your > non-privileged hypervisor applications from the hypervisor's memory > anyhow, it may be fine. > > I don't know enough about Xen's impementation to say whether this is > doable or not, but I think there are potential advantages in terms of > performance and using the architecture as designed. You might be right and it is definitely worth investigating. > Thinking about this, if you run something in EL1 on top of Xen without > setting the TGE bit, what happens if your application misbehaves, for > example causes a page fault? That would be taken at EL1, right? So > will you install an EL1 handler for exceptions that somehow forwards the > trap into EL2? Sounds cleaner to me to catch anything weird the > application does to EL2. Yes, I think it would be better to handle that kind of faults at EL2. __
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hi Christoffer, On 30/11/16 16:24, Christoffer Dall wrote: On Wed, Nov 30, 2016 at 03:29:32PM +, Julien Grall wrote: Hi all, Few months ago, Linaro has published the version 2 of the VM specification [1]. For those who don't know, the specification provides guidelines to guarantee a compliant OS images could run on various hypervisor (e.g Xen, KVM). Looking at the specification, it will require Xen to expose new devices to the guest: pl011, rtc, persistent flash (for UEFI variables). The RTC and persistent will only be used by the UEFI firwmare. Why would a guest booting without UEFI not want to use the RTC directly? Sorry, I had only the VM spec in mind. Of course, an RTC could be used for guest booting without UEFI. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote: > On Wed, 30 Nov 2016, Julien Grall wrote: > > Hi all, > > > > Few months ago, Linaro has published the version 2 of the VM specification > > [1]. > > > > For those who don't know, the specification provides guidelines to > > guarantee a > > compliant OS images could run on various hypervisor (e.g Xen, KVM). > > > > Looking at the specification, it will require Xen to expose new devices to > > the > > guest: pl011, rtc, persistent flash (for UEFI variables). > > > > The RTC and persistent will only be used by the UEFI firwmare. The firwmare > > is > > custom made for Xen guest and be loaded by the toolstack, so we could > > theoretically provide PV drivers for those. > > > > This is not the case for the PL011. The guest will be shipped with a > > PL011/SBSA UART driver,.This means it will expect to access it through MMIO. > > > > So we have to emulate a PL011. The question is where? Before suggesting some > > ideas, the guest/user will expect to be able to interact with the console > > through the UART. This means that the UART and xenconsoled needs to > > communicate together. > > > > I think we can distinct two places where the PL011 could be emulated: > > in the hypervisor, or outside the hypervisor. > > > > Emulating the UART in the hypervisor means that we take the risk to increase > > to the attack surface of Xen if there is a bug in the emulation code. The > > attack surface could be reduced by emulating the UART in another exception > > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually > > the guest is communicating between with xenconsoled using a ring. For the > > first console this could be discovered using hypercall HVMOP_get_param. For > > the second and onwards, it described in xenstore. I would not worry too much > > about emulating multiple PL011s, so we could implement the PV frontend in > > Xen. > > > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain) > > would require to bring the concept of ioreq server on ARM. Which left the > > question where do we emulate the PL011? The best place would be xenconsoled. > > But I am not sure how would be the security impact here. Does all guest > > consoles are emulated within the same daemon? > > One xenconsoled instance handles all PV console frontends. However QEMU, > not xenconsoled, handles secondary PV consoles and emulated serials. One > QEMU instance per VM. The PV console protocol is pretty trivial and not > easy to exploit. > > Instead of emulating PL011 in xenconsoled we could do it in QEMU (or > something like it). But I think we should do something else instead, see > below. > > > > I would lean towards the first solution if we implement all the security > > safety I mentioned. Although, the second solution would be a good move if we > > decide to implement more devices (e.g RTC, pflash) in the future. > > > > Do you have any opinions? > > As I have just written in this other email: > > http://marc.info/?l=xen-devel&m=148054285829397 > > I don't think we should introduce userspace emulators in Dom0 on ARM. > The reason is that they are bad both for security and performance. In > general I think that the best solution is to introduce emulators in EL1 > in Xen. Slightly hijacking the discussion: Why would you run stuff in EL1 as opposed to EL0? My understanding of the HCR_EL2.TGE bit is exactly designed to run bare-metal applications on top of a hypervisor running in EL2 and has the added benefit that I think you can switch to your bare-metal application without having to context-switch any EL1 tate, because TGE just disabled the whole stage 1 MMU. Obviously this doesn't give you virtual memory, unless you map that using stage 2 tables, but that might actually be preferred, and since the whole point is to separate your non-privileged hypervisor applications from the hypervisor's memory anyhow, it may be fine. I don't know enough about Xen's impementation to say whether this is doable or not, but I think there are potential advantages in terms of performance and using the architecture as designed. Thinking about this, if you run something in EL1 on top of Xen without setting the TGE bit, what happens if your application misbehaves, for example causes a page fault? That would be taken at EL1, right? So will you install an EL1 handler for exceptions that somehow forwards the trap into EL2? Sounds cleaner to me to catch anything weird the application does to EL2. -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, 30 Nov 2016, Julien Grall wrote: > Hi all, > > Few months ago, Linaro has published the version 2 of the VM specification > [1]. > > For those who don't know, the specification provides guidelines to guarantee a > compliant OS images could run on various hypervisor (e.g Xen, KVM). > > Looking at the specification, it will require Xen to expose new devices to the > guest: pl011, rtc, persistent flash (for UEFI variables). > > The RTC and persistent will only be used by the UEFI firwmare. The firwmare is > custom made for Xen guest and be loaded by the toolstack, so we could > theoretically provide PV drivers for those. > > This is not the case for the PL011. The guest will be shipped with a > PL011/SBSA UART driver,.This means it will expect to access it through MMIO. > > So we have to emulate a PL011. The question is where? Before suggesting some > ideas, the guest/user will expect to be able to interact with the console > through the UART. This means that the UART and xenconsoled needs to > communicate together. > > I think we can distinct two places where the PL011 could be emulated: > in the hypervisor, or outside the hypervisor. > > Emulating the UART in the hypervisor means that we take the risk to increase > to the attack surface of Xen if there is a bug in the emulation code. The > attack surface could be reduced by emulating the UART in another exception > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually > the guest is communicating between with xenconsoled using a ring. For the > first console this could be discovered using hypercall HVMOP_get_param. For > the second and onwards, it described in xenstore. I would not worry too much > about emulating multiple PL011s, so we could implement the PV frontend in Xen. > > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain) > would require to bring the concept of ioreq server on ARM. Which left the > question where do we emulate the PL011? The best place would be xenconsoled. > But I am not sure how would be the security impact here. Does all guest > consoles are emulated within the same daemon? One xenconsoled instance handles all PV console frontends. However QEMU, not xenconsoled, handles secondary PV consoles and emulated serials. One QEMU instance per VM. The PV console protocol is pretty trivial and not easy to exploit. Instead of emulating PL011 in xenconsoled we could do it in QEMU (or something like it). But I think we should do something else instead, see below. > I would lean towards the first solution if we implement all the security > safety I mentioned. Although, the second solution would be a good move if we > decide to implement more devices (e.g RTC, pflash) in the future. > > Do you have any opinions? As I have just written in this other email: http://marc.info/?l=xen-devel&m=148054285829397 I don't think we should introduce userspace emulators in Dom0 on ARM. The reason is that they are bad both for security and performance. In general I think that the best solution is to introduce emulators in EL1 in Xen. That said, PL011 is pretty small and the PL011 emulator needs to feed data back to Dom0 for user interaction, so, if we need to, we could make an exception for it and run it in userspace Dom0. But do we really need to make an exception? I don't think so. The PL011 emulator could run in EL1 in Xen and connect to xenconsoled using the regular PV console frontend/backend protocol, as you suggested. One caveat: I would avoid introducing xenstore support in Xen, but fortunately we don't need xenstore support for the first PV console connection, so we should be OK. In any case if the xenconsole protocol or xenconsoled are the problem, we could easily rewrite them or add specific support for this new use case. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
Hello Julien, On 30 November 2016 at 17:29, Julien Grall wrote: [...] > I think we can distinct two places where the PL011 could be emulated: > in the hypervisor, or outside the hypervisor. > > Emulating the UART in the hypervisor means that we take the risk to increase > to the attack surface of Xen if there is a bug in the emulation code. The > attack surface could be reduced by emulating the UART in another exception > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually > the guest is communicating between with xenconsoled using a ring. For the > first console this could be discovered using hypercall HVMOP_get_param. For > the second and onwards, it described in xenstore. I would not worry too much > about emulating multiple PL011s, so we could implement the PV frontend in > Xen. > [...] > I would lean towards the first solution if we implement all the security > safety I mentioned. Although, the second solution would be a good move if we > decide to implement more devices (e.g RTC, pflash) in the future. > > Do you have any opinions? Looks like this topic have some in common with OP-TEE thread. I like first solution, because if there will be easy and reliable way to run code in XEN's EL1/EL0, then this will be ideal solution for TEE emulation/mediation layer. So, if you'll choose this way, please bear in mind other uses, like TEE emulation. -- WBR Volodymyr Babchuk aka lorc [+380976646013] mailto: vlad.babc...@gmail.com ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest
On Wed, Nov 30, 2016 at 03:29:32PM +, Julien Grall wrote: > Hi all, > > Few months ago, Linaro has published the version 2 of the VM > specification [1]. > > For those who don't know, the specification provides guidelines to > guarantee a compliant OS images could run on various hypervisor (e.g > Xen, KVM). > > Looking at the specification, it will require Xen to expose new > devices to the guest: pl011, rtc, persistent flash (for UEFI > variables). > > The RTC and persistent will only be used by the UEFI firwmare. Why would a guest booting without UEFI not want to use the RTC directly? Linux does this on KVM today... -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel