Re: Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
On 2018-03-08 16:07, jasperkuijs...@gmail.com wrote: > On Wednesday, March 7, 2018 at 6:22:31 PM UTC+1, J. Kiszka wrote: >> On 2018-03-07 15:52, jasperkuijs...@gmail.com wrote: >>> On Wednesday, March 7, 2018 at 2:41:29 PM UTC+1, jasperk...@gmail.com wrote: Hi, I'm trying to set up a system with a bare-metal guest/inmate, with shared memory between Linux root cell and inmate, and an (IPI) doorbell interrupt from inmate to Linux. My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified to only have only one IVSHMEM device instead of two. Many things are working; gic-demo demo application runs correctly and from Linux I can access the shared memory via an uio device. However I'm stuck at generating the doorbell interrupt from inmate to Linux. # lsuio uio0: name=uio_ivshmem, version=0.0.1, events=0 map[0]: addr=0xFC10, size=4096 map[1]: addr=0x00080040, size=1048576 #cat /proc/interrupts ... 229: 0 0 0 GICv2 136 Edge uio_ivshmem ... I modified the gic-demo application slightly to generate a doorbell in the timer interrupt handler method. I attempt the generate the doorbell by writing the doorbell register in the ivshmem register region (map0). (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt) unsigned long* reg = (unsigned long*)0xFC10; *(reg + 3) = 1; // doorbell at offset 12byte On Linux side the doorbell is seen (events=1) only as soon as the inmate is destroyed, see log below. For the inmate, the gic-demo timer interrupt is only handled once, likely because there is another interrupt pending. # ./jailhouse cell start inmate0 Started cell "inmate0" # lsuio uio0: name=uio_ivshmem, version=0.0.1, events=0 map[0]: addr=0xFC10, size=4096 map[1]: addr=0x00080040, size=1048576 # ./jailhouse cell shutdown inmate0 Cell "inmate0" can be loaded # lsuio uio0: name=uio_ivshmem, version=0.0.1, events=0 map[0]: addr=0xFC10, size=4096 map[1]: addr=0x00080040, size=1048576 # ./jailhouse cell destroy inmate0 Closing cell "inmate0" Page pool usage after cell destruction: mem 43/996, remap 69/131072 [ 222.899818] ivshmem_handler irq: 229 [ 222.920548] Detected VIPT I-cache on CPU3 [ 222.920599] CPU3: Booted secondary processor [410fd034] [ 222.934074] Destroyed Jailhouse cell "inmate0" # lsuio uio0: name=uio_ivshmem, version=0.0.1, events=1 map[0]: addr=0xFC10, size=4096 map[1]: addr=0x00080040, size=1048576 Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info *dev_info)", which seems to be called only when the cpu is returned to Linux. It seems the doorbell is generated for the core I perform the register write on. How do I generate doorbells for other cores? Am I using a correct method to generate doorbells in the first place? Are there other options to do so? Kind regards, Jasper >>> >>> I want to add that since my original post I have changed flags for the >>> IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION. >> >> Thanks wrong. >> >>> >>> /* IVSHMEM register region for 00:00.0 */ { >>> .phys_start = 0xfc10, >>> .virt_start = 0xfc10, >>> .size = 4096, >>> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | >>> JAILHOUSE_MEM_COMM_REGION, >>> }, >>> >>> The result is that the inmate no longer hangs when attempting to access the >>> register space (I discovered that even reading IVPosition resulted in hang). >>> >>> However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in >>> line below: >>> >>> mmio_write32(reg + (i*8) + 3, 1); >>> >> >> Configuration of these setups is unfortunately a bit tricky. >> >> Please check first if you get ivshmem-net communication between root and >> non-root cell using the original setup. That is known to work. Then >> stepwise modify settings and test setups to your needs. >> >> The (a bit unfortunate) alternative is instrumenting the hypervisor /wrt >> ivshmem interactions to find out what happens from its POV. The code >> isn't very complex. >> >> Jan >> >> -- >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE >> Corporate Competence Center Embedded Linux > > Hi Jan, > > I managed to generate doorbells succesfully and they are picked up by the > root cell. > > I still use: > > unsigned int* reg = (unsigned int*)0xfc10; //address of /dev/uio0 map[0] > in root cell > mmio_w
Re: Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
On Wednesday, March 7, 2018 at 6:22:31 PM UTC+1, J. Kiszka wrote: > On 2018-03-07 15:52, jasperkuijs...@gmail.com wrote: > > On Wednesday, March 7, 2018 at 2:41:29 PM UTC+1, jasperk...@gmail.com wrote: > >> Hi, > >> > >> I'm trying to set up a system with a bare-metal guest/inmate, with shared > >> memory between Linux root cell and inmate, and an (IPI) doorbell interrupt > >> from inmate to Linux. > >> > >> My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified > >> to only have only one IVSHMEM device instead of two. > >> > >> Many things are working; gic-demo demo application runs correctly and from > >> Linux I can access the shared memory via an uio device. However I'm stuck > >> at generating the doorbell interrupt from inmate to Linux. > >> > >> # lsuio > >> uio0: name=uio_ivshmem, version=0.0.1, events=0 > >> map[0]: addr=0xFC10, size=4096 > >> map[1]: addr=0x00080040, size=1048576 > >> > >> #cat /proc/interrupts > >> ... > >> 229: 0 0 0 GICv2 136 Edge uio_ivshmem > >> ... > >> > >> I modified the gic-demo application slightly to generate a doorbell in the > >> timer interrupt handler method. I attempt the generate the doorbell by > >> writing the doorbell register in the ivshmem register region (map0). > >> (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt) > >> > >> unsigned long* reg = (unsigned long*)0xFC10; > >> *(reg + 3) = 1; // doorbell at offset 12byte > >> > >> On Linux side the doorbell is seen (events=1) only as soon as the inmate > >> is destroyed, see log below. For the inmate, the gic-demo timer interrupt > >> is only handled once, likely because there is another interrupt pending. > >> > >> # ./jailhouse cell start inmate0 > >> Started cell "inmate0" > >> # lsuio > >> uio0: name=uio_ivshmem, version=0.0.1, events=0 > >> map[0]: addr=0xFC10, size=4096 > >> map[1]: addr=0x00080040, size=1048576 > >> # ./jailhouse cell shutdown inmate0 > >> Cell "inmate0" can be loaded > >> # lsuio > >> uio0: name=uio_ivshmem, version=0.0.1, events=0 > >> map[0]: addr=0xFC10, size=4096 > >> map[1]: addr=0x00080040, size=1048576 > >> # ./jailhouse cell destroy inmate0 > >> Closing cell "inmate0" > >> Page pool usage after cell destruction: mem 43/996, remap 69/131072 > >> [ 222.899818] ivshmem_handler irq: 229 > >> [ 222.920548] Detected VIPT I-cache on CPU3 > >> [ 222.920599] CPU3: Booted secondary processor [410fd034] > >> [ 222.934074] Destroyed Jailhouse cell "inmate0" > >> # lsuio > >> uio0: name=uio_ivshmem, version=0.0.1, events=1 > >> map[0]: addr=0xFC10, size=4096 > >> map[1]: addr=0x00080040, size=1048576 > >> > >> Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk > >> statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info > >> *dev_info)", which seems to be called only when the cpu is returned to > >> Linux. > >> > >> It seems the doorbell is generated for the core I perform the register > >> write on. How do I generate doorbells for other cores? Am I using a > >> correct method to generate doorbells in the first place? Are there other > >> options to do so? > >> > >> Kind regards, Jasper > > > > I want to add that since my original post I have changed flags for the > > IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION. > > Thanks wrong. > > > > > /* IVSHMEM register region for 00:00.0 */ { > > .phys_start = 0xfc10, > > .virt_start = 0xfc10, > > .size = 4096, > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | > > JAILHOUSE_MEM_COMM_REGION, > > }, > > > > The result is that the inmate no longer hangs when attempting to access the > > register space (I discovered that even reading IVPosition resulted in hang). > > > > However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in > > line below: > > > > mmio_write32(reg + (i*8) + 3, 1); > > > > Configuration of these setups is unfortunately a bit tricky. > > Please check first if you get ivshmem-net communication between root and > non-root cell using the original setup. That is known to work. Then > stepwise modify settings and test setups to your needs. > > The (a bit unfortunate) alternative is instrumenting the hypervisor /wrt > ivshmem interactions to find out what happens from its POV. The code > isn't very complex. > > Jan > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux Hi Jan, I managed to generate doorbells succesfully and they are picked up by the root cell. I still use: unsigned int* reg = (unsigned int*)0xfc10; //address of /dev/uio0 map[0] in root cell mmio_write32(reg + 3, 1); However, I completely removed the memory region from the inmate
Re: Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
On 2018-03-07 15:52, jasperkuijs...@gmail.com wrote: > On Wednesday, March 7, 2018 at 2:41:29 PM UTC+1, jasperk...@gmail.com wrote: >> Hi, >> >> I'm trying to set up a system with a bare-metal guest/inmate, with shared >> memory between Linux root cell and inmate, and an (IPI) doorbell interrupt >> from inmate to Linux. >> >> My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified to >> only have only one IVSHMEM device instead of two. >> >> Many things are working; gic-demo demo application runs correctly and from >> Linux I can access the shared memory via an uio device. However I'm stuck at >> generating the doorbell interrupt from inmate to Linux. >> >> # lsuio >> uio0: name=uio_ivshmem, version=0.0.1, events=0 >> map[0]: addr=0xFC10, size=4096 >> map[1]: addr=0x00080040, size=1048576 >> >> #cat /proc/interrupts >> ... >> 229: 0 0 0 GICv2 136 Edge uio_ivshmem >> ... >> >> I modified the gic-demo application slightly to generate a doorbell in the >> timer interrupt handler method. I attempt the generate the doorbell by >> writing the doorbell register in the ivshmem register region (map0). >> (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt) >> >> unsigned long* reg = (unsigned long*)0xFC10; >> *(reg + 3) = 1; // doorbell at offset 12byte >> >> On Linux side the doorbell is seen (events=1) only as soon as the inmate is >> destroyed, see log below. For the inmate, the gic-demo timer interrupt is >> only handled once, likely because there is another interrupt pending. >> >> # ./jailhouse cell start inmate0 >> Started cell "inmate0" >> # lsuio >> uio0: name=uio_ivshmem, version=0.0.1, events=0 >> map[0]: addr=0xFC10, size=4096 >> map[1]: addr=0x00080040, size=1048576 >> # ./jailhouse cell shutdown inmate0 >> Cell "inmate0" can be loaded >> # lsuio >> uio0: name=uio_ivshmem, version=0.0.1, events=0 >> map[0]: addr=0xFC10, size=4096 >> map[1]: addr=0x00080040, size=1048576 >> # ./jailhouse cell destroy inmate0 >> Closing cell "inmate0" >> Page pool usage after cell destruction: mem 43/996, remap 69/131072 >> [ 222.899818] ivshmem_handler irq: 229 >> [ 222.920548] Detected VIPT I-cache on CPU3 >> [ 222.920599] CPU3: Booted secondary processor [410fd034] >> [ 222.934074] Destroyed Jailhouse cell "inmate0" >> # lsuio >> uio0: name=uio_ivshmem, version=0.0.1, events=1 >> map[0]: addr=0xFC10, size=4096 >> map[1]: addr=0x00080040, size=1048576 >> >> Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk >> statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info >> *dev_info)", which seems to be called only when the cpu is returned to Linux. >> >> It seems the doorbell is generated for the core I perform the register write >> on. How do I generate doorbells for other cores? Am I using a correct method >> to generate doorbells in the first place? Are there other options to do so? >> >> Kind regards, Jasper > > I want to add that since my original post I have changed flags for the > IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION. Thanks wrong. > > /* IVSHMEM register region for 00:00.0 */ { > .phys_start = 0xfc10, > .virt_start = 0xfc10, > .size = 4096, > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | > JAILHOUSE_MEM_COMM_REGION, > }, > > The result is that the inmate no longer hangs when attempting to access the > register space (I discovered that even reading IVPosition resulted in hang). > > However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in > line below: > > mmio_write32(reg + (i*8) + 3, 1); > Configuration of these setups is unfortunately a bit tricky. Please check first if you get ivshmem-net communication between root and non-root cell using the original setup. That is known to work. Then stepwise modify settings and test setups to your needs. The (a bit unfortunate) alternative is instrumenting the hypervisor /wrt ivshmem interactions to find out what happens from its POV. The code isn't very complex. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
On Wednesday, March 7, 2018 at 3:52:07 PM UTC+1, jasperk...@gmail.com wrote: > On Wednesday, March 7, 2018 at 2:41:29 PM UTC+1, jasperk...@gmail.com wrote: > > Hi, > > > > I'm trying to set up a system with a bare-metal guest/inmate, with shared > > memory between Linux root cell and inmate, and an (IPI) doorbell interrupt > > from inmate to Linux. > > > > My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified > > to only have only one IVSHMEM device instead of two. > > > > Many things are working; gic-demo demo application runs correctly and from > > Linux I can access the shared memory via an uio device. However I'm stuck > > at generating the doorbell interrupt from inmate to Linux. > > > > # lsuio > > uio0: name=uio_ivshmem, version=0.0.1, events=0 > > map[0]: addr=0xFC10, size=4096 > > map[1]: addr=0x00080040, size=1048576 > > > > #cat /proc/interrupts > > ... > > 229: 0 0 0 GICv2 136 Edge uio_ivshmem > > ... > > > > I modified the gic-demo application slightly to generate a doorbell in the > > timer interrupt handler method. I attempt the generate the doorbell by > > writing the doorbell register in the ivshmem register region (map0). > > (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt) > > > > unsigned long* reg = (unsigned long*)0xFC10; > > *(reg + 3) = 1; // doorbell at offset 12byte > > > > On Linux side the doorbell is seen (events=1) only as soon as the inmate is > > destroyed, see log below. For the inmate, the gic-demo timer interrupt is > > only handled once, likely because there is another interrupt pending. > > > > # ./jailhouse cell start inmate0 > > Started cell "inmate0" > > # lsuio > > uio0: name=uio_ivshmem, version=0.0.1, events=0 > > map[0]: addr=0xFC10, size=4096 > > map[1]: addr=0x00080040, size=1048576 > > # ./jailhouse cell shutdown inmate0 > > Cell "inmate0" can be loaded > > # lsuio > > uio0: name=uio_ivshmem, version=0.0.1, events=0 > > map[0]: addr=0xFC10, size=4096 > > map[1]: addr=0x00080040, size=1048576 > > # ./jailhouse cell destroy inmate0 > > Closing cell "inmate0" > > Page pool usage after cell destruction: mem 43/996, remap 69/131072 > > [ 222.899818] ivshmem_handler irq: 229 > > [ 222.920548] Detected VIPT I-cache on CPU3 > > [ 222.920599] CPU3: Booted secondary processor [410fd034] > > [ 222.934074] Destroyed Jailhouse cell "inmate0" > > # lsuio > > uio0: name=uio_ivshmem, version=0.0.1, events=1 > > map[0]: addr=0xFC10, size=4096 > > map[1]: addr=0x00080040, size=1048576 > > > > Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk > > statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info > > *dev_info)", which seems to be called only when the cpu is returned to > > Linux. > > > > It seems the doorbell is generated for the core I perform the register > > write on. How do I generate doorbells for other cores? Am I using a correct > > method to generate doorbells in the first place? Are there other options to > > do so? > > > > Kind regards, Jasper > > I want to add that since my original post I have changed flags for the > IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION. > > /* IVSHMEM register region for 00:00.0 */ { > .phys_start = 0xfc10, > .virt_start = 0xfc10, > .size = 4096, > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | > JAILHOUSE_MEM_COMM_REGION, > }, > > The result is that the inmate no longer hangs when attempting to access the > register space (I discovered that even reading IVPosition resulted in hang). > > However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in > line below: > > mmio_write32(reg + (i*8) + 3, 1); Some additional comments.. Fixed my mistake in inmate0.c: "unsigned int" instead of "unsigned long" to access correct offsets via pointer aritmetics.. sorry. But that still does not give working doorbells. Following: https://github.com/siemens/jailhouse/blob/master/Documentation/inter-cell-communication.txt I also attempt to mmio_write the LSTATE register but no success. -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Doorbell interrupt from bare-metal inmate to Linux via uoi_ivshmem (ARM64 / ZCU102)
On Wednesday, March 7, 2018 at 2:41:29 PM UTC+1, jasperk...@gmail.com wrote: > Hi, > > I'm trying to set up a system with a bare-metal guest/inmate, with shared > memory between Linux root cell and inmate, and an (IPI) doorbell interrupt > from inmate to Linux. > > My configs are based on zynqmp-zcu102 (root), gic-demo (inmate). Modified to > only have only one IVSHMEM device instead of two. > > Many things are working; gic-demo demo application runs correctly and from > Linux I can access the shared memory via an uio device. However I'm stuck at > generating the doorbell interrupt from inmate to Linux. > > # lsuio > uio0: name=uio_ivshmem, version=0.0.1, events=0 > map[0]: addr=0xFC10, size=4096 > map[1]: addr=0x00080040, size=1048576 > > #cat /proc/interrupts > ... > 229: 0 0 0 GICv2 136 Edge uio_ivshmem > ... > > I modified the gic-demo application slightly to generate a doorbell in the > timer interrupt handler method. I attempt the generate the doorbell by > writing the doorbell register in the ivshmem register region (map0). > (https://github.com/henning-schild-work/ivshmem-guest-code/blob/master/device_spec.txt) > > unsigned long* reg = (unsigned long*)0xFC10; > *(reg + 3) = 1; // doorbell at offset 12byte > > On Linux side the doorbell is seen (events=1) only as soon as the inmate is > destroyed, see log below. For the inmate, the gic-demo timer interrupt is > only handled once, likely because there is another interrupt pending. > > # ./jailhouse cell start inmate0 > Started cell "inmate0" > # lsuio > uio0: name=uio_ivshmem, version=0.0.1, events=0 > map[0]: addr=0xFC10, size=4096 > map[1]: addr=0x00080040, size=1048576 > # ./jailhouse cell shutdown inmate0 > Cell "inmate0" can be loaded > # lsuio > uio0: name=uio_ivshmem, version=0.0.1, events=0 > map[0]: addr=0xFC10, size=4096 > map[1]: addr=0x00080040, size=1048576 > # ./jailhouse cell destroy inmate0 > Closing cell "inmate0" > Page pool usage after cell destruction: mem 43/996, remap 69/131072 > [ 222.899818] ivshmem_handler irq: 229 > [ 222.920548] Detected VIPT I-cache on CPU3 > [ 222.920599] CPU3: Booted secondary processor [410fd034] > [ 222.934074] Destroyed Jailhouse cell "inmate0" > # lsuio > uio0: name=uio_ivshmem, version=0.0.1, events=1 > map[0]: addr=0xFC10, size=4096 > map[1]: addr=0x00080040, size=1048576 > > Message "[ 222.899818] ivshmem_handler irq: 229" originates from a printk > statement I added to "irqreturn_t ivshmem_handler(int irq, struct uio_info > *dev_info)", which seems to be called only when the cpu is returned to Linux. > > It seems the doorbell is generated for the core I perform the register write > on. How do I generate doorbells for other cores? Am I using a correct method > to generate doorbells in the first place? Are there other options to do so? > > Kind regards, Jasper I want to add that since my original post I have changed flags for the IVSHMEM device register region to include JAILHOUSE_MEM_COMM_REGION. /* IVSHMEM register region for 00:00.0 */ { .phys_start = 0xfc10, .virt_start = 0xfc10, .size = 4096, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_COMM_REGION, }, The result is that the inmate no longer hangs when attempting to access the register space (I discovered that even reading IVPosition resulted in hang). However, Linux still does not see the doorbell. I have tried i=0,1,2,3 in line below: mmio_write32(reg + (i*8) + 3, 1); -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.