Jailhouse + Erika on qemu for arm64

[Giovani Gracioli]

Just wondering if someone has tried jailhouse and Erika 3 on qemu for arm64? Do 
they run fine?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: SGI among cells

[Giovani Gracioli]

Hello,

I realized that the IPI number (15) is not enabled in Erika. However, when I 
use an ivshmem, its interrupt is enabled. For instance, the output of the 
following printks:

printk("is IRQ 15 enabled? %d\n", osEE_gic_v2_is_enabled_irq(15));

printk("is IRQ %d enabled? %d\n", IVSHMEM_IRQ, 
osEE_gic_v2_is_enabled_irq(IVSHMEM_IRQ));

Are:

Level: is IRQ 15 enabled? 0

Level: is IRQ 145 enabled? 1

Both ISRs are created at the same way in Erika. So my conclusion is that 
Jailhouse is not enabling it for the cell.  

I tried to enable it in the .pin_bitmap field of the irqchips config, but no 
success:

.irqchips = {
/* GIC */ {
.address = 0xf901,
.pin_base = 32,
.pin_bitmap = {
1 << 15 | 1 << (54 - 32),
0,
0,
(1 << (144 - 128)) | (1 << (145 - 128)) | (1 << (146 - 128)) | 
(1 << (147 - 128)) | (1 << (148 - 128)) | (1 << (149 - 128)),   
 
},  
},
},

Any suggestion on how to enable the SGI 15 for the cell?

> gic-demo is able to receive the IPI. So the problem is within Erika interrupt 
> configuration.
> 
> > Hi,
> > 
> > On 9/5/18 8:28 PM, Giovani Gracioli wrote:
> > > Yes, I did enable with Erika API.
> > 
> > No, I meant if you tried to receive the SGI via Jailhouse inmates. I
> > don't know Erika internals.
> > 
> > Some test like:
> > 
> > diff --git a/inmates/demos/arm/gic-demo.c b/inmates/demos/arm/gic-demo.c
> > index b2246deb..86be1b1e 100644
> > --- a/inmates/demos/arm/gic-demo.c
> > +++ b/inmates/demos/arm/gic-demo.c
> > @@ -35,6 +35,8 @@ static void handle_IRQ(unsigned int irqn)
> > 
> > if (irqn != TIMER_IRQ)
> > return;
> > +   else
> > +   printk("Hey, there's some IRQ %u\n", irqn);
> > 
> > delta = timer_get_ticks() - expected_ticks;
> > if (delta < min_delta)
> > @@ -59,6 +61,7 @@ void inmate_main(void)
> > printk("Initializing the GIC...\n");
> > gic_setup(handle_IRQ);
> > gic_enable_irq(TIMER_IRQ);
> > +   gic_enable_irq(15);
> > 
> > printk("Initializing the timer...\n");
> > ticks_per_beat = timer_get_frequency() / BEATS_PER_SEC;
> > 
> > 
> > 
> > Thanks
> >   Ralf
> > 
> > > 
> > > s |= CreateTask( _ivshmem, OSEE_TASK_TYPE_ISR2, handle_IRQ, 1U, 1U, 
> > > 1U, OSEE_SYSTEM_STACK );
> > > 
> > > SetISR2Source(isr_ivshmem, i);
> > > gic_enable_irq(i); //this is actually from jailhouse api
> > > 
> > > Where "i" is the interrupt number. I tried having "i" with 15. Maybe 
> > > something in Erika is different. I will try with the gic-demo as well.
> > > 
> > >>> Hi,
> > >>>
> > >>> I am playing around IPI among non-root cells (I know it breaks the 
> > >>> isolation among cells, but I would like to understand the steps). I 
> > >>> forced a call to irqchip_set_pending from CPU 3 to CPU 2 in the 
> > >>> gic_handle_sgir_write() function:
> > >>>
> > >>> irqchip_set_pending(per_cpu(2), sgi->id);
> > >>>
> > >>> sgi->id is 15. I can see that CPU 2 receives the IPI:
> > >>>
> > >>> CPU 2 received an SGI 0
> > >>> irqchip_inject_pending irq_id = 15, sender 3
> > >>> gicv2_inject_irq() sender = 3, irq_id = 15
> > >>> CPU 2 writing 268438543 to lr reg (first_free = 0)
> > >>>
> > >>> However, the non-root cell does not receive the IPI (I am running Erika 
> > >>> on it).
> > >>
> > >> Did you try to receive the SGI in the gic-demo? You only have to
> > >> instrument handle_IRQ and enable the interrupt.
> > >>
> > >>   Ralf
> > >>
> > >>>
> > >>> I do not know in which IRQ number the SGI ID 15 is mapped to. What is 
> > >>> the relation of pin_base and pin_bitmap from the config file with SGIs? 
> > >>> What value should I write in the irqchips config?
> > >>>
> > >>> Best
> > >>> Giovani
> > >>>
> > >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: SGI among cells

[Giovani Gracioli]

gic-demo is able to receive the IPI. So the problem is within Erika interrupt 
configuration.

> Hi,
> 
> On 9/5/18 8:28 PM, Giovani Gracioli wrote:
> > Yes, I did enable with Erika API.
> 
> No, I meant if you tried to receive the SGI via Jailhouse inmates. I
> don't know Erika internals.
> 
> Some test like:
> 
> diff --git a/inmates/demos/arm/gic-demo.c b/inmates/demos/arm/gic-demo.c
> index b2246deb..86be1b1e 100644
> --- a/inmates/demos/arm/gic-demo.c
> +++ b/inmates/demos/arm/gic-demo.c
> @@ -35,6 +35,8 @@ static void handle_IRQ(unsigned int irqn)
> 
> if (irqn != TIMER_IRQ)
> return;
> +   else
> +   printk("Hey, there's some IRQ %u\n", irqn);
> 
> delta = timer_get_ticks() - expected_ticks;
> if (delta < min_delta)
> @@ -59,6 +61,7 @@ void inmate_main(void)
> printk("Initializing the GIC...\n");
> gic_setup(handle_IRQ);
> gic_enable_irq(TIMER_IRQ);
> +   gic_enable_irq(15);
> 
> printk("Initializing the timer...\n");
> ticks_per_beat = timer_get_frequency() / BEATS_PER_SEC;
> 
> 
> 
> Thanks
>   Ralf
> 
> > 
> > s |= CreateTask( _ivshmem, OSEE_TASK_TYPE_ISR2, handle_IRQ, 1U, 1U, 1U, 
> > OSEE_SYSTEM_STACK );
> > 
> > SetISR2Source(isr_ivshmem, i);
> > gic_enable_irq(i); //this is actually from jailhouse api
> > 
> > Where "i" is the interrupt number. I tried having "i" with 15. Maybe 
> > something in Erika is different. I will try with the gic-demo as well.
> > 
> >>> Hi,
> >>>
> >>> I am playing around IPI among non-root cells (I know it breaks the 
> >>> isolation among cells, but I would like to understand the steps). I 
> >>> forced a call to irqchip_set_pending from CPU 3 to CPU 2 in the 
> >>> gic_handle_sgir_write() function:
> >>>
> >>> irqchip_set_pending(per_cpu(2), sgi->id);
> >>>
> >>> sgi->id is 15. I can see that CPU 2 receives the IPI:
> >>>
> >>> CPU 2 received an SGI 0
> >>> irqchip_inject_pending irq_id = 15, sender 3
> >>> gicv2_inject_irq() sender = 3, irq_id = 15
> >>> CPU 2 writing 268438543 to lr reg (first_free = 0)
> >>>
> >>> However, the non-root cell does not receive the IPI (I am running Erika 
> >>> on it).
> >>
> >> Did you try to receive the SGI in the gic-demo? You only have to
> >> instrument handle_IRQ and enable the interrupt.
> >>
> >>   Ralf
> >>
> >>>
> >>> I do not know in which IRQ number the SGI ID 15 is mapped to. What is the 
> >>> relation of pin_base and pin_bitmap from the config file with SGIs? What 
> >>> value should I write in the irqchips config?
> >>>
> >>> Best
> >>> Giovani
> >>>
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: SGI among cells

[Giovani Gracioli]

Yes, I did enable with Erika API.

s |= CreateTask( _ivshmem, OSEE_TASK_TYPE_ISR2, handle_IRQ, 1U, 1U, 1U, 
OSEE_SYSTEM_STACK );

SetISR2Source(isr_ivshmem, i);
gic_enable_irq(i); //this is actually from jailhouse api

Where "i" is the interrupt number. I tried having "i" with 15. Maybe something 
in Erika is different. I will try with the gic-demo as well.

> > Hi,
> > 
> > I am playing around IPI among non-root cells (I know it breaks the 
> > isolation among cells, but I would like to understand the steps). I forced 
> > a call to irqchip_set_pending from CPU 3 to CPU 2 in the 
> > gic_handle_sgir_write() function:
> > 
> > irqchip_set_pending(per_cpu(2), sgi->id);
> > 
> > sgi->id is 15. I can see that CPU 2 receives the IPI:
> > 
> > CPU 2 received an SGI 0
> > irqchip_inject_pending irq_id = 15, sender 3
> > gicv2_inject_irq() sender = 3, irq_id = 15
> > CPU 2 writing 268438543 to lr reg (first_free = 0)
> > 
> > However, the non-root cell does not receive the IPI (I am running Erika on 
> > it).
> 
> Did you try to receive the SGI in the gic-demo? You only have to
> instrument handle_IRQ and enable the interrupt.
> 
>   Ralf
> 
> > 
> > I do not know in which IRQ number the SGI ID 15 is mapped to. What is the 
> > relation of pin_base and pin_bitmap from the config file with SGIs? What 
> > value should I write in the irqchips config?
> > 
> > Best
> > Giovani
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

SGI among cells

[Giovani Gracioli]

Hi,

I am playing around IPI among non-root cells (I know it breaks the isolation 
among cells, but I would like to understand the steps). I forced a call to 
irqchip_set_pending from CPU 3 to CPU 2 in the gic_handle_sgir_write() function:

irqchip_set_pending(per_cpu(2), sgi->id);

sgi->id is 15. I can see that CPU 2 receives the IPI:

CPU 2 received an SGI 0
irqchip_inject_pending irq_id = 15, sender 3
gicv2_inject_irq() sender = 3, irq_id = 15
CPU 2 writing 268438543 to lr reg (first_free = 0)

However, the non-root cell does not receive the IPI (I am running Erika on it).

I do not know in which IRQ number the SGI ID 15 is mapped to. What is the 
relation of pin_base and pin_bitmap from the config file with SGIs? What value 
should I write in the irqchips config?

Best
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Our work around was forwarding the smc call to the firmware, using the original 
code in the Linux firmware.c file. It is not the best solution, but it works. 
Maybe the next step is to refine it and have a better integration with 
Jailhouse.

Files are attached:

- hypervisor/arch/arm64/traps.c
- hypervisor/arch/arm64/smccc-call.S
- hypervisor/arch/arm64/include/asm/arm-smccc.h

Giovani

> > The only doc that I found is the comments in the source file. Most of the 
> > important functions are implemented here: 
> > https://github.com/Xilinx/linux-xlnx/blob/master/drivers/firmware/xilinx/zynqmp/firmware.c
> > 
> > I could map the function call flow as follows:
> > 
> > zynqmp_r5_remoteproc driver has function pointers to some functions defined 
> > in the firmware.c file.
> > 
> > Specifically for the R5 status, it calls zynqmp_pm_get_node_status, which 
> > creates an u32 array called ret_payload and calls zynqmp_invoke_pm:
> > 
> > zynqmp_invoke_pm(PM_GET_NODE_STATUS, node, 0, 0, 0, ret_payload)
> > 
> > PM_GET_NODE_STATUS is defined as 3, node is the RPU (R5) ID. The function 
> > uses the ret_payload to populate the arguments it received from the driver:
> > 
> > if (ret_payload[0] == XST_PM_SUCCESS) {
> >  *status = ret_payload[1];
> >  if (requirements)
> > *requirements = ret_payload[2];
> >  if (usage)
> > *usage = ret_payload[3];
> > }
> > 
> >   * zynqmp_pm_invoke_fn - Invoke the system-level platform management layer
> >   *  caller function depending on the configuration
> >   * @pm_api_id: Requested PM-API call
> >   * @arg0:  Argument 0 to requested PM-API call
> >   * @arg1:  Argument 1 to requested PM-API call
> >   * @arg2:  Argument 2 to requested PM-API call
> >   * @arg3:  Argument 3 to requested PM-API call
> >   * @ret_payload:   Returned value array
> >   *
> >   * Return: Returns status, either success or error+reason
> >   *
> >   * Invoke platform management function for SMC or HVC call, depending on
> >   * configuration
> >   * Following SMC Calling Convention (SMCCC) for SMC64:
> >   * Pm Function Identifier,
> >   * PM_SIP_SVC + PM_API_ID =
> >   * ((SMC_TYPE_FAST << FUNCID_TYPE_SHIFT)
> >   * ((SMC_64) << FUNCID_CC_SHIFT)
> >   * ((SIP_START) << FUNCID_OEN_SHIFT)
> >   * ((PM_API_ID) & FUNCID_NUM_MASK))
> >   *
> >   * PM_SIP_SVC  - Registered ZynqMP SIP Service Call
> >   * PM_API_ID   - Platform Management API ID
> >   */
> > int zynqmp_pm_invoke_fn(u32 pm_api_id, u32 arg0, u32 arg1, u32 arg2, u32 
> > arg3, u32 *ret_payload)
> > {
> > /*
> >  * Added SIP service call Function Identifier
> >  * Make sure to stay in x0 register
> >  */
> > u64 smc_arg[4];
> > 
> > smc_arg[0] = PM_SIP_SVC | pm_api_id;
> > smc_arg[1] = ((u64)arg1 << 32) | arg0;
> > smc_arg[2] = ((u64)arg3 << 32) | arg2;
> > 
> > return do_fw_call(smc_arg[0], smc_arg[1], smc_arg[2], ret_payload);
> > }
> > 
> > PM_SIP_SVC is defined as 0xC200 and pm_api_id is 3. arg0 is the RPU ID, 
> > arg1, arg2, and arg3 are zeros in this case. do_fw_call is a pointer to 
> > do_fw_call_smc:
> > 
> > /**
> >   * do_fw_call_smc - Call system-level platform management layer (SMC)
> >   * @arg0:  Argument 0 to SMC call
> >   * @arg1:  Argument 1 to SMC call
> >   * @arg2:  Argument 2 to SMC call
> >   * @ret_payload:   Returned value array
> >   *
> >   * Return: Returns status, either success or error+reason
> >   *
> >   * Invoke platform management function via SMC call (no hypervisor present)
> >   */
> > 
> > This function calls arm_smccc_smc:
> > 
> > struct arm_smccc_res res;
> > arm_smccc_smc(arg0, arg1, arg2, 0, 0, 0, 0, 0, );
> > 
> > arm_smccc_smc is defined at arc/arm64/kernel/smccc-call.S (the assembly 
> > code I pasted previously).
> > 
> > IS_SIP_64 from Jailhouse v0.9.1 (the one I am using right now) tests 
> > whether reg[0] >> 24 is 0xC2, which is true in this test, and not handled. 
> > It seems that all requests from the r5 driver have the format 0xC0X.
> 
> That could be power-management calls to the ATF. You may find the other 
> side here: https://github.com/ARM-software/arm-trusted-firmware/
> 
> Unfortunately, that's all platform-specific. So every whitelisting we 
> may implement will require platform logic. The SMCCC [1] may at least 
> help with identifying calls and forwarding parameters if we allow them. 
> But when it comes to things like "power down co-processor X", and X is 
> encoded in some parameter, there will be no way around filtering also on 
> those values in order to possibly split accesses.
> 
> In any case, maybe a generic SMCCC-based request filter that can be 
> configured via the cell config would already move us forward and catch 
> some 80% of the use cases...
> 
> Jan
> 
> [1] 
> http://infocenter.arm.com/help/topic/com.arm.doc.den0028b/ARM_DEN0028B_SMC_Calling_Convention.pdf
> 
> -- 
> Siemens AG, Corporate 

Re: ARM Inter-cell Data Handling Basics

[Giovani Gracioli]

> > 
> > Hello,
> > 
> > I am interested in inter-cell communication between an ARM linux root cell 
> > and the baremetal inmate. I'd like the baremetal to process I/O and write 
> > data to a buffer in RAM. On the linux side, I'd like to know when that 
> > buffer was full and copy it to a file on the filesystem or a removable 
> > storage device. Is ivshem the way I need to go? Looking at the examples, 
> > I'm not understanding the role or need of the PCI device.
> 
> ivshmem is the way to go, yes. It is exposed to the guest as virtual PCI 
> device. If you search the list you will find some references to 
> (unfortunately unfinished) ports of the ivshmem-demo for bare-metal 
> inmates to ARM. You should derive from that. On the linux side, there is 
> the uio driver as referenced in the docs.
> 
> ivshmem gives you that shared memory region and a mechanism to kick the 
> remote side via an interrupt. That should allow you to build your 
> preferred application-specific communication on top.
> 
> > 
> > I am a total newb at Jailhouse, just learned about its existence last week.
> > 
> 
> A lot to explore then ;).
> 
> Jan

Hi Arob,

Look for some posts I've published. I could setup the ivshmem on arm64. I 
posted a non-official patch that adds support for ivshmem on arm64 and examples 
of configuration files (to properly configure the memory regions).

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Hello all,

I tracked the calls in the zynqmp_r5_remoteproc driver. It is not using hvc, 
but smc instead.

The driver ended up calling do_fw_call_smc (in 
drivers/firmware/xilinx/zynqmp/firmware.c), which then calls the assembly code 
I posted before but with another parameter:

/*
 * void arm_smccc_smc(unsigned long a0, unsigned long a1, unsigned long a2,
 *unsigned long a3, unsigned long a4, unsigned long a5,
 *unsigned long a6, unsigned long a7, struct arm_smccc_res *res,
 *struct arm_smccc_quirk *quirk)
 */
ENTRY(__arm_smccc_smc)
SMCCC   smc
ENDPROC(__arm_smccc_smc)

This is handled in Jailhouse by handle_smc in hypervisor/arch/arm64/traps.c. In 
that function, the request is not handled in this else if statement:

} else if (IS_SIP_32(regs[0]) || IS_SIP_64(regs[0])) {
/* This can be ignored */
printk("handle_smc() 3 - regs[0] = %d\n", regs[0]);
regs[0] = SIP_NOT_SUPPORTED;
} else {

I added the printk to check the function call flow. 

So my question to the main developers is how can I integrate the handling of 
this call in Jailhouse? What would be the best way to proceed?

Best
Giovani

> Hi Peng,
> 
> Thank you for the answer.
> 
> I have inserted a printk in the handle_hvc to check whether it is called when 
> the driver runs. I could not see any output. 
> 
> Any other function is executed before and acts as a filter, which does not 
> call handle_hvc?
> 
> Giovani
> 
> > > 
> > > On 2018-08-13 15:15, Giovani Gracioli wrote:
> > > > There is only the root cell. The problem does not happen when the root 
> > > > cell is
> > > not loaded.
> > > >
> > > > My understanding is that the zynqmp_r5_remoteproc driver issues and hvc
> > > instruction to get information whether the R5 is running or not. This 
> > > instruction
> > > is somehow intercepted by Jailhouse and does not return the expected (the 
> > > R5 is
> > > actually running).
> > > >
> > > 
> > > OK, starting to understand that assembly fragment you posted. But I'm
> > > confused: Even without Jailhouse, that hvc should end up being rejected 
> > > by the
> > > hyp stub of the mainline kernel and never get through to a provider 
> > > outside of
> > > the kernel. Or are you running some downstream Xilinx kernel?
> > > 
> > > In any case, that call won't work under Jailhouse. But we first of all 
> > > need to
> > > understand where it should be handled (ATF?) in order to clarify if 
> > > replacing *hvc
> > > with *smc can help.
> > 
> > Currently handle_hvc only handles jailhouse driver hvc call. So if other
> > Linux components use hvc, jailhouse will report failure. Need to add support
> > in handle_hvc/hypercall.
> > 
> > Regards,
> > Peng
> > 
> > > 
> > > Jan
> > > 
> > > >>>> Hi,
> > > >>>>
> > > >>>> On 08/09/2018 06:35 PM, Giovani Gracioli wrote:
> > > >>>>> Thanks Ralf.
> > > >>>>>
> > > >>>>> I updated to the latest version, 0.9.1, and the original problem is 
> > > >>>>> gone.
> > > >>>>
> > > >>>> Great.
> > > >>>>
> > > >>>>>
> > > >>>>> Then I got an unhandled data write:
> > > >>>>
> > > >>>> One bug down, the next shows up ;-)
> > > >>>>
> > > >>>>>
> > > >>>>> Unhandled data write at 0xffe01800(1)
> > > >>>>
> > > >>>> That's weird, I'm wondering why this didn't happen with the old
> > > >>>> Jailhouse version.
> > > >>>>
> > > >>>>>
> > > >>>
> > > >>> It did not happen because in version 0.8 I got the exception 0x17, 
> > > >>> which is
> > > solved by version 0.9.1.
> > > >>>
> > > >>>>> FATAL: unhandled trap (exception class 0x24)
> > > >>>>>
> > > >>>>> To solve this, I added the TCM memory region 0xffe0 to the root 
> > > >>>>> cell
> > > config. This memory is used by the R5 to boot its image:
> > > >>>>>
> > > >>>>> /* TCM region for the R5 */ {
> > > >>>>> .phys_start = 0xffe0,
> > > >>>>

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Hi Peng,

Thank you for the answer.

I have inserted a printk in the handle_hvc to check whether it is called when 
the driver runs. I could not see any output. 

Any other function is executed before and acts as a filter, which does not call 
handle_hvc?

Giovani

> > 
> > On 2018-08-13 15:15, Giovani Gracioli wrote:
> > > There is only the root cell. The problem does not happen when the root 
> > > cell is
> > not loaded.
> > >
> > > My understanding is that the zynqmp_r5_remoteproc driver issues and hvc
> > instruction to get information whether the R5 is running or not. This 
> > instruction
> > is somehow intercepted by Jailhouse and does not return the expected (the 
> > R5 is
> > actually running).
> > >
> > 
> > OK, starting to understand that assembly fragment you posted. But I'm
> > confused: Even without Jailhouse, that hvc should end up being rejected by 
> > the
> > hyp stub of the mainline kernel and never get through to a provider outside 
> > of
> > the kernel. Or are you running some downstream Xilinx kernel?
> > 
> > In any case, that call won't work under Jailhouse. But we first of all need 
> > to
> > understand where it should be handled (ATF?) in order to clarify if 
> > replacing *hvc
> > with *smc can help.
> 
> Currently handle_hvc only handles jailhouse driver hvc call. So if other
> Linux components use hvc, jailhouse will report failure. Need to add support
> in handle_hvc/hypercall.
> 
> Regards,
> Peng
> 
> > 
> > Jan
> > 
> > >>>> Hi,
> > >>>>
> > >>>> On 08/09/2018 06:35 PM, Giovani Gracioli wrote:
> > >>>>> Thanks Ralf.
> > >>>>>
> > >>>>> I updated to the latest version, 0.9.1, and the original problem is 
> > >>>>> gone.
> > >>>>
> > >>>> Great.
> > >>>>
> > >>>>>
> > >>>>> Then I got an unhandled data write:
> > >>>>
> > >>>> One bug down, the next shows up ;-)
> > >>>>
> > >>>>>
> > >>>>> Unhandled data write at 0xffe01800(1)
> > >>>>
> > >>>> That's weird, I'm wondering why this didn't happen with the old
> > >>>> Jailhouse version.
> > >>>>
> > >>>>>
> > >>>
> > >>> It did not happen because in version 0.8 I got the exception 0x17, 
> > >>> which is
> > solved by version 0.9.1.
> > >>>
> > >>>>> FATAL: unhandled trap (exception class 0x24)
> > >>>>>
> > >>>>> To solve this, I added the TCM memory region 0xffe0 to the root 
> > >>>>> cell
> > config. This memory is used by the R5 to boot its image:
> > >>>>>
> > >>>>> /* TCM region for the R5 */ {
> > >>>>>   .phys_start = 0xffe0,
> > >>>>>   .virt_start = 0xffe0,
> > >>>>>   .size =   0xC,
> > >>>>>   .flags = JAILHOUSE_MEM_READ |
> > JAILHOUSE_MEM_WRITE |
> > >>>>> JAILHOUSE_MEM_EXECUTE,
> > >>>>
> > >>>> Is it IO? JAILHOUSE_MEM_IO?
> > >>>>
> > >>>
> > >>> The TCM memory is not I/O. It is the memory the R5s use.
> > >>>
> > >>>
> > >>>>> },
> > >>>>>
> > >>>>> However, when I use the zynqmp_r5_remoteproc driver with the
> > jailhouse, the driver has an error, and the code is not ran:
> > >>>>>
> > >>>>> remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> > >>>>> remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size
> > >>>>> 815824 Failed to get RPU node status.
> > >>>>> Failed to get RPU node status.
> > >>>>
> > >>>> Hmm, don't know what's exactly behind that memory area or how R5
> > >>>> works, but I'd guess it's iomem. Try to add JAILHOUSE_MEM_IO to the
> > region.
> > >>>
> > >>> Does not work with JAILHOUSE_MEM_IO either. I got the following:
> > >>>
> > >>> remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> > >>> remoteproc remotepro

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Hi Jan,

Yes I am using the latest Xilinx kernel because I need some drivers only 
available on it.

> On 2018-08-13 15:15, Giovani Gracioli wrote:
> > There is only the root cell. The problem does not happen when the root cell 
> > is not loaded.
> > 
> > My understanding is that the zynqmp_r5_remoteproc driver issues and hvc 
> > instruction to get information whether the R5 is running or not. This 
> > instruction is somehow intercepted by Jailhouse and does not return the 
> > expected (the R5 is actually running).
> > 
> 
> OK, starting to understand that assembly fragment you posted. But I'm 
> confused: Even without Jailhouse, that hvc should end up being rejected 
> by the hyp stub of the mainline kernel and never get through to a 
> provider outside of the kernel. Or are you running some downstream 
> Xilinx kernel?
> 
> In any case, that call won't work under Jailhouse. But we first of all 
> need to understand where it should be handled (ATF?) in order to clarify 
> if replacing *hvc with *smc can help.
> 
> Jan
> 
> >>>> Hi,
> >>>>
> >>>> On 08/09/2018 06:35 PM, Giovani Gracioli wrote:
> >>>>> Thanks Ralf.
> >>>>>
> >>>>> I updated to the latest version, 0.9.1, and the original problem is 
> >>>>> gone.
> >>>>
> >>>> Great.
> >>>>
> >>>>>
> >>>>> Then I got an unhandled data write:
> >>>>
> >>>> One bug down, the next shows up ;-)
> >>>>
> >>>>>
> >>>>> Unhandled data write at 0xffe01800(1)
> >>>>
> >>>> That's weird, I'm wondering why this didn't happen with the old
> >>>> Jailhouse version.
> >>>>
> >>>>>
> >>>
> >>> It did not happen because in version 0.8 I got the exception 0x17, which 
> >>> is solved by version 0.9.1.
> >>>
> >>>>> FATAL: unhandled trap (exception class 0x24)
> >>>>>
> >>>>> To solve this, I added the TCM memory region 0xffe0 to the root 
> >>>>> cell config. This memory is used by the R5 to boot its image:
> >>>>>
> >>>>> /* TCM region for the R5 */ {
> >>>>> .phys_start = 0xffe0,
> >>>>> .virt_start = 0xffe0,
> >>>>> .size =   0xC,
> >>>>> .flags = JAILHOUSE_MEM_READ | 
> >>>>> JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
> >>>>
> >>>> Is it IO? JAILHOUSE_MEM_IO?
> >>>>
> >>>
> >>> The TCM memory is not I/O. It is the memory the R5s use.
> >>>
> >>>
> >>>>> },
> >>>>>
> >>>>> However, when I use the zynqmp_r5_remoteproc driver with the jailhouse, 
> >>>>> the driver has an error, and the code is not ran:
> >>>>>
> >>>>> remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> >>>>> remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
> >>>>> Failed to get RPU node status.
> >>>>> Failed to get RPU node status.
> >>>>
> >>>> Hmm, don't know what's exactly behind that memory area or how R5 works,
> >>>> but I'd guess it's iomem. Try to add JAILHOUSE_MEM_IO to the region.
> >>>
> >>> Does not work with JAILHOUSE_MEM_IO either. I got the following:
> >>>
> >>> remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> >>> remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
> >>> Failed to get RPU node status.
> >>> Failed to get RPU node status.
> >>> Unhandled data write at 0xffe01800(1)
> >>>
> >>> FATAL: unhandled trap (exception class 0x24)
> >>> Cell state before exception:
> >>>pc: ff80089f5dac   lr: ff8008789a78 spsr: 4145 EL1
> >>>sp: ff80097d3c10  esr: 24 1 061
> >>>x0: ff8009421800   x1:    x2: 77c0
> >>>x3: 0004   x4:    x5: 0040
> >>>x6: 003f   x7:    x8: ff8009421800
> >>>x9:   x10:   x11: 
> >>> x12:   x13:   x14: 
> >>> x15:   x16: 40034850  x17: 1b6b01bd
> >>> x18: 0010  x19: ff8009c01054  x20: 0001
> >>> x21: 1800  x22: 9000  x23: ffc04b0ac800
> >>> x24: ff8009c01000  x25: ffc04b0ac838  x26: ffc04c3e3480
> >>> x27: ff800942  x28:   x29: ff80097d3c10
> >>>
> >>> Parking CPU 1 (Cell: "ZynqMP-ZCU102")
> >>>
> >>
> >> Could it be that the non-root cell config steals that memory region from
> >> the root cell? Or is only the root cell present at that point?
> >>
> >> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Not printing on UART after moving to version 0.9.1

[Giovani Gracioli]

git bisect log
git bisect start
# old: [006918caa2b90f3de2bce0e9732e2761955f4ddd] inmate: arm-common: remove 
heartbeat
git bisect old 006918caa2b90f3de2bce0e9732e2761955f4ddd

git bisect bad
git bisect good
006918caa2b90f3de2bce0e9732e2761955f4ddd was both good and bad


Is this what you wanted?

> Hi,
> 
> On 08/13/2018 03:18 PM, Giovani Gracioli wrote:
> > Yes, that's right. It seems that something that has changed between 0.9.1 
> > and master somehow broke the UART when Erika is running.
> 
> could you please run git bisect to spot the commit?
> 
> Thanks
>   Ralf
> 
> > 
> >> Hi Giovani,
> >>
> >>> Just confirmed that the UART in the v0.9.1 worked with Erika and vPCI 
> >>> devices on arm64.
> >>
> >> Ok, great, but it should work on master/next as well. Would be good to
> >> know if it's some regression in Jailhouse or not.
> >>
> >>   Ralf
> >>
> >>>
> >>> Thanks!
> >>>
> >>> Giovani
> >>>
> >>>> Hi Ralf,
> >>>>
> >>>> Yes, you are right. I cloned the master branch. 
> >>>>
> >>>> I downloaded this one 
> >>>> (https://github.com/siemens/jailhouse/archive/v0.9.1.tar.gz). Will test 
> >>>> and get back to you.
> >>>>
> >>>> Giovani 
> >>>>
> >>>>> Hi Giovani,
> >>>>>
> >>>>> On 08/09/2018 10:26 PM, Giovani Gracioli wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I was using Jailhouse version 0.8 with Erika on the ultrascale+ 
> >>>>>> platform.
> >>>>>>
> >>>>>> I migrated to version 0.9.1, but there is no printing on the UART 
> >>>>>> after I start the Erika cells.
> >>>>>
> >>>>> Umm... Did you migrate to 0.9.1 or to master/next? 0.9.1 doesn't contain
> >>>>> the patches where I changed the UART config format, so I guess you
> >>>>> migrated to master or next.
> >>>>>
> >>>>>>
> >>>>>> When I run the gic-demo with the zynqmp-zcu102-gic-demo.cell, UART 
> >>>>>> works fine.
> >>>>>
> >>>>> Which means that the UART is basically functional, and Jailhouse inmates
> >>>>> still use it correctly.
> >>>>>
> >>>>>>
> >>>>>> I attached both the root and non-root configurations.
> >>>>>>
> >>>>>> Something related with UART has changed between the versions that is 
> >>>>>> not handled anymore by Erika or is it a config problem?
> >>>>>
> >>>>> Regarding external non-Jailhouse inmates, nothing has changed. The new
> >>>>> format just adds information of the UART device to the communication 
> >>>>> region.
> >>>>>
> >>>>> If there's really a commit between 0.8..next/0.9.1 that introduces some
> >>>>> change that causes your problem, you should be able to find it with git
> >>>>> bisect.
> >>>>>
> >>>>> HTH
> >>>>>   Ralf
> >>>>>
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Giovani
> >>>>>>
> >>>
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Not printing on UART after moving to version 0.9.1

[Giovani Gracioli]

Yes, that's right. It seems that something that has changed between 0.9.1 and 
master somehow broke the UART when Erika is running.

> Hi Giovani,
> 
> > Just confirmed that the UART in the v0.9.1 worked with Erika and vPCI 
> > devices on arm64.
> 
> Ok, great, but it should work on master/next as well. Would be good to
> know if it's some regression in Jailhouse or not.
> 
>   Ralf
> 
> > 
> > Thanks!
> > 
> > Giovani
> > 
> >> Hi Ralf,
> >>
> >> Yes, you are right. I cloned the master branch. 
> >>
> >> I downloaded this one 
> >> (https://github.com/siemens/jailhouse/archive/v0.9.1.tar.gz). Will test 
> >> and get back to you.
> >>
> >> Giovani 
> >>
> >>> Hi Giovani,
> >>>
> >>> On 08/09/2018 10:26 PM, Giovani Gracioli wrote:
> >>>> Hello,
> >>>>
> >>>> I was using Jailhouse version 0.8 with Erika on the ultrascale+ platform.
> >>>>
> >>>> I migrated to version 0.9.1, but there is no printing on the UART after 
> >>>> I start the Erika cells.
> >>>
> >>> Umm... Did you migrate to 0.9.1 or to master/next? 0.9.1 doesn't contain
> >>> the patches where I changed the UART config format, so I guess you
> >>> migrated to master or next.
> >>>
> >>>>
> >>>> When I run the gic-demo with the zynqmp-zcu102-gic-demo.cell, UART works 
> >>>> fine.
> >>>
> >>> Which means that the UART is basically functional, and Jailhouse inmates
> >>> still use it correctly.
> >>>
> >>>>
> >>>> I attached both the root and non-root configurations.
> >>>>
> >>>> Something related with UART has changed between the versions that is not 
> >>>> handled anymore by Erika or is it a config problem?
> >>>
> >>> Regarding external non-Jailhouse inmates, nothing has changed. The new
> >>> format just adds information of the UART device to the communication 
> >>> region.
> >>>
> >>> If there's really a commit between 0.8..next/0.9.1 that introduces some
> >>> change that causes your problem, you should be able to find it with git
> >>> bisect.
> >>>
> >>> HTH
> >>>   Ralf
> >>>
> >>>>
> >>>> Best regards,
> >>>> Giovani
> >>>>
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Hi Peng,

No, I have not tried. Actually this code is part of the Xilinx driver, it is 
not mine. It works fine when Jailhouse root cell is not loaded. So I would 
rather keep the same code.

> > 
> > The zynqmp_r5_remoteproc driver apparently is doing the following (I am not
> > an expert on the arm64 assembly):
> > 
> > ENTRY(__arm_smccc_hvc)
> > SMCCC   hvc
> > ENDPROC(__arm_smccc_hvc)
> 
> Have you tried __arm_smccc_smc?
> 
> Regards,
> Peng
> 
> > 
> > .macro SMCCC instr
> > .cfi_startproc
> > \instr  #0
> > ldr x4, [sp]
> > stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]
> > stp x2, x3, [x4, #ARM_SMCCC_RES_X2_OFFS]
> > ldr x4, [sp, #8]
> > cbz x4, 1f /* no quirk structure */
> > ldr x9, [x4, #ARM_SMCCC_QUIRK_ID_OFFS]
> > cmp x9, #ARM_SMCCC_QUIRK_QCOM_A6
> > b.ne1f
> > str x6, [x4, ARM_SMCCC_QUIRK_STATE_OFFS]
> > 1:  ret
> > .cfi_endproc
> > .endm
> > 
> > Giovani
> > 
> > >
> > >   Ralf
> > >
> > > >
> > > > This does not happen when Jailhouse is loaded. Should the flags contain
> > anything else?
> > > >
> > > > Best
> > > > Giovani
> > > >
> > > >> Hi,
> > > >>
> > > >> On 08/09/2018 04:26 PM, Giovani Gracioli wrote:
> > > >>> Hello,
> > > >>>
> > > >>> I am running Jailhouse v0.8 on the ultrascale+. I have enabled the 
> > > >>> root cell
> > then started the remote R5 processor (using the zynqmp_r5_remoteproc and
> > rpmsg_user_dev_driver drivers). The R5 communicates to A53 through
> > interrupts from the remote proc filesystem.
> > > >>>
> > > >>> Once I started the R5 code, I got an "unhandled trap (exception class
> > 0x17)".
> > > >>
> > > >> This is a SMC trap. Try updating to latest next, there's a high
> > > >> chance that this error will disappear.
> > > >>
> > > >>   Ralf
> > > >>
> > > >>>
> > > >>> FATAL: unhandled trap (exception class 0x17) Cell state before
> > > >>> exception:
> > > >>>  pc: ff800808f428   lr: ff8008769094 spsr: 6145 EL1
> > > >>>  sp: ff8009f6bbf0  esr: 17 1 000
> > > >>>  x0: c203   x1: 0007   x2:
> > 
> > > >>>  x3:    x4:    x5:
> > 
> > > >>>  x6:    x7:    x8:
> > ffc04c1feb00
> > > >>>  x9:   x10: 5f49e37b  x11:
> > > >>> 18c74a31
> > > >>> x12: a87bd568  x13: a78138a0  x14:
> > > >>> 1b3408f1
> > > >>> x15: 175d87f8  x16: 40034850  x17:
> > > >>> 1b6b01bd
> > > >>> x18: af4aa165  x19: ff8009f6bc78  x20:
> > > >>> ff8009f6bcb8
> > > >>> x21: ff8009f6bcbc  x22: ffc04b890838  x23:
> > > >>> ffc04177a800
> > > >>> x24: 0044  x25: ffc04c1fef44  x26:
> > > >>> 0040
> > > >>> x27: ff8008a21000  x28: ffc04cd80080  x29:
> > > >>> ff8009f6bc00
> > > >>>
> > > >>> Parking CPU 1 (Cell: "ZynqMP-ZCU102")
> > > >>>
> > > >>> Is this a configuration problem or something else?
> > > >>>
> > > >>> Best
> > > >>> Giovani
> > > >>>
> > > >
> > 
> > --

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

There is only the root cell. The problem does not happen when the root cell is 
not loaded.

My understanding is that the zynqmp_r5_remoteproc driver issues and hvc 
instruction to get information whether the R5 is running or not. This 
instruction is somehow intercepted by Jailhouse and does not return the 
expected (the R5 is actually running). 

> >> Hi,
> >>
> >> On 08/09/2018 06:35 PM, Giovani Gracioli wrote:
> >>> Thanks Ralf.
> >>>
> >>> I updated to the latest version, 0.9.1, and the original problem is gone.
> >>
> >> Great.
> >>
> >>>
> >>> Then I got an unhandled data write:
> >>
> >> One bug down, the next shows up ;-)
> >>
> >>>
> >>> Unhandled data write at 0xffe01800(1)
> >>
> >> That's weird, I'm wondering why this didn't happen with the old
> >> Jailhouse version.
> >>
> >>>
> > 
> > It did not happen because in version 0.8 I got the exception 0x17, which is 
> > solved by version 0.9.1.
> > 
> >>> FATAL: unhandled trap (exception class 0x24)
> >>>
> >>> To solve this, I added the TCM memory region 0xffe0 to the root cell 
> >>> config. This memory is used by the R5 to boot its image:
> >>>
> >>> /* TCM region for the R5 */ {
> >>>   .phys_start = 0xffe0,
> >>>   .virt_start = 0xffe0,
> >>>   .size =   0xC,
> >>>   .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> >>> JAILHOUSE_MEM_EXECUTE,
> >>
> >> Is it IO? JAILHOUSE_MEM_IO?
> >>
> > 
> > The TCM memory is not I/O. It is the memory the R5s use.
> > 
> > 
> >>> },
> >>>
> >>> However, when I use the zynqmp_r5_remoteproc driver with the jailhouse, 
> >>> the driver has an error, and the code is not ran:
> >>>
> >>> remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> >>> remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
> >>> Failed to get RPU node status.
> >>> Failed to get RPU node status.
> >>
> >> Hmm, don't know what's exactly behind that memory area or how R5 works,
> >> but I'd guess it's iomem. Try to add JAILHOUSE_MEM_IO to the region.
> > 
> > Does not work with JAILHOUSE_MEM_IO either. I got the following:
> > 
> > remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> > remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
> > Failed to get RPU node status.
> > Failed to get RPU node status.
> > Unhandled data write at 0xffe01800(1)
> > 
> > FATAL: unhandled trap (exception class 0x24)
> > Cell state before exception:
> >   pc: ff80089f5dac   lr: ff8008789a78 spsr: 4145 EL1
> >   sp: ff80097d3c10  esr: 24 1 061
> >   x0: ff8009421800   x1:    x2: 77c0
> >   x3: 0004   x4:    x5: 0040
> >   x6: 003f   x7:    x8: ff8009421800
> >   x9:   x10:   x11: 
> > x12:   x13:   x14: 
> > x15:   x16: 40034850  x17: 1b6b01bd
> > x18: 0010  x19: ff8009c01054  x20: 0001
> > x21: 1800  x22: 9000  x23: ffc04b0ac800
> > x24: ff8009c01000  x25: ffc04b0ac838  x26: ffc04c3e3480
> > x27: ff800942  x28:   x29: ff80097d3c10
> > 
> > Parking CPU 1 (Cell: "ZynqMP-ZCU102")
> > 
> 
> Could it be that the non-root cell config steals that memory region from 
> the root cell? Or is only the root cell present at that point?
> 
> Jan

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Em quinta-feira, 9 de agosto de 2018 19:20:36 UTC-4, Ralf Ramsauer  escreveu:
> Hi,
> 
> On 08/09/2018 06:35 PM, Giovani Gracioli wrote:
> > Thanks Ralf.
> > 
> > I updated to the latest version, 0.9.1, and the original problem is gone.
> 
> Great.
> 
> > 
> > Then I got an unhandled data write:
> 
> One bug down, the next shows up ;-)
> 
> > 
> > Unhandled data write at 0xffe01800(1)
> 
> That's weird, I'm wondering why this didn't happen with the old
> Jailhouse version.
> 
> > 

It did not happen because in version 0.8 I got the exception 0x17, which is 
solved by version 0.9.1.

> > FATAL: unhandled trap (exception class 0x24)
> > 
> > To solve this, I added the TCM memory region 0xffe0 to the root cell 
> > config. This memory is used by the R5 to boot its image:
> > 
> > /* TCM region for the R5 */ {
> > .phys_start = 0xffe0,
> > .virt_start = 0xffe0,
> > .size =   0xC,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> > JAILHOUSE_MEM_EXECUTE,
> 
> Is it IO? JAILHOUSE_MEM_IO?
> 

The TCM memory is not I/O. It is the memory the R5s use.


> > },
> > 
> > However, when I use the zynqmp_r5_remoteproc driver with the jailhouse, the 
> > driver has an error, and the code is not ran:
> > 
> > remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
> > remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
> > Failed to get RPU node status.
> > Failed to get RPU node status.
> 
> Hmm, don't know what's exactly behind that memory area or how R5 works,
> but I'd guess it's iomem. Try to add JAILHOUSE_MEM_IO to the region.

Does not work with JAILHOUSE_MEM_IO either. I got the following:

remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
Failed to get RPU node status.
Failed to get RPU node status.
Unhandled data write at 0xffe01800(1)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: ff80089f5dac   lr: ff8008789a78 spsr: 4145 EL1
 sp: ff80097d3c10  esr: 24 1 061
 x0: ff8009421800   x1:    x2: 77c0
 x3: 0004   x4:    x5: 0040
 x6: 003f   x7:    x8: ff8009421800
 x9:   x10:   x11: 
x12:   x13:   x14: 
x15:   x16: 40034850  x17: 1b6b01bd
x18: 0010  x19: ff8009c01054  x20: 0001
x21: 1800  x22: 9000  x23: ffc04b0ac800
x24: ff8009c01000  x25: ffc04b0ac838  x26: ffc04c3e3480
x27: ff800942  x28:   x29: ff80097d3c10

Parking CPU 1 (Cell: "ZynqMP-ZCU102")

The zynqmp_r5_remoteproc driver apparently is doing the following (I am not an 
expert on the arm64 assembly):

ENTRY(__arm_smccc_hvc)
SMCCC   hvc
ENDPROC(__arm_smccc_hvc)

.macro SMCCC instr
.cfi_startproc
\instr  #0
ldr x4, [sp]
stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]
stp x2, x3, [x4, #ARM_SMCCC_RES_X2_OFFS]
ldr x4, [sp, #8]
cbz x4, 1f /* no quirk structure */
ldr x9, [x4, #ARM_SMCCC_QUIRK_ID_OFFS]
cmp x9, #ARM_SMCCC_QUIRK_QCOM_A6
b.ne1f
str x6, [x4, ARM_SMCCC_QUIRK_STATE_OFFS]
1:  ret
    .cfi_endproc
.endm

Giovani

> 
>   Ralf
> 
> > 
> > This does not happen when Jailhouse is loaded. Should the flags contain 
> > anything else?
> > 
> > Best
> > Giovani
> > 
> >> Hi,
> >>
> >> On 08/09/2018 04:26 PM, Giovani Gracioli wrote:
> >>> Hello,
> >>>
> >>> I am running Jailhouse v0.8 on the ultrascale+. I have enabled the root 
> >>> cell then started the remote R5 processor (using the zynqmp_r5_remoteproc 
> >>> and rpmsg_user_dev_driver drivers). The R5 communicates to A53 through 
> >>> interrupts from the remote proc filesystem. 
> >>>
> >>> Once I started the R5 code, I got an "unhandled trap (exception class 
> >>> 0x17)".
> >>
> >> This is a SMC trap. Try updating to latest next, there's a high chance
> >> that this error will disappear.
> >>
> >>   Ralf
> >>
> >>>
> >>> FATAL: unhandled trap (exception class 0x17)
> >>> Cell state before exception:
> >>>  pc: ff800808f428   lr: ff8

Re: Not printing on UART after moving to version 0.9.1

[Giovani Gracioli]

Just confirmed that the UART in the v0.9.1 worked with Erika and vPCI devices 
on arm64.

Thanks!

Giovani

> Hi Ralf,
> 
> Yes, you are right. I cloned the master branch. 
> 
> I downloaded this one 
> (https://github.com/siemens/jailhouse/archive/v0.9.1.tar.gz). Will test and 
> get back to you.
> 
> Giovani 
> 
> > Hi Giovani,
> > 
> > On 08/09/2018 10:26 PM, Giovani Gracioli wrote:
> > > Hello,
> > > 
> > > I was using Jailhouse version 0.8 with Erika on the ultrascale+ platform.
> > > 
> > > I migrated to version 0.9.1, but there is no printing on the UART after I 
> > > start the Erika cells.
> > 
> > Umm... Did you migrate to 0.9.1 or to master/next? 0.9.1 doesn't contain
> > the patches where I changed the UART config format, so I guess you
> > migrated to master or next.
> > 
> > > 
> > > When I run the gic-demo with the zynqmp-zcu102-gic-demo.cell, UART works 
> > > fine.
> > 
> > Which means that the UART is basically functional, and Jailhouse inmates
> > still use it correctly.
> > 
> > > 
> > > I attached both the root and non-root configurations.
> > > 
> > > Something related with UART has changed between the versions that is not 
> > > handled anymore by Erika or is it a config problem?
> > 
> > Regarding external non-Jailhouse inmates, nothing has changed. The new
> > format just adds information of the UART device to the communication region.
> > 
> > If there's really a commit between 0.8..next/0.9.1 that introduces some
> > change that causes your problem, you should be able to find it with git
> > bisect.
> > 
> > HTH
> >   Ralf
> > 
> > > 
> > > Best regards,
> > > Giovani
> > >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Not printing on UART after moving to version 0.9.1

[Giovani Gracioli]

Hi Ralf,

Yes, you are right. I cloned the master branch. 

I downloaded this one 
(https://github.com/siemens/jailhouse/archive/v0.9.1.tar.gz). Will test and get 
back to you.

Giovani 

> Hi Giovani,
> 
> On 08/09/2018 10:26 PM, Giovani Gracioli wrote:
> > Hello,
> > 
> > I was using Jailhouse version 0.8 with Erika on the ultrascale+ platform.
> > 
> > I migrated to version 0.9.1, but there is no printing on the UART after I 
> > start the Erika cells.
> 
> Umm... Did you migrate to 0.9.1 or to master/next? 0.9.1 doesn't contain
> the patches where I changed the UART config format, so I guess you
> migrated to master or next.
> 
> > 
> > When I run the gic-demo with the zynqmp-zcu102-gic-demo.cell, UART works 
> > fine.
> 
> Which means that the UART is basically functional, and Jailhouse inmates
> still use it correctly.
> 
> > 
> > I attached both the root and non-root configurations.
> > 
> > Something related with UART has changed between the versions that is not 
> > handled anymore by Erika or is it a config problem?
> 
> Regarding external non-Jailhouse inmates, nothing has changed. The new
> format just adds information of the UART device to the communication region.
> 
> If there's really a commit between 0.8..next/0.9.1 that introduces some
> change that causes your problem, you should be able to find it with git
> bisect.
> 
> HTH
>   Ralf
> 
> > 
> > Best regards,
> > Giovani
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Not printing on UART after moving to version 0.9.1

[Giovani Gracioli]

Hello,

I was using Jailhouse version 0.8 with Erika on the ultrascale+ platform.

I migrated to version 0.9.1, but there is no printing on the UART after I start 
the Erika cells.

When I run the gic-demo with the zynqmp-zcu102-gic-demo.cell, UART works fine.

I attached both the root and non-root configurations.

Something related with UART has changed between the versions that is not 
handled anymore by Erika or is it a config problem?

Best regards,
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for Xilinx ZynqMP ZCU102 eval board
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 *
 * Reservation via device tree: 0x8..0x83fff
 */

#include 
#include 

#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))

struct {
	struct jailhouse_system header;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[9];
	struct jailhouse_irqchip irqchips[1];
	struct jailhouse_pci_device pci_devices[5];
} __attribute__((packed)) config = {
	.header = {
		.signature = JAILHOUSE_SYSTEM_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
		.hypervisor_memory = {
			.phys_start = 0x8,
			.size =   0x00040,
		},
		.debug_console = {
			.address = 0xff00,
			.size = 0x1000,
			/*.flags = JAILHOUSE_CON1_TYPE_XUARTPS |
 JAILHOUSE_CON1_ACCESS_MMIO |
 JAILHOUSE_CON1_REGDIST_4 |
 JAILHOUSE_CON2_TYPE_ROOTPAGE,*/
			.type = JAILHOUSE_CON_TYPE_XUARTPS,
			.flags = JAILHOUSE_CON_ACCESS_MMIO |
 	 JAILHOUSE_CON_REGDIST_4,
		},
		.platform_info = {
			.pci_mmconfig_base = 0xfc00,
			.pci_mmconfig_end_bus = 0,
			.pci_is_virtual = 1,
			.arm = {
.gic_version = 2,
.gicd_base = 0xf901,
.gicc_base = 0xf902f000,
.gich_base = 0xf904,
.gicv_base = 0xf906f000,
.maintenance_irq = 25,
			},
		},
		.root_cell = {
			.name = "ZynqMP-ZCU102",

			.cpu_set_size = sizeof(config.cpus),
			.num_memory_regions = ARRAY_SIZE(config.mem_regions),
			.num_irqchips = ARRAY_SIZE(config.irqchips),
			.num_pci_devices = ARRAY_SIZE(config.pci_devices),

			.vpci_irq_base = 136-32,
		},
	},

	.cpus = {
		0xf,
	},

	.mem_regions = {
		/* MMIO (permissive) */ {
			.phys_start = 0xfd00,
			.virt_start = 0xfd00,
			.size =	  0x0300,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO,
		},
		/* RAM */ {
			.phys_start = 0x0,
			.virt_start = 0x0,
			.size = 0x8000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
		},
		/* RAM */ {
			.phys_start = 0x80060,
			.virt_start = 0x80060,
			.size = 0x7fa0,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
		},
		/* IVSHMEM shared memory region for 00:00.0 - shared with the spike cell */ {
			.phys_start = 0x80070,
			.virt_start = 0x80070,
			.size = 0x1000, //4KB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
		/* IVSHMEM shared memory region for 00:01.0 - shared with the nfer cell */ {
			.phys_start = 0x800701000,
			.virt_start = 0x800701000,
			.size = 0x1000, //4KB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
		/* IVSHMEM shared memory region for 00:02.0 - shared with the level cell */ {
			.phys_start = 0x800702000,
			.virt_start = 0x800702000,
			.size = 0x1000, //4KB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},

		// /* IVSHMEM shared memory region for 00:03.0 */ {
		// 	.phys_start = 0x800703000,
		// 	.virt_start = 0x800703000,
		// 	.size = 0x1000, //4KB
		// 	.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		// },

		/* IVSHMEM shared memory region for 00:04.0 */ {
			.phys_start = 0x800704000,
			.virt_start = 0x800704000,
			.size = 0x1000, //4KB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},

		/* IVSHMEM shared memory region for 00:05.0 */ {
			.phys_start = 0x800705000,
			.virt_start = 0x800705000,
			.size = 0x1000, //4KB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},

		/* TCM region for the R5 */ {
			.phys_start = 0xffe0,
			.virt_start = 0xffe0,
			.size =	  0xC,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
		},

		/* BRAM or DRAM*/ /*{
			.phys_start = 0x5, //0xA000, 
			.virt_start = 0x5, //0xA000, 
			.size = 0x20, //2MB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},*/
	},

	.irqchips = {
		/* GIC */ {
			.address = 0xf901,
			.pin_base = 32,
			.pin_bitmap = {
0x, 0x, 0x, 0x,
			},
		},
	},

	.pci_devices = 

Re: Share memory among cells on arm64

[Giovani Gracioli]

Not sure the situation of this, but I created a patch that provides PCI support 
for arm (Jailhouse version 0.9.1). It is attached.

It is based on the Evidence's code. I tested it on the ultrascale+ (arm64).

It may be useful for some people.

Best
Giovani

> On 2018-04-18 21:52, Giovani Gracioli wrote:
> > Thanks Jan.
> > 
> > It is working now. the irqchip in the non-root cell was missing.
> 
> Good to hear!
> 
> > 
> > I added the following to the non-root cell:
> > 
> > struct jailhouse_irqchip irqchips[1];
> > ...
> > .vpci_irq_base = 140-32
> > ...
> > 
> > .irqchips = {
> > /* GIC */ {
> > .address = 0xf901,
> > .pin_base = 32,
> > .pin_bitmap = {
> > 1 << (54 - 32),
> > 0,
> > 0,
> > (1 << (140 - 128)) | (1 << (142 - 128))
> > },
> > },
> > },
> 
> Yeah, configuration is nasty, many things have to be correct at the same
> time. That's why I pointed to the Linux demos as reference. Still, we
> need better tooling here.
> 
> > 
> > I attached here both configurations for those that want to use ivhsmem 
> > interrupts in arm64. I would be happy to help to incorporate this working 
> > demo into the master branch.
> 
> Would be very welcome! Start with crafting a patch series and posting it
> for review.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
diff --git a/inmates/lib/arm-common/Makefile.lib b/inmates/lib/arm-common/Makefile.lib
index 669ff48..ee34b8c 100644
--- a/inmates/lib/arm-common/Makefile.lib
+++ b/inmates/lib/arm-common/Makefile.lib
@@ -37,8 +37,8 @@
 #
 
 objs-y := ../string.o ../cmdline.o ../setup.o ../alloc.o ../uart-8250.o
-objs-y += ../printk.o
-objs-y += printk.o gic.o mem.o timer.o setup.o uart.o
+objs-y += ../printk.o ../pci.o
+objs-y += printk.o gic.o mem.o timer.o setup.o uart.o pci.o
 objs-y += uart-xuartps.o uart-mvebu.o uart-hscif.o uart-scifa.o uart-imx.o
 objs-y += uart-pl011.o
 objs-y += gic-v2.o gic-v3.o
diff --git a/inmates/lib/arm-common/include/inmate.h b/inmates/lib/arm-common/include/inmate.h
index 61e383a..159b889 100644
--- a/inmates/lib/arm-common/include/inmate.h
+++ b/inmates/lib/arm-common/include/inmate.h
@@ -106,9 +106,39 @@ void timer_start(u64 timeout);
 
 void arch_mmu_enable(void);
 
+#define PCI_CFG_VENDOR_ID	0x000
+#define PCI_CFG_DEVICE_ID	0x002
+#define PCI_CFG_COMMAND		0x004
+# define PCI_CMD_IO		(1 << 0)
+# define PCI_CMD_MEM		(1 << 1)
+# define PCI_CMD_MASTER		(1 << 2)
+# define PCI_CMD_INTX_OFF	(1 << 10)
+#define PCI_CFG_STATUS		0x006
+# define PCI_STS_INT		(1 << 3)
+# define PCI_STS_CAPS		(1 << 4)
+#define PCI_CFG_BAR		0x010
+# define PCI_BAR_64BIT		0x4
+#define PCI_CFG_CAP_PTR		0x034
+
+#define PCI_ID_ANY		0x
+
+#define PCI_DEV_CLASS_OTHER	0xff
+
+#define PCI_CAP_MSI		0x05
+#define PCI_CAP_MSIX		0x11
+
+#define MSIX_CTRL_ENABLE	0x8000
+#define MSIX_CTRL_FMASK		0x4000
+
+u32 pci_read_config(u16 bdf, unsigned int addr, unsigned int size);
+void pci_write_config(u16 bdf, unsigned int addr, u32 value,
+		  unsigned int size);
+int pci_find_device(u16 vendor, u16 device, u16 start_bdf);
+int pci_find_cap(u16 bdf, u16 cap);
+
 #include 
 #include 
 
-#include 
+#include "../../include/inmate_common.h"
 
 #endif /* !_JAILHOUSE_INMATE_H */
diff --git a/inmates/lib/arm-common/pci.c b/inmates/lib/arm-common/pci.c
new file mode 100644
index 000..f822f78
--- /dev/null
+++ b/inmates/lib/arm-common/pci.c
@@ -0,0 +1,34 @@
+#include 
+
+#define PCI_CFG_BASE	0xfc00
+
+u32 pci_read_config(u16 bdf, unsigned int addr, unsigned int size)
+{	
+	u64 reg_addr = PCI_CFG_BASE | ((u32)bdf << 8) | (addr & 0xfc);
+	switch (size) {
+	case 1:
+		return mmio_read8((u8 *)(reg_addr + (addr & 0x3)));
+	case 2:
+return mmio_read16((u16 *)(reg_addr + (addr & 0x3)));
+	case 4:
+		return mmio_read32((u32 *)(reg_addr));
+	default:
+		return -1;
+	}
+}
+
+void pci_write_config(u16 bdf, unsigned int addr, u32 value, unsigned int size)
+{
+	u64 reg_addr = PCI_CFG_BASE | ((u32)bdf << 8) | (addr & 0xfc);
+	switch (size) {
+	case 1:
+		mmio_write8((u8 *)(reg_addr + (addr & 0x3)), value);
+		break;
+	case 2:
+		mmio_write16((u16 *)(reg_addr + (addr & 0x3)), value);
+		break;
+	case 4:
+		mmio_write32((u32 *)(reg_addr), value);
+		break;
+	}
+}

Re: ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Thanks Ralf.

I updated to the latest version, 0.9.1, and the original problem is gone.

Then I got an unhandled data write:

Unhandled data write at 0xffe01800(1)

FATAL: unhandled trap (exception class 0x24)

To solve this, I added the TCM memory region 0xffe0 to the root cell 
config. This memory is used by the R5 to boot its image:

/* TCM region for the R5 */ {
.phys_start = 0xffe0,
.virt_start = 0xffe0,
.size =   0xC,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
JAILHOUSE_MEM_EXECUTE,
},

However, when I use the zynqmp_r5_remoteproc driver with the jailhouse, the 
driver has an error, and the code is not ran:

remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
remoteproc remoteproc0: Booting fw image dma_scheduler.elf, size 815824
Failed to get RPU node status.
Failed to get RPU node status.

This does not happen when Jailhouse is loaded. Should the flags contain 
anything else?

Best
Giovani

> Hi,
> 
> On 08/09/2018 04:26 PM, Giovani Gracioli wrote:
> > Hello,
> > 
> > I am running Jailhouse v0.8 on the ultrascale+. I have enabled the root 
> > cell then started the remote R5 processor (using the zynqmp_r5_remoteproc 
> > and rpmsg_user_dev_driver drivers). The R5 communicates to A53 through 
> > interrupts from the remote proc filesystem. 
> > 
> > Once I started the R5 code, I got an "unhandled trap (exception class 
> > 0x17)".
> 
> This is a SMC trap. Try updating to latest next, there's a high chance
> that this error will disappear.
> 
>   Ralf
> 
> > 
> > FATAL: unhandled trap (exception class 0x17)
> > Cell state before exception:
> >  pc: ff800808f428   lr: ff8008769094 spsr: 6145 EL1
> >  sp: ff8009f6bbf0  esr: 17 1 000
> >  x0: c203   x1: 0007   x2: 
> >  x3:    x4:    x5: 
> >  x6:    x7:    x8: ffc04c1feb00
> >  x9:   x10: 5f49e37b  x11: 18c74a31
> > x12: a87bd568  x13: a78138a0  x14: 1b3408f1
> > x15: 175d87f8  x16: 40034850  x17: 1b6b01bd
> > x18: af4aa165  x19: ff8009f6bc78  x20: ff8009f6bcb8
> > x21: ff8009f6bcbc  x22: ffc04b890838  x23: ffc04177a800
> > x24: 0044  x25: ffc04c1fef44  x26: 0040
> > x27: ff8008a21000  x28: ffc04cd80080  x29: ff8009f6bc00
> > 
> > Parking CPU 1 (Cell: "ZynqMP-ZCU102")
> > 
> > Is this a configuration problem or something else?
> > 
> > Best
> > Giovani
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Relation between the PCI bar_mask and physical mapped address

[Giovani Gracioli]

Basically what I want is to have the root cell (using only one core) sending 
individual interrupts to other 3 non-root cells (also using only one core each) 
on the arm64 with 4 cores. Then, each non-root cell should send interrupt 
individually to each other (more 3 interrupts). In total, there are 6 different 
IPIs in my setup. In the tests I made, the 4 ivshmem devices have individual 
interrupt numbers, but the next 2 (shared among non-root cells) use the same 
interrupt numbers.

> On 2018-07-30 21:13, Giovani Gracioli wrote:
> > That is because the number of IRQ pins (four) in PCI, I suppose.
> > 
> > This implies that if I have more than 4 vPCI devices sending interrupts, 
> > some of them will share the same IRQ number, correct? In this case, for 
> > instance, when two non-root cells send interrupts to each other, is there a 
> > way to find out the source IRQ to proper handling it?
> > 
> 
> If you have more than 4 ivshmem devices *per cell* and you are using 
> INTx, then, yes, those devices will start sharing interrupts. But that's 
> not an issue for normal guest drivers, specifically under Linux.
> 
> Jan
> 
> > 
> >> Thanks Jan. The problem had relation with the size of the memory relations 
> >> and not with the bdf.
> >>
> >> I have configured 6 vPCI devices in the root cell. Each ivshmem region has 
> >> 4KB of size. When I enable Jailhouse, the ivshmem driver maps 3 devices 
> >> with the same interrupt number (number 53):
> >>
> >> 00:00.0
> >>  Interrupt: pin A routed to IRQ 53
> >>  Region 0: Memory at fc10 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> 00:01.0
> >>  Interrupt: pin B routed to IRQ 54
> >>  Region 0: Memory at fc101000 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> 00:02.0
> >>  Interrupt: pin C routed to IRQ 55
> >>  Region 0: Memory at fc102000 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> 00:03.0
> >>  Interrupt: pin D routed to IRQ 56
> >>  Region 0: Memory at fc103000 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> 00:04.0
> >>  Interrupt: pin A routed to IRQ 53
> >>  Region 0: Memory at fc104000 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> 00:14.0
> >>  Interrupt: pin A routed to IRQ 53
> >>  Region 0: Memory at fc105000 (64-bit, non-prefetchable) [size=4K]
> >>  Kernel driver in use: uio_ivshmem
> >>
> >> Is there a config problem in this case? Is the bdf number related to the 
> >> assigned interrupt number? I would like to have a different interrupt 
> >> number per device.
> >>
> >>> On 2018-07-24 16:30, Giovani Gracioli wrote:
> >>>> Hi,
> >>>>
> >>>> I would like to understand how the bar_mask option within the 
> >>>> .pci_device config defines the physical mapped address in Linux to 
> >>>> provide IPIs among cells.
> >>>
> >>> The BAR masks just define, which bits of a BAR are in principle writable
> >>> by a guest. That is used by the guest to
> >>>
> >>> a) explore the size of the resource region (write -1 to the BAR, read
> >>>  back what was actually set, invert, add 1, and you have the size -
> >>>  PCI standard)
> >>>
> >>> b) locate the region in guest-physical memory - but only for virtual
> >>>  devices lie ivshmem (well, currently or only virtual device type)
> >>>
> >>> That's all, not direct relationship to IPIs.
> >>>
> >>> Now you have to know how large the MMIO regions of an ivshmem device at
> >>> least have to be. And that is currently 256 for the register region. If
> >>> the device supports MSI-X, you also need 128 bytes for an MSI-X register
> >>> area. But you can find both patterns in existing configs.
> >>>
> >>> Jan
> >>>
> >>> -- 
> >>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >>> Corporate Competence Center Embedded Linux
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ARM64 unhandled trap (exception class 0x17)

[Giovani Gracioli]

Hello,

I am running Jailhouse v0.8 on the ultrascale+. I have enabled the root cell 
then started the remote R5 processor (using the zynqmp_r5_remoteproc and 
rpmsg_user_dev_driver drivers). The R5 communicates to A53 through interrupts 
from the remote proc filesystem. 

Once I started the R5 code, I got an "unhandled trap (exception class 0x17)".

FATAL: unhandled trap (exception class 0x17)
Cell state before exception:
 pc: ff800808f428   lr: ff8008769094 spsr: 6145 EL1
 sp: ff8009f6bbf0  esr: 17 1 000
 x0: c203   x1: 0007   x2: 
 x3:    x4:    x5: 
 x6:    x7:    x8: ffc04c1feb00
 x9:   x10: 5f49e37b  x11: 18c74a31
x12: a87bd568  x13: a78138a0  x14: 1b3408f1
x15: 175d87f8  x16: 40034850  x17: 1b6b01bd
x18: af4aa165  x19: ff8009f6bc78  x20: ff8009f6bcb8
x21: ff8009f6bcbc  x22: ffc04b890838  x23: ffc04177a800
x24: 0044  x25: ffc04c1fef44  x26: 0040
x27: ff8008a21000  x28: ffc04cd80080  x29: ff8009f6bc00

Parking CPU 1 (Cell: "ZynqMP-ZCU102")

Is this a configuration problem or something else?

Best
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Page coloring on Jailhouse

[Giovani Gracioli]

Hello

Is there an implementation of page coloring (cache partitioning) in Jailhouse? 
Has someone already thought about it?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] core: Fix paging_create when mapping single hugepages

[Giovani Gracioli]

Just tested on the ZCU102 platform and the problem was solved with the patch.

Thank you.

Giovani

> When mapping hugepages, we try to clean up previous mappings of smaller
> granularity for the target region first. Unfortunately, that could
> destroy table entries in higher levels when the specific tree became
> empty. This was easily reproducible by trying to map a single 2M region
> at an address where there are no other mappings around.
> 
> The fix is to start the unmap with a subtree, rather than the full
> paging tree. This way we will only clear entries at levels below the one
> we are about to fill with a hugepage.
> 
> As it seems, the unmapping path is currently not used for practical
> setups. The releasing impact could only by tested by artificially
> defining two memory regions where the first one builds up a mapping with
> 3 levels and the second one overwrites this with only one level and a
> hugepage. This case seems unlikely to occur with normal Jailhouse
> reconfigurations. So we may consider dropping the cleanup path in the
> future. It's fixed and kept for now to avoid surprises if the assumption
> above no longer holds.
> 
> Reported-by: Giovani Gracioli <giova...@gmail.com>
> Signed-off-by: Jan Kiszka <jan.kis...@siemens.com>
> ---
>  hypervisor/paging.c | 8 ++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/hypervisor/paging.c b/hypervisor/paging.c
> index b950a816..b5d627e3 100644
> --- a/hypervisor/paging.c
> +++ b/hypervisor/paging.c
> @@ -289,6 +289,7 @@ int paging_create(const struct paging_structures 
> *pg_structs,
>   while (size > 0) {
>   const struct paging *paging = pg_structs->root_paging;
>   page_table_t pt = pg_structs->root_table;
> + struct paging_structures sub_structs;
>   pt_entry_t pte;
>   int err;
>  
> @@ -303,10 +304,13 @@ int paging_create(const struct paging_structures 
> *pg_structs,
>* fail as we are working along hugepage
>* boundaries.
>*/
> - if (paging->page_size > PAGE_SIZE)
> - paging_destroy(pg_structs, virt,
> + if (paging->page_size > PAGE_SIZE) {
> + sub_structs.root_paging = paging;
> + sub_structs.root_table = pt;
> + paging_destroy(_structs, virt,
>  paging->page_size,
>  coherent);
> + }
>   paging->set_terminal(pte, phys, flags);
>   flush_pt_entry(pte, coherent);
>   break;
> -- 
> 2.13.6

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Em quinta-feira, 24 de maio de 2018 14:57:18 UTC-4, J. Kiszka  escreveu:
> On 2018-05-24 16:50, Giovani Gracioli wrote:
> > Em quinta-feira, 24 de maio de 2018 10:44:10 UTC-4, J. Kiszka  escreveu:
> >> On 2018-05-24 16:27, Giovani Gracioli wrote:
> >>>> On 2018-05-24 15:59, Giovani Gracioli wrote:
> >>>>> Not sure about MMU on Erika.
> >>>>>
> >>>>> Yes, the fault is reported as well. This address is just another 
> >>>>> memory. 0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same 
> >>>>> faulty behavior when the size is 2MB.
> >>>>
> >>>> You see the issue also when mapping some arbitrary DRAM region with a
> >>>> size of 2MB? Can you reproduce this with an IVSHMEM region as well? What
> >>>> if you increase the size beyond 2M, to 3M e.g.?
> >>>>
> >>>> Jan
> >>>>
> >>>> -- 
> >>>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >>>> Corporate Competence Center Embedded Linux
> >>>
> >>> DRAM: works with 1MB, 3MB, and 32MB.
> >>>   does not work with 2MB.
> >>>
> >>> BRAM (is limited to 2MB): works with 1MB, does not with 2MB.
> >>>
> >>> IVSHMEM: tested with 1 and 2MB and worked in both cases.
> >>>
> >>
> >> That's not a consistent pattern yet because IVSHMEM and DRAM is just the
> >> same, use the same flags etc.
> >>
> >> Can you send a config that demonstrates the DRAM issue? Ideally very
> >> close to the upstream zynqmp-zcu102 configs.
> >>
> >> Thanks,
> >> Jan
> >>
> >> -- 
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
> > 
> > Just tested these two attached and got:
> > 
> > Unhandled data write at 0x5(4)
> > 
> > FATAL: unhandled trap (exception class 0x24)
> > Cell state before exception:
> >  pc: 6944   lr: 6944 spsr: 6005 EL1
> >  sp: 6870  esr: 24 1 1970045
> >  x0:    x1: 9670   x2: 
> >  x3: 0004   x4: 27b8   x5: 
> >  x6:    x7: 0080   x8: 68b0
> >  x9: 23e0  x10:   x11: 9292
> > x12:   x13:   x14: 0020
> > x15:   x16:   x17: 
> > x18:   x19: 1118  x20: 92cf
> > x21: 00080070  x22: 0005  x23: 000a
> > x24:   x25:   x26: 
> > x27:   x28:   x29: 
> > 
> > Parking CPU 1 (Cell: "nfer")
> > 
> 
> OK, good news, I've reproduced something very strange, and that even
> inside qemu. Debugging...
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Let me know if you want me to test on the real hardware once you have figure it 
out.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Em quinta-feira, 24 de maio de 2018 10:44:10 UTC-4, J. Kiszka  escreveu:
> On 2018-05-24 16:27, Giovani Gracioli wrote:
> >> On 2018-05-24 15:59, Giovani Gracioli wrote:
> >>> Not sure about MMU on Erika.
> >>>
> >>> Yes, the fault is reported as well. This address is just another memory. 
> >>> 0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same faulty 
> >>> behavior when the size is 2MB.
> >>
> >> You see the issue also when mapping some arbitrary DRAM region with a
> >> size of 2MB? Can you reproduce this with an IVSHMEM region as well? What
> >> if you increase the size beyond 2M, to 3M e.g.?
> >>
> >> Jan
> >>
> >> -- 
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
> > 
> > DRAM: works with 1MB, 3MB, and 32MB.
> >   does not work with 2MB.
> > 
> > BRAM (is limited to 2MB): works with 1MB, does not with 2MB.
> > 
> > IVSHMEM: tested with 1 and 2MB and worked in both cases.
> > 
> 
> That's not a consistent pattern yet because IVSHMEM and DRAM is just the
> same, use the same flags etc.
> 
> Can you send a config that demonstrates the DRAM issue? Ideally very
> close to the upstream zynqmp-zcu102 configs.
> 
> Thanks,
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Just tested these two attached and got:

Unhandled data write at 0x5(4)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: 6944   lr: 6944 spsr: 6005 EL1
 sp: 6870  esr: 24 1 1970045
 x0:    x1: 9670   x2: 
 x3: 0004   x4: 27b8   x5: 
 x6:    x7: 0080   x8: 68b0
 x9: 23e0  x10:   x11: 9292
x12:   x13:   x14: 0020
x15:   x16:   x17: 
x18:   x19: 1118  x20: 92cf
x21: 00080070  x22: 0005  x23: 000a
x24:   x25:   x26: 
x27:   x28:   x29: 

Parking CPU 1 (Cell: "nfer")

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for Xilinx ZynqMP ZCU102 eval board
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka <jan.kis...@siemens.com>
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 *
 * Reservation via device tree: 0x8..0x83fff
 */

#include 
#include 

#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))

struct {
	struct jailhouse_system header;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[7];
	struct jailhouse_irqchip irqchips[1];
	struct jailhouse_pci_device pci_devices[3];
} __attribute__((packed)) config = {
	.header = {
		.signature = JAILHOUSE_SYSTEM_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.hypervisor_memory = {
			.phys_start = 0x8,
			.size =   0x00040,
		},
		.debug_console = {
			.address = 0xff00,
			.size = 0x1000,
			.flags = JAILHOUSE_CON1_TYPE_XUARTPS |
 JAILHOUSE_CON1_ACCESS_MMIO |
 JAILHOUSE_CON1_REGDIST_4 |
 JAILHOUSE_CON2_TYPE_ROOTPAGE,
		},
		.platform_info = {
			.pci_mmconfig_base = 0xfc00,
			.pci_mmconfig_end_bus = 0,
			.pci_is_virtual = 1,
			.arm = {
.gic_version = 2,
.gicd_base = 0xf901,
.gicc_base = 0xf902f000,
.gich_base = 0xf904,
.gicv_base = 0xf906f000,
.maintenance_irq = 25,
			},
		},
		.root_cell = {
			.name = "ZynqMP-ZCU102",

			.cpu_set_size = sizeof(config.cpus),
			.num_memory_regions = ARRAY_SIZE(config.mem_regions),
			.num_irqchips = ARRAY_SIZE(config.irqchips),
			.num_pci_devices = ARRAY_SIZE(config.pci_devices),

			.vpci_irq_base = 136-32,
		},
	},

	.cpus = {
		0xf,
	},

	.mem_regions = {
		/* MMIO (permissive) */ {
			.phys_start = 0xfd00,
			.virt_start = 0xfd00,
			.size =	  0x0300,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_IO,
		},
		/* RAM */ {
			.phys_start = 0x0,
			.virt_start = 0x0,
			.size = 0x8000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE,
		},
		/* RAM */ {
			.phys_start = 0x8006000

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

> On 2018-05-24 15:59, Giovani Gracioli wrote:
> > Not sure about MMU on Erika.
> > 
> > Yes, the fault is reported as well. This address is just another memory. 
> > 0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same faulty 
> > behavior when the size is 2MB.
> 
> You see the issue also when mapping some arbitrary DRAM region with a
> size of 2MB? Can you reproduce this with an IVSHMEM region as well? What
> if you increase the size beyond 2M, to 3M e.g.?
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

DRAM: works with 1MB, 3MB, and 32MB.
  does not work with 2MB.

BRAM (is limited to 2MB): works with 1MB, does not with 2MB.

IVSHMEM: tested with 1 and 2MB and worked in both cases.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Not sure about MMU on Erika.

Yes, the fault is reported as well. This address is just another memory. 
0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same faulty behavior 
when the size is 2MB.

> > Hi,
> > 
> > The configs are attached. I am running Erika on the non-root cell.
> 
> Does Erika enable the MMU? Is it prepared to map memory from higher
> addresses?
> 
> I'm now seeing in your configs that you map a region at 0x5 - Is
> the access fault reported for that address as well?
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Hi,

The configs are attached. I am running Erika on the non-root cell.

> > Don't know exactly why, but if I change the size for 1MB instead of 2MB, it 
> > works.
> > 
> > Is this because of this line in the mmu_cell.c?
> > 
> > size = MIN(region_size, NUM_TEMPORARY_PAGES * PAGE_SIZE);
> > 
> > Thus, is the maximum size of any memory region 1MB?
> > 
> 
> No, the minimum size is 4K on all supported architectures.
> 
> Without the configs you were using, it's hard to say where the mistake is.
> 
> Also, what kind of inmate are you booting in the non-root cell?
> 
> Jan
> 
> >> Hi,
> >>
> >> I checked the array sizes and they are correct. num_memory_regions uses 
> >> the ARRAY_SIZE macro:
> >>
> >> .num_memory_regions = ARRAY_SIZE(config.mem_regions)
> >>
> >> In Linux I can mmap and access the mapped region without errors.
> >>
> >> Any other thought?
> >>
> >>>
> >>> Maybe the region was not properly registered. Did you increase the
> >>> mem_regions array size? Did you change num_memory_regions so that it
> >>> requires manual updates? That's better discussed over the full config.
> >>>
> >>> Note that, if you want to share a region between root and non-root cell,
> >>> the latter also needs JAILHOUSE_MEM_ROOTSHARED.
> >>>
> >>> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for Xilinx ZynqMP ZCU102 eval board
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 *
 * Reservation via device tree: 0x8..0x83fff
 */

#include 
#include 

#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))

struct {
	struct jailhouse_system header;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[7];
	struct jailhouse_irqchip irqchips[1];
	struct jailhouse_pci_device pci_devices[3];
} __attribute__((packed)) config = {
	.header = {
		.signature = JAILHOUSE_SYSTEM_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.hypervisor_memory = {
			.phys_start = 0x8,
			.size =   0x00040,
		},
		.debug_console = {
			.address = 0xff00,
			.size = 0x1000,
			.flags = JAILHOUSE_CON1_TYPE_XUARTPS |
 JAILHOUSE_CON1_ACCESS_MMIO |
 JAILHOUSE_CON1_REGDIST_4 |
 JAILHOUSE_CON2_TYPE_ROOTPAGE,
		},
		.platform_info = {
			.pci_mmconfig_base = 0xfc00,
			.pci_mmconfig_end_bus = 0,
			.pci_is_virtual = 1,
			.arm = {
.gic_version = 2,
.gicd_base = 0xf901,
.gicc_base = 0xf902f000,
.gich_base = 0xf904,
.gicv_base = 0xf906f000,
.maintenance_irq = 25,
			},
		},
		.root_cell = {
			.name = "ZynqMP-ZCU102",

			.cpu_set_size = sizeof(config.cpus),
			.num_memory_regions = ARRAY_SIZE(config.mem_regions),
			.num_irqchips = ARRAY_SIZE(config.irqchips),
			.num_pci_devices = ARRAY_SIZE(config.pci_devices),

			.vpci_irq_base = 136-32,
		},
	},

	.cpus = {
		0xf,
	},

	.mem_regions = {
		/* MMIO (permissive) */ {
			.phys_start = 0xfd00,
			.virt_start = 0xfd00,
			.size =	  0x0300,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_IO,
		},
		/* RAM */ {
			.phys_start = 0x0,
			.virt_start = 0x0,
			.size = 0x8000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE,
		},
		/* RAM */ {
			.phys_start = 0x80060,
			.virt_start = 0x80060,
			.size = 0x7fa0,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE,
		},
		/* IVSHMEM shared memory region for 00:00.0 */ {
			.phys_start = 0x80040,
			.virt_start = 0x80040,
			.size = 0x10,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
		/* IVSHMEM shared memory region for 00:02.0 */ {
			.phys_start = 0x80080,
			.virt_start = 0x80080,
			.size = 0x10,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
		/* IVSHMEM shared memory region for 00:01.0 */ {
			.phys_start = 0x80070,
			.virt_start = 0x80070,
			.size = 0x10,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
		/* BRAM */ {
			.phys_start = 0x5,
			.virt_start = 0x5,
			.size = 0x10, //1MB
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
		},
	},

	.irqchips = {
		/* GIC */ {
			.address = 0xf901,
			.pin_base = 32,
			.pin_bitmap = {
0x, 0x, 0x, 0x,
			},
		},
	},

	.pci_devices = {
		/* 00:00.0 */ {
			.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
			.bdf = 0,
			.bar_mask = {
0xff00, 0x, 0x,
0x, 0x, 0x,
			

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Don't know exactly why, but if I change the size for 1MB instead of 2MB, it 
works.

Is this because of this line in the mmu_cell.c?

size = MIN(region_size, NUM_TEMPORARY_PAGES * PAGE_SIZE);

Thus, is the maximum size of any memory region 1MB?

> Hi,
> 
> I checked the array sizes and they are correct. num_memory_regions uses the 
> ARRAY_SIZE macro:
> 
> .num_memory_regions = ARRAY_SIZE(config.mem_regions)
> 
> In Linux I can mmap and access the mapped region without errors.
> 
> Any other thought?
> 
> > 
> > Maybe the region was not properly registered. Did you increase the
> > mem_regions array size? Did you change num_memory_regions so that it
> > requires manual updates? That's better discussed over the full config.
> > 
> > Note that, if you want to share a region between root and non-root cell,
> > the latter also needs JAILHOUSE_MEM_ROOTSHARED.
> > 
> > Jan

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Hi,

I checked the array sizes and they are correct. num_memory_regions uses the 
ARRAY_SIZE macro:

.num_memory_regions = ARRAY_SIZE(config.mem_regions)

In Linux I can mmap and access the mapped region without errors.

Any other thought?

> 
> Maybe the region was not properly registered. Did you increase the
> mem_regions array size? Did you change num_memory_regions so that it
> requires manual updates? That's better discussed over the full config.
> 
> Note that, if you want to share a region between root and non-root cell,
> the latter also needs JAILHOUSE_MEM_ROOTSHARED.
> 
> Jan

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Hello,

I have a BRAM and DRAM blocks available on the FPGA side of the ZCU102 platform.

I would like to make them accessible for jailhouse cells. For instance, a BRAM 
block is mapped to the 0xa000 address.

To do that, I inserted this address into the root and non-root cell 
configurations as mem regions:

Non-root:

{
.phys_start = 0x00A000,
.virt_start = 0x00A000,
.size = 0x20, //2MB
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE 
},

Root:

{
.phys_start = 0x00A000,
.virt_start = 0x00A000,
.size = 0x20, //2MB
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
},

When I run the non-root cell and try to write into the address:

int *bram_ptr = (int *) 0x00A000;
*bram_ptr = 10;

I got an unhandled data write at 0xa000. What am I missing here in the 
configs?

Thanks
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Jailhouse and Xilinx kernel 4.14 (ultrascale+ arm64)

[Giovani Gracioli]

I was right, the problem was related with the uio driver. The driver requires 
the PCI_HOST_GENERIC to be enabled in the kernel. After enabling it (Bus 
support -> PCI host controller drivers -> Generic PCI host controller), 
jailhouse with vpci and interrupts works fine.

Best
Giovani

> Thank you Jan. Yes, the issue was related to the UART clock as pointed out in 
> the another link.
> 
> To fix it, I added the following to the system-user.dtsi file:
> 
> /include/ "system-conf.dtsi"
> /include/ "zynqmp-clk-ccf.dtsi"
> / {
>   clk100: clk100 {
>   compatible = "fixed-clock";
>   #clock-cells = <0>;
>   clock-frequency = <1>;
>   u-boot,dm-pre-reloc;
>   };
> };
> 
>  {
>   status = "disabled";
> };
> 
>  {
>   clocks = < >;
> };
> 
> After that, it was possible to see the output from the gic-demo.bin in the 
> /dev/ttyUSB1.
> 
> Regarding the address range 80-8000ff, it is related to the PCI 
> bridge. After disabling the pcie (see the dtsi above), the address range is 
> not mapped anymore and lspci runs correctly.
> 
> However, the problem now is related to the vPCI mapping. I can see the 
> following outputs regarding vPCI when I run the root and non-root cells:
> 
> Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0 (from 
> the root cell)
> 
> Shared memory connection established: "gic-demo-ivshmem" <--> "ZynqMP-ZCU102" 
> (from the non-root cell).
> 
> But no devices are mapped (lspci is empty), so I believe the problem is in 
> the uio driver. 
> 
> Does the uio driver 
> (https://github.com/henning-schild-work/ivshmem-guest-code) need some 
> specific flag enabled in the kernel? Or does it have a conflict with other 
> driver? In the vanilla kernel, it creates interrupt 38 for the new vPCI 
> device, but in the Xilinx kernel, interrupt 38 is already in use:
> 
> 38:  0  0  0   GICv2  58 Level ffa6.rtc
> 
> The uio driver does not display the usual messages informing that new devices 
> are mapped when jailhouse root cell is created. Also, /proc/interrupts does 
> not contain the new interrupt for the vPCI device that should have been 
> created.
> 
> > > Hello,
> > > 
> > > I have a full Jailhouse setup (including ivshmem and interrupts) running 
> > > on top of the vanilla 4.14 kernel on the Xilinx Ultrascale+ platform. 
> > > However, I need to use some drivers only available on the Xilinx 4.14 
> > > kernel, so I am trying to run Jailhouse on it.
> > > 
> > > I was able to compile and load Jailhouse using the Xilinx kernel. But 
> > > when I load the gic-demo, I can't see any output on the terminal 
> > > (/dev/ttyUSB1). Also, when I check the cell stats, it seems that it not 
> > > running (the stats do not increment).
> > 
> > This might be the same issue that was reported earlier today:
> > 
> > https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg05086.html
> > 
> > > 
> > > # jailhouse enable zynqmp-zcu102.cell 
> > > 
> > > Initializing Jailhouse hypervisor v0.8 (37-g1fa90014-dirty) on CPU 1
> > > Code location: 0xc0200060
> > > Page pool usage after early setup: mem 33/995, remap 64/131072
> > > Initializing processors:
> > >  CPU 1... OK
> > >  CPU 3... OK
> > >  CPU 0... OK
> > >  CPU 2... OK
> > > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
> > > Page pool usage after late setup: mem 41/995, remap 69/131072
> > > Activating hypervisor
> > > [   94.525270] The Jailhouse is opening.
> > > 
> > > # jailhouse cell create zynqmp-zcu102-gic-demo.cell
> > > 
> > > [  107.815785] CPU3: shutdown
> > > [  107.818406] psci: CPU3 killed.
> > > Created cell "gic-demo"
> > > Page pool usage after cell creation: mem 53/995, remap 69/131072
> > > [  107.829653] Created Jailhouse cell "gic-demo"
> > > 
> > > #jailhouse cell load 1 gic-demo.bin 
> > > 
> > > Cell "gic-demo" can be loaded
> > > 
> > > #jailhouse cell start 1
> > > 
> > > Started cell "gic-demo"
> > > 
> > > Any suggestion about the causes of this issue?
> > > 
> > > Another problem is that I have something mapped in 0x80:
> > > 
> > > 80-8000ff : cfg (does anyone know that is this? - I generated 
> > > the linux images using petalinux tools)
> > > 
> > > After I load the root-cell and type lspci, I got an unhandled data read 
> > > in that address:
> > > 
> > > root@linaro-gnome:~# lspci
> > > Unhandled data read at 0x80(4)
> > > 
> > > FATAL: unhandled trap (exception class 0x24)
> > > Cell state before exception:
> > >  pc: ff800847c8dc   lr: ff800847c8c8 spsr: 21c5 EL1
> > >  sp: ff800a0abc90  esr: 24 1 1810005
> > >  x0: ff800b00   x1: ff800b00   x2: 
> > >  x3: ffc04cf8bb98   x4: ffc04cf8b000   x5: ff8008de0308
> > >  x6:    x7: ff800848ac48   x8: ffc04cf8ce80
> > >  x9: 007ffecb7c80  x10: 0101010101010101  x11: 001b
> > > x12: 000f  

Re: Jailhouse and Xilinx kernel 4.14 (ultrascale+ arm64)

[Giovani Gracioli]

Thank you Jan. Yes, the issue was related to the UART clock as pointed out in 
the another link.

To fix it, I added the following to the system-user.dtsi file:

/include/ "system-conf.dtsi"
/include/ "zynqmp-clk-ccf.dtsi"
/ {
clk100: clk100 {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <1>;
u-boot,dm-pre-reloc;
};
};

 {
status = "disabled";
};

 {
clocks = < >;
};

After that, it was possible to see the output from the gic-demo.bin in the 
/dev/ttyUSB1.

Regarding the address range 80-8000ff, it is related to the PCI 
bridge. After disabling the pcie (see the dtsi above), the address range is not 
mapped anymore and lspci runs correctly.

However, the problem now is related to the vPCI mapping. I can see the 
following outputs regarding vPCI when I run the root and non-root cells:

Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0 (from the 
root cell)

Shared memory connection established: "gic-demo-ivshmem" <--> "ZynqMP-ZCU102" 
(from the non-root cell).

But no devices are mapped (lspci is empty), so I believe the problem is in the 
uio driver. 

Does the uio driver (https://github.com/henning-schild-work/ivshmem-guest-code) 
need some specific flag enabled in the kernel? Or does it have a conflict with 
other driver? In the vanilla kernel, it creates interrupt 38 for the new vPCI 
device, but in the Xilinx kernel, interrupt 38 is already in use:

38:  0  0  0   GICv2  58 Level ffa6.rtc

The uio driver does not display the usual messages informing that new devices 
are mapped when jailhouse root cell is created. Also, /proc/interrupts does not 
contain the new interrupt for the vPCI device that should have been created.

> > Hello,
> > 
> > I have a full Jailhouse setup (including ivshmem and interrupts) running on 
> > top of the vanilla 4.14 kernel on the Xilinx Ultrascale+ platform. However, 
> > I need to use some drivers only available on the Xilinx 4.14 kernel, so I 
> > am trying to run Jailhouse on it.
> > 
> > I was able to compile and load Jailhouse using the Xilinx kernel. But when 
> > I load the gic-demo, I can't see any output on the terminal (/dev/ttyUSB1). 
> > Also, when I check the cell stats, it seems that it not running (the stats 
> > do not increment).
> 
> This might be the same issue that was reported earlier today:
> 
> https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg05086.html
> 
> > 
> > # jailhouse enable zynqmp-zcu102.cell 
> > 
> > Initializing Jailhouse hypervisor v0.8 (37-g1fa90014-dirty) on CPU 1
> > Code location: 0xc0200060
> > Page pool usage after early setup: mem 33/995, remap 64/131072
> > Initializing processors:
> >  CPU 1... OK
> >  CPU 3... OK
> >  CPU 0... OK
> >  CPU 2... OK
> > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
> > Page pool usage after late setup: mem 41/995, remap 69/131072
> > Activating hypervisor
> > [   94.525270] The Jailhouse is opening.
> > 
> > # jailhouse cell create zynqmp-zcu102-gic-demo.cell
> > 
> > [  107.815785] CPU3: shutdown
> > [  107.818406] psci: CPU3 killed.
> > Created cell "gic-demo"
> > Page pool usage after cell creation: mem 53/995, remap 69/131072
> > [  107.829653] Created Jailhouse cell "gic-demo"
> > 
> > #jailhouse cell load 1 gic-demo.bin 
> > 
> > Cell "gic-demo" can be loaded
> > 
> > #jailhouse cell start 1
> > 
> > Started cell "gic-demo"
> > 
> > Any suggestion about the causes of this issue?
> > 
> > Another problem is that I have something mapped in 0x80:
> > 
> > 80-8000ff : cfg (does anyone know that is this? - I generated 
> > the linux images using petalinux tools)
> > 
> > After I load the root-cell and type lspci, I got an unhandled data read in 
> > that address:
> > 
> > root@linaro-gnome:~# lspci
> > Unhandled data read at 0x80(4)
> > 
> > FATAL: unhandled trap (exception class 0x24)
> > Cell state before exception:
> >  pc: ff800847c8dc   lr: ff800847c8c8 spsr: 21c5 EL1
> >  sp: ff800a0abc90  esr: 24 1 1810005
> >  x0: ff800b00   x1: ff800b00   x2: 
> >  x3: ffc04cf8bb98   x4: ffc04cf8b000   x5: ff8008de0308
> >  x6:    x7: ff800848ac48   x8: ffc04cf8ce80
> >  x9: 007ffecb7c80  x10: 0101010101010101  x11: 001b
> > x12: 000f  x13:   x14: 
> > x15: 007fbdc31cc0  x16: ff8008198598  x17: 007fbdbed170
> > x18: 0a03  x19: 0004  x20: ff800a0abcec
> > x21: ff8008e8b000  x22: ff800a0abd5c  x23: 0040
> > x24: 000f  x25: ffc04ccfa480  x26: 0004
> > x27: ffc04ccfa480  x28:   x29: ff800a0abc90
> 
> Did you mark the RAM that is used for the hypervisor and also the
> inmates as reserved in the device tree that Linux 

Jailhouse and Xilinx kernel 4.14 (ultrascale+ arm64)

[Giovani Gracioli]

Hello,

I have a full Jailhouse setup (including ivshmem and interrupts) running on top 
of the vanilla 4.14 kernel on the Xilinx Ultrascale+ platform. However, I need 
to use some drivers only available on the Xilinx 4.14 kernel, so I am trying to 
run Jailhouse on it.

I was able to compile and load Jailhouse using the Xilinx kernel. But when I 
load the gic-demo, I can't see any output on the terminal (/dev/ttyUSB1). Also, 
when I check the cell stats, it seems that it not running (the stats do not 
increment).

# jailhouse enable zynqmp-zcu102.cell 

Initializing Jailhouse hypervisor v0.8 (37-g1fa90014-dirty) on CPU 1
Code location: 0xc0200060
Page pool usage after early setup: mem 33/995, remap 64/131072
Initializing processors:
 CPU 1... OK
 CPU 3... OK
 CPU 0... OK
 CPU 2... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
Page pool usage after late setup: mem 41/995, remap 69/131072
Activating hypervisor
[   94.525270] The Jailhouse is opening.

# jailhouse cell create zynqmp-zcu102-gic-demo.cell

[  107.815785] CPU3: shutdown
[  107.818406] psci: CPU3 killed.
Created cell "gic-demo"
Page pool usage after cell creation: mem 53/995, remap 69/131072
[  107.829653] Created Jailhouse cell "gic-demo"

#jailhouse cell load 1 gic-demo.bin 

Cell "gic-demo" can be loaded

#jailhouse cell start 1

Started cell "gic-demo"

Any suggestion about the causes of this issue?

Another problem is that I have something mapped in 0x80:

80-8000ff : cfg (does anyone know that is this? - I generated the 
linux images using petalinux tools)

After I load the root-cell and type lspci, I got an unhandled data read in that 
address:

root@linaro-gnome:~# lspci
Unhandled data read at 0x80(4)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: ff800847c8dc   lr: ff800847c8c8 spsr: 21c5 EL1
 sp: ff800a0abc90  esr: 24 1 1810005
 x0: ff800b00   x1: ff800b00   x2: 
 x3: ffc04cf8bb98   x4: ffc04cf8b000   x5: ff8008de0308
 x6:    x7: ff800848ac48   x8: ffc04cf8ce80
 x9: 007ffecb7c80  x10: 0101010101010101  x11: 001b
x12: 000f  x13:   x14: 
x15: 007fbdc31cc0  x16: ff8008198598  x17: 007fbdbed170
x18: 0a03  x19: 0004  x20: ff800a0abcec
x21: ff8008e8b000  x22: ff800a0abd5c  x23: 0040
x24: 000f  x25: ffc04ccfa480  x26: 0004
x27: ffc04ccfa480  x28:   x29: ff800a0abc90

Thanks
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Configuring more than one vPCI device for ivshmem interrupts

[Giovani Gracioli]

Thanks Jan.

By changing the bar_mask and having the two vpci devices mapped in different 
pages, I was able to run it.

Can you please explain how the bar mask is used to define the address in which 
the device is mapped to?

Giovani

> On 2018-05-04 16:59, Giovani Gracioli wrote:
> > Maybe the problem is with the Linux side and/or UIO driver, because when I 
> > start both cell, I can see that both send interrupts to Linux and they are 
> > received. However, Linux can only send interrupts to cell 1.
> 
> The UIO setup is not mature on ARM yet, that's why I was pointing to the
> ivshmem-net scenario. There might be issues remaining.
> 
> > 
> > I also did an user space program that maps the PCI regions using mmap and 
> > sends interrupts through the mapped addresses. I can map region 1 
> > (0xfc10) with size 0x100 without program, but for region 2 (0xfc100100) 
> > I can only map using size 4096. With size 0x100, I get an invalid mapping 
> > from mmap (maybe some align problem?). Using 4096 in region 2, I can issue 
> > an interrupt, but it is sent to cell 1, instead of cell 2.
> 
> Memory mappings have to be page-aligned, /wrt the size and start
> address. In order to make Linux reserve a full page for an MMIO resource
> of some device (so that it can be handed out to userspace separately),
> tune the bar_mask for the respective region. 0xf000, 0x, ...
> should do the trick here.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Configuring more than one vPCI device for ivshmem interrupts

[Giovani Gracioli]

Maybe the problem is with the Linux side and/or UIO driver, because when I 
start both cell, I can see that both send interrupts to Linux and they are 
received. However, Linux can only send interrupts to cell 1.

I also did an user space program that maps the PCI regions using mmap and sends 
interrupts through the mapped addresses. I can map region 1 (0xfc10) with 
size 0x100 without program, but for region 2 (0xfc100100) I can only map using 
size 4096. With size 0x100, I get an invalid mapping from mmap (maybe some 
align problem?). Using 4096 in region 2, I can issue an interrupt, but it is 
sent to cell 1, instead of cell 2.

Any advise on how to test this to find the error?

> On 2018-05-03 20:26, Giovani Gracioli wrote:
> > I noticed that if I set a higher bdf value (0x0 in one cell config and 0x10 
> > in the another one) in the .pci_devices, I can see two different interrupts 
> > mapped in Linux:
> > 
> > 38:  2  0 GICv2 136 Edge  uio_ivshmem
> > 39:  5  0 GICv2 138 Edge  uio_ivshmem
> 
> Yes, INTx interrupts are assigned according to slot numbers:
> 
> Pin = (bdf >> 3) & 0x3) + 1
> 
> With 1 = "Pin A", 2 = "Pin B" and so forth - following PCI
> recommendations. The vpci_irq_base you configure per cell is the
> interrupt baseline for the final SPI:
> 
> SPI = vpci_irq_base + pin - 1
> 
> > 
> > # lspci -vv
> > 
> > 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > Stepping- SERR- FastB2B- Di-
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > SERR-  > Latency: 0
> > Interrupt: pin A routed to IRQ 38
> > Region 0: Memory at fc10 (64-bit, non-prefetchable) [size=256]
> > Kernel driver in use: uio_ivshmem
> > 
> > 00:02.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > Stepping- SERR- FastB2B- Di-
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > SERR-  > Latency: 0
> > Interrupt: pin C routed to IRQ 39
> > Region 0: Memory at fc100100 (64-bit, non-prefetchable) [size=256]
> > Kernel driver in use: uio_ivshmem
> > 
> > Then, I loaded and started both bare metal cells. The two cells were able 
> > to find PCI devices. However, cell 2 identified a wrong PCI (01:00.0) with 
> > a wrong bdf (256 instead of 0x10):
> > 
> > Found 1af4:1110 at 01:00.0
> > shmem is at 0x00080050, shmemsz is 0x0010
> > bar0 is at 0xfc100100
> > mapped shmem and bars, got position 0x - bdf = 256
> > 
> > Note that bdf may have changed after the call to pci_find_device(), 256 
> > instead of 0x10 and 01:00.0 insteaf of 02:00.0 as shown in Linux.
> 
> That might be issues of the driver or the userspace test programs. lspci
> and Linux kernel core messages tell the truth.
> 
> > 
> > Also, in Linux, I can send an interrupt to cell 1 by using /dev/uio0 and 
> > the uio_send tool. But when I try to send an interrupt to cell 2 through 
> > /dev/uio1, I get a mmap failed:
> > 
> > [UIO] opening file /dev/uio1
> > [UIO] count is 1
> > mmap failed (0x0x)
> > 
> > I believe something is still not right in the config files. I attached all 
> > of them here. Any help is appreciated.
> 
> You can find a working setup with a full-meshed ivshmem-net network
> between three cells, which implies two distinct ivshmem devices per cell
> under configs/arm64/zynqmp-zcu102{,-linux-demo,-linux-demo-2}.c. You
> should be able to transfer that to your UIO setup (change shmem_protocol
> for the existing device, for a starter).
> 
> Jan
> 
> > 
> >> Hello,
> >>
> >> I have configured ivshmem interrupts between the root-cell (Linux) and 
> >> another bare-metal cell on the ultrascale+ (arm64) platform.
> >>
> >> What I would like to have now is another bare-metal cell, and then 
> >> interrupts among:
> >>
> >> Linux <-> bare metal cell 1
> >> Linux <-> bare metal cell 2
> >>
> >> The idea is that bare metal cell 1 uses the shared memory region 
> >> 0x80040 and vpci 0xfc10 and cell 2 uses 0x80050 and vpci 
> >> 0xfc100100.
> 

Re: Configuring more than one vPCI device for ivshmem interrupts

[Giovani Gracioli]

I noticed that if I set a higher bdf value (0x0 in one cell config and 0x10 in 
the another one) in the .pci_devices, I can see two different interrupts mapped 
in Linux:

38:  2  0 GICv2 136 Edge  uio_ivshmem
39:  5  0 GICv2 138 Edge  uio_ivshmem

# lspci -vv

00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- Di-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR-  Hello,
> 
> I have configured ivshmem interrupts between the root-cell (Linux) and 
> another bare-metal cell on the ultrascale+ (arm64) platform.
> 
> What I would like to have now is another bare-metal cell, and then interrupts 
> among:
> 
> Linux <-> bare metal cell 1
> Linux <-> bare metal cell 2
> 
> The idea is that bare metal cell 1 uses the shared memory region 0x80040 
> and vpci 0xfc10 and cell 2 uses 0x80050 and vpci 0xfc100100.
> 
> In my original root-cell configuration, when I add another vPCI, I can see in 
> Linux that both devices use the same interrupt number (38). Then, when I 
> write to the first doorbell vpci device reg (mapped from 0xfc10), cell 2 
> receives the interrupt, instead of cell 1.
> 
> How can I completely separate both vPCI devices, so each one have different 
> interrupts in Linux and are routed to different Jailhouse cells?
> 
> Best regards,
> Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


zynqmp-zcu102-cpus0-1.cell
Description: Binary data
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for gic-demo inmate on Xilinx ZynqMP ZCU102 eval board:
 * 1 CPU, 64K RAM, 1 serial port
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 */

#include 
#include 

#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])

struct {
	struct jailhouse_cell_desc cell;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[3];
	struct jailhouse_irqchip irqchips[1];
struct jailhouse_pci_device pci_devices[1];
} __attribute__((packed)) config = {
	.cell = {
		.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.name = "level",
		.flags = JAILHOUSE_CELL_PASSIVE_COMMREG,

		.cpu_set_size = sizeof(config.cpus),
		.num_memory_regions = ARRAY_SIZE(config.mem_regions),
		.num_irqchips = ARRAY_SIZE(config.irqchips),
		.num_pci_devices = ARRAY_SIZE(config.pci_devices),

		.vpci_irq_base = 140-32,
	},

	.cpus = {
		0x4,
	},

	.mem_regions = {
		/* UART */ {
			.phys_start = 0xff01,
			.virt_start = 0xff01,
			.size = 0x1000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,
		},
		/* RAM */ {
			.phys_start = 0x80070,
			.virt_start = 0,
			.size = 0x0001,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
		},
		/* IVSHMEM shared memory region for 00:01.0 */ {
			.phys_start = 0x80050,
			.virt_start = 0x80050,
			.size = 0x10,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED,
		},
	},

	.irqchips = {
		/* GIC */ {
			.address = 0xf901,
			.pin_base = 32,
			.pin_bitmap = {
1 << (54 - 32),
0,
0,
(1 << (142 - 128))
			},
		},
	},

.pci_devices = {
		/* 00:02.0 */ {
			.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
			.bdf = 0x10,
			.bar_mask = {
0xff00, 0x, 0x,
0x, 0x, 0x,
			},
			.shmem_region = 2,
			.shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
			.num_msix_vectors = 0,
			.domain = 0,
		},
	},

};
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for gic-demo inmate on Xilinx ZynqMP ZCU102 eval board:
 * 1 CPU, 64K RAM, 1 serial port
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 */

#include 
#include 

#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])

struct {
	struct jailhouse_cell_desc cell;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[3];
	struct jailhouse_irqchip irqchips[1];
struct jailhouse_pci_device pci_devices[1];
} __attribute__((packed)) config = {
	.cell = {
		.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.name = "spike",
		.flags = JAILHOUSE_CELL_PASSIVE_COMMREG,

		.cpu_set_size = sizeof(config.cpus),
		

Configuring more than one vPCI device for ivshmem interrupts

[Giovani Gracioli]

Hello,

I have configured ivshmem interrupts between the root-cell (Linux) and another 
bare-metal cell on the ultrascale+ (arm64) platform.

What I would like to have now is another bare-metal cell, and then interrupts 
among:

Linux <-> bare metal cell 1
Linux <-> bare metal cell 2

The idea is that bare metal cell 1 uses the shared memory region 0x80040 
and vpci 0xfc10 and cell 2 uses 0x80050 and vpci 0xfc100100.

In my original root-cell configuration, when I add another vPCI, I can see in 
Linux that both devices use the same interrupt number (38). Then, when I write 
to the first doorbell vpci device reg (mapped from 0xfc10), cell 2 receives 
the interrupt, instead of cell 1.

How can I completely separate both vPCI devices, so each one have different 
interrupts in Linux and are routed to different Jailhouse cells?

Best regards,
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

I realized that I had forgot .num_irqchips = ARRAY_SIZE(config.irqchips) in the 
non-root cell config.
After adding it to the non-root cell config and created the cell, I got the 
following:

irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending()-> memory_barrier()
irqchip_set_pending() -> send sgi to cpu:1, sgi_id:0, urrent cpu = 3
irqchip_handle_irq(cpu_data:0xc021f000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc021f000, current cpu 1, 
cpu_data->cpu_id = 1
arch_handle_sgi SGI_EVENT(cpu_data=0xc021f000, irqn=1), cpu id = 1, 
current cpu = 1
irqchip_handle_irq(cpu_data:0xc0221000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0221000, current cpu 2, 
cpu_data->cpu_id = 2
arch_handle_sgi SGI_EVENT(cpu_data=0xc0221000, irqn=1), cpu id = 2, 
current cpu = 2
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
pci_cell_init - cell gic-demo-ivshmem, id = 1, num_pci_devices 1
pci_cell_init - pci type = 0
Adding PCI device 00:00.0 to cell "gic-demo-ivshmem"
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
JAILHOUSE_CELL_CREATE: Cannot allocate memory

If I change .vpci_irq_base = 144-32, I got I totally wrong behavior:


irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending()-> memory_barrier()
irqchip_set_pending() -> send sgi to cpu:1, sgi_id:0, urrent cpu = 3
irqchip_handle_irq(cpu_data:0xc022, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc022, current cpu 1, 
cpu_data->cpu_id = 1
arch_handle_sgi SGI_EVENT(cpu_data=0xc022, irqn=1), cpu id = 1, 
current cpu = 1
irqchip_handle_irq(cpu_data:0xc0222000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0222000, current cpu 2, 
cpu_data->cpu_id = 2
arch_handle_sgi SGI_EVENT(cpu_data=0xc0222000, irqn=1), cpu id = 2, 
current cpu = 2
irqchip_handle_irq(cpu_data:0xc0224000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0224000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0224000, irqn=1), cpu id = 3, 
current cpu = 3
pci_cell_init - cell gic-demo-ivshmem, id = 1, num_pci_devices 1
pci_cell_init - pci type = 0, bdf = 0, shmem_region = 0
pci_cell_init - protocol = 0, cell name = gic-demo-ivshmem, peer_mem->size = 
4096, phy add = 4278255616
Removing PCI device 00:00.0 from cell "ZynqMP-ZCU102"
Adding PCI device 00:00.0 to cell "gic-demo-ivshmem"
Created cell "gic-demo-ivshmem"

pci_cell_init is not getting the correct values from the .pci_device, it is 
getting the values from the first memory region instead. Then, if I change 
.num_irqchips = 0, it gets the correct .pci_device values but then I dont have 
interrupts in the bare-metal cell.


> Thanks Jan.
> 
> So what I should do is to have a different IRQ for the root and non-root 
> cells?
> 
> The root cell config has the following: 
> 
> ...
> .vpci_irq_base = 136-32
> ...
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   0x, 0x, 0x, 0x,
>   },
>   },
>   },
> 
> I changed the non-root cell config to be similar to the linux-demo config:
> 
> .vpci_irq_base = 140-32
> 
> and
> 
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   1, 0, 0, 0,
>   },
>   },
>   },
> 
> 
> When I start the non-root cell, I can see the interrupt arriving in the root 
> cell, with irq_id = 136:
> 
> ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu = 3
> ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu = 3
> arch_ivshmem_update_intx(ive=0xc02450a0), 
> device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
> ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 
> 3
> @ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current cpu 
> = 3
> arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current 
> cpu = 3
> irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
> irqchip_set_pending() -> inside if(!cpu_data), irq_id = 136, 

Re: Share memory among cells on arm64

[Giovani Gracioli]

I realized that I had forgot .num_irqchips = ARRAY_SIZE(config.irqchips) in the 
non-root cell config.

After adding it to the non-root cell config and created the cell, I got the 
following:

irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending()-> memory_barrier()
irqchip_set_pending() -> send sgi to cpu:1, sgi_id:0, urrent cpu = 3
irqchip_handle_irq(cpu_data:0xc021f000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc021f000, current cpu 1, 
cpu_data->cpu_id = 1
arch_handle_sgi SGI_EVENT(cpu_data=0xc021f000, irqn=1), cpu id = 1, 
current cpu = 1
irqchip_handle_irq(cpu_data:0xc0221000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0221000, current cpu 2, 
cpu_data->cpu_id = 2
arch_handle_sgi SGI_EVENT(cpu_data=0xc0221000, irqn=1), cpu id = 2, 
current cpu = 2
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
pci_cell_init - cell gic-demo-ivshmem, id = 1, num_pci_devices 1
pci_cell_init - pci type = 0
Adding PCI device 00:00.0 to cell "gic-demo-ivshmem"
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
JAILHOUSE_CELL_CREATE: Cannot allocate memory

> Thanks Jan.
> 
> So what I should do is to have a different IRQ for the root and non-root 
> cells?
> 
> The root cell config has the following: 
> 
> ...
> .vpci_irq_base = 136-32
> ...
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   0x, 0x, 0x, 0x,
>   },
>   },
>   },
> 
> I changed the non-root cell config to be similar to the linux-demo config:
> 
> .vpci_irq_base = 140-32
> 
> and
> 
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   1, 0, 0, 0,
>   },
>   },
>   },
> 
> 
> When I start the non-root cell, I can see the interrupt arriving in the root 
> cell, with irq_id = 136:
> 
> ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu = 3
> ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu = 3
> arch_ivshmem_update_intx(ive=0xc02450a0), 
> device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
> ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 
> 3
> @ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current cpu 
> = 3
> arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current 
> cpu = 3
> irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
> irqchip_set_pending() -> inside if(!cpu_data), irq_id = 136, gicd_base = 
> 0x0014, GICD_ISPENDR = 200, 16
> irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) = > 0
> irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
> cpu_data->cpu_id = 0
> arch_handle_phys_irq calling irqchip_set_pending(cpu_data=0xc021d000, 
> irqn=136), cpu_id = 0
> irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> irqchip_set_pending() -> just return after irqchip.inject_irq, 
> cpu_data->cpu_id = 0
> irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> irqchip_set_pending() -> just return after irqchip.inject_irq, 
> cpu_data->cpu_id = 0
> 
> When I write from Linux, the interrupt counter is not incremented 
> (/proc/interrupt remains the same) and I see that in Jailhouse generates an 
> interrupt with irq_id = 140 (the base for the non-root cell):
> 
> Writing 1 to 0x7f910bf00c
> ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 
> 2
> @ ivshmem_remote_interrupt() ive->remote: 0xc02450a0, current cpu 
> = 2
> arch_ivshmem_trigger_interrupt(ive:0xc02450a0), irq_id:140, current 
> cpu = 2
> irqchip_set_pending(), local_injection = 0, sender (current cpu) = 2
> irqchip_set_pending() -> inside if(!cpu_data), irq_id = 140, gicd_base = 
> 0x0014, GICD_ISPENDR = 200, 16
> 
> 
> However, the interrupt is not handled by the non-root cell (cpu 3). I pass 
> 140 to gic_setup() and enable the irq by writing 0x to 0xfc10:
> 
> static void enable_irq(struct ivshmem_dev_data *d)
> {
>   printk("IVSHMEM: Enabling IVSHMEM_IRQs registers = %p\n", d->registers);
>   

Re: Share memory among cells on arm64

[Giovani Gracioli]

I realized that I had forgot .num_irqchips = ARRAY_SIZE(config.irqchips) in the 
non-root cell config.

When I added it to the cell config and created the cell, I got the following:

jailhouse cell create zynqmp-zcu102-gic-demo-i
irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending()-> memory_barrier()
irqchip_set_pending() -> send sgi to cpu:1, sgi_id:0, urrent cpu = 3
irqchip_handle_irq(cpu_data:0xc021f000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc021f000, current cpu 1, 
cpu_data->cpu_id = 1
arch_handle_sgi SGI_EVENT(cpu_data=0xc021f000, irqn=1), cpu id = 1, 
current cpu = 1
irqchip_handle_irq(cpu_data:0xc0221000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0221000, current cpu 2, 
cpu_data->cpu_id = 2
arch_handle_sgi SGI_EVENT(cpu_data=0xc0221000, irqn=1), cpu id = 2, 
current cpu = 2
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
pci_cell_init - cell gic-demo-ivshmem, id = 1, num_pci_devices 1
pci_cell_init - pci type = 0
Removing PCI device 00:00.0 from cell "ZynqMP-ZCU102"
Adding PCI device 00:00.0 to cell "gic-demo-ivshmem"
irqchip_handle_irq(cpu_data:0xc0223000, irq_id:1), is_sgi(irq_id) = 1
irqchip_handle_irq() this_cpu_data() = 0xc0223000, current cpu 3, 
cpu_data->cpu_id = 3
arch_handle_sgi SGI_EVENT(cpu_data=0xc0223000, irqn=1), cpu id = 3, 
current cpu = 3
JAILHOUSE_CELL_CREATE: Cannot allocate memory


> Thanks Jan.
> 
> So what I should do is to have a different IRQ for the root and non-root 
> cells?
> 
> The root cell config has the following: 
> 
> ...
> .vpci_irq_base = 136-32
> ...
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   0x, 0x, 0x, 0x,
>   },
>   },
>   },
> 
> I changed the non-root cell config to be similar to the linux-demo config:
> 
> .vpci_irq_base = 140-32
> 
> and
> 
> .irqchips = {
>   /* GIC */ {
>   .address = 0xf901,
>   .pin_base = 32,
>   .pin_bitmap = {
>   1, 0, 0, 0,
>   },
>   },
>   },
> 
> 
> When I start the non-root cell, I can see the interrupt arriving in the root 
> cell, with irq_id = 136:
> 
> ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu = 3
> ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu = 3
> arch_ivshmem_update_intx(ive=0xc02450a0), 
> device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
> ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 
> 3
> @ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current cpu 
> = 3
> arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current 
> cpu = 3
> irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
> irqchip_set_pending() -> inside if(!cpu_data), irq_id = 136, gicd_base = 
> 0x0014, GICD_ISPENDR = 200, 16
> irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) = > 0
> irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
> cpu_data->cpu_id = 0
> arch_handle_phys_irq calling irqchip_set_pending(cpu_data=0xc021d000, 
> irqn=136), cpu_id = 0
> irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> irqchip_set_pending() -> just return after irqchip.inject_irq, 
> cpu_data->cpu_id = 0
> irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> irqchip_set_pending() -> just return after irqchip.inject_irq, 
> cpu_data->cpu_id = 0
> 
> When I write from Linux, the interrupt counter is not incremented 
> (/proc/interrupt remains the same) and I see that in Jailhouse generates an 
> interrupt with irq_id = 140 (the base for the non-root cell):
> 
> Writing 1 to 0x7f910bf00c
> ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 
> 2
> @ ivshmem_remote_interrupt() ive->remote: 0xc02450a0, current cpu 
> = 2
> arch_ivshmem_trigger_interrupt(ive:0xc02450a0), irq_id:140, current 
> cpu = 2
> irqchip_set_pending(), local_injection = 0, sender (current cpu) = 2
> irqchip_set_pending() -> inside if(!cpu_data), irq_id = 140, gicd_base = 
> 0x0014, GICD_ISPENDR = 200, 16
> 
> 
> However, the interrupt is not handled by the non-root cell (cpu 3). I pass 
> 140 to gic_setup() and enable the irq by writing 0x to 0xfc10:
> 
> static void enable_irq(struct ivshmem_dev_data *d)
> {
>   

Re: Share memory among cells on arm64

[Giovani Gracioli]

Thanks Jan.

So what I should do is to have a different IRQ for the root and non-root cells?

The root cell config has the following: 

...
.vpci_irq_base = 136-32
...
.irqchips = {
/* GIC */ {
.address = 0xf901,
.pin_base = 32,
.pin_bitmap = {
0x, 0x, 0x, 0x,
},
},
},

I changed the non-root cell config to be similar to the linux-demo config:

.vpci_irq_base = 140-32

and

.irqchips = {
/* GIC */ {
.address = 0xf901,
.pin_base = 32,
.pin_bitmap = {
1, 0, 0, 0,
},
},
},


When I start the non-root cell, I can see the interrupt arriving in the root 
cell, with irq_id = 136:

ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu = 3
ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu = 3
arch_ivshmem_update_intx(ive=0xc02450a0), 
device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 3
@ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current cpu = 
3
arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current cpu 
= 3
irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending() -> inside if(!cpu_data), irq_id = 136, gicd_base = 
0x0014, GICD_ISPENDR = 200, 16
irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) = 0
irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
cpu_data->cpu_id = 0
arch_handle_phys_irq calling irqchip_set_pending(cpu_data=0xc021d000, 
irqn=136), cpu_id = 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0

When I write from Linux, the interrupt counter is not incremented 
(/proc/interrupt remains the same) and I see that in Jailhouse generates an 
interrupt with irq_id = 140 (the base for the non-root cell):

Writing 1 to 0x7f910bf00c
ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 2
@ ivshmem_remote_interrupt() ive->remote: 0xc02450a0, current cpu = 
2
arch_ivshmem_trigger_interrupt(ive:0xc02450a0), irq_id:140, current cpu 
= 2
irqchip_set_pending(), local_injection = 0, sender (current cpu) = 2
irqchip_set_pending() -> inside if(!cpu_data), irq_id = 140, gicd_base = 
0x0014, GICD_ISPENDR = 200, 16


However, the interrupt is not handled by the non-root cell (cpu 3). I pass 140 
to gic_setup() and enable the irq by writing 0x to 0xfc10:

static void enable_irq(struct ivshmem_dev_data *d)
{
printk("IVSHMEM: Enabling IVSHMEM_IRQs registers = %p\n", d->registers);
mmio_write32(d->registers, 0x);
}

It is not clear to me the relation between the root and non-root cells with 
respect to the irqchips and vpci_irq_base configs. Clearly something is wrong 
with my configs.

Appreciate your help!
Giovani

> 
> Root cell and non-root cell are still sharing interrupts /wrt the
> virtual pci host controller and ivshmem. Just derive your inmate config
> from configs/arm64/zynqmp-zcu102-linux-demo[-2].c, those work.
> 
> Jan
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

It is not completely the same configurations, one is the root-cell and another 
one is a bare-metal cell based on the gic-demo. I attached both here.

I believe the Linux program is correct, because I can see the number of 
interrupts increasing by outputting /proc/interrupts. It basically maps 
0xfc10 using mmap and then writes 1 to mapped_adress + 3.

> On 2018-04-16 23:53, Giovani Gracioli wrote:
> > I instrumented the code with several prints.
> > 
> > When I start the bare-metal cell, I can see that an interrupt from CPU 3 is 
> > issued and handled by CPU 0:
> > 
> > Started cell "gic-demo-ivshmem"
> > ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu 
> > = 3
> > ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu 
> > = 3
> > arch_ivshmem_update_intx(ive=0xc02450a0), 
> > device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
> > ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu 
> > = 3
> > @ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current 
> > cpu = 3
> > arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current 
> > cpu = 3
> > irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
> > irqchip_set_pending() -> inside if(!cpu_data)
> > irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) 
> > = 0
> > irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
> > cpu_data->cpu_id = 0
> > arch_handle_phys_irq calling 
> > irqchip_set_pending(cpu_data=0xc021d000, irqn=136), cpu_id = 0
> > irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> > irqchip_set_pending() -> just return after irqchip.inject_irq, 
> > cpu_data->cpu_id = 0
> > irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> > irqchip_set_pending() -> just return after irqchip.inject_irq, 
> > cpu_data->cpu_id = 0
> > 
> > When I run the program to generate an interrupt from Linux, I got the 
> > following output:
> > 
> > /dev/mem opened.
> > Memory mapped at address 0x7f988f2000.
> > Writing 1 to 0x7f988f200c
> > ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu 
> > = 1
> > @ ivshmem_remote_interrupt() ive->remote: 0xc02450a0, current 
> > cpu = 1
> > arch_ivshmem_trigger_interrupt(ive:0xc02450a0), irq_id:136, current 
> > cpu = 1
> > irqchip_set_pending(), local_injection = 0, sender (current cpu) = 1
> > irqchip_set_pending() -> inside if(!cpu_data)
> > irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) 
> > = 0
> > irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
> > cpu_data->cpu_id = 0
> > arch_handle_phys_irq calling 
> > irqchip_set_pending(cpu_data=0xc021d000, irqn=136), cpu_id = 0
> > irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> > irqchip_set_pending() -> just return after irqchip.inject_irq, 
> > cpu_data->cpu_id = 0
> > irqchip_handle_irq(cpu_data:0xc021d000, irq_id:25), is_sgi(irq_id) 
> > = 0
> > irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
> > cpu_data->cpu_id = 0
> > irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
> > irqchip_set_pending() -> just return after irqchip.inject_irq, 
> > cpu_data->cpu_id = 0
> > 
> > Seems that the connection between Linux (root-cell) and the bare-metal one 
> > is not correct, but the other way around is fine. Why is that?
> 
> Just to confirm my understanding: You have the same cell configurations,
> just switching the binary executed in the non-root cell between Linux
> and a bare-metal inmate, right?
> 
> Is the root Linux programming the INTX_CTRL register properly (not part
> of your trace)?
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Configuration for Xilinx ZynqMP ZCU102 eval board
 *
 * Copyright (c) Siemens AG, 2016
 *
 * Authors:
 *  Jan Kiszka <jan.kis...@siemens.com>
 *
 * This work is licensed under th

Re: Share memory among cells on arm64

[Giovani Gracioli]

I instrumented the code with several prints.

When I start the bare-metal cell, I can see that an interrupt from CPU 3 is 
issued and handled by CPU 0:

Started cell "gic-demo-ivshmem"
ivshmem_register_mmio() mmio->address = 8, mmio->is_write = 0, current cpu = 3
ivshmem_register_mmio() mmio->address = 0, mmio->is_write = 1, current cpu = 3
arch_ivshmem_update_intx(ive=0xc02450a0), 
device->info->num_msix_vectors = 0, ctrl eg intx enable = 1
ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 3
@ ivshmem_remote_interrupt() ive->remote: 0xc0245000, current cpu = 
3
arch_ivshmem_trigger_interrupt(ive:0xc0245000), irq_id:136, current cpu 
= 3
irqchip_set_pending(), local_injection = 0, sender (current cpu) = 3
irqchip_set_pending() -> inside if(!cpu_data)
irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) = 0
irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
cpu_data->cpu_id = 0
arch_handle_phys_irq calling irqchip_set_pending(cpu_data=0xc021d000, 
irqn=136), cpu_id = 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0

When I run the program to generate an interrupt from Linux, I got the following 
output:

/dev/mem opened.
Memory mapped at address 0x7f988f2000.
Writing 1 to 0x7f988f200c
ivshmem_register_mmio() mmio->address = 12, mmio->is_write = 1, current cpu = 1
@ ivshmem_remote_interrupt() ive->remote: 0xc02450a0, current cpu = 
1
arch_ivshmem_trigger_interrupt(ive:0xc02450a0), irq_id:136, current cpu 
= 1
irqchip_set_pending(), local_injection = 0, sender (current cpu) = 1
irqchip_set_pending() -> inside if(!cpu_data)
irqchip_handle_irq(cpu_data:0xc021d000, irq_id:136), is_sgi(irq_id) = 0
irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
cpu_data->cpu_id = 0
arch_handle_phys_irq calling irqchip_set_pending(cpu_data=0xc021d000, 
irqn=136), cpu_id = 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0
irqchip_handle_irq(cpu_data:0xc021d000, irq_id:25), is_sgi(irq_id) = 0
irqchip_handle_irq() this_cpu_data() = 0xc021d000, current cpu 0, 
cpu_data->cpu_id = 0
irqchip_set_pending(), local_injection = 1, sender (current cpu) = 0
irqchip_set_pending() -> just return after irqchip.inject_irq, cpu_data->cpu_id 
= 0

Seems that the connection between Linux (root-cell) and the bare-metal one is 
not correct, but the other way around is fine. Why is that?

> On 2018-04-16 20:35, Giovani Gracioli wrote:
> > I added .vpci_irq_base = 136-32 to the bare-metal cell config (.cell) and 
> > now I can see the interrupts generated by uio_send through the output of 
> > cat /proc/interrupts:
> > 
> >CPU0   CPU1   CPU2
> >  38: 38  0  0 GICv2 136 Edge  uio_ivshmem, 
> > uio_ivshmem
> > 
> > When uio_send writes, the interrupt handler of the uio driver is called 
> > twice per write (why??).
> 
> Well, Linux says that the line is shared by two uio devices, see above.
> So both instances will be called on each arriving interrupt.
> 
> > 
> > .vpci_irq_base = 136-32 has the same value as defined in the root-cell. 
> > 
> > I added some prints to the 
> > hypervisor/arch/arm-common/ivshmem.c:arch_ivshmem_trigger_interrupt and 
> > hypervisor/arch/arm-common/irqchip.c as described in another thread 
> > (https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg02605.html),
> >  but I can't see any output.
> > 
> > So, something is still not well configured.
> 
> You can start such trace in hypervisor/ivshmem.c, ivshmem_register_mmio,
> IVSHMEM_REG_DBELL access.
> 
> > 
> > The bare-metal inmate outputs the following when it starts:
> > 
> > IVSHMEM: Found 1af4:1110 at 00:00.0
> > IVSHMEM ERROR: device is not MSI-X capable
> > IVSHMEM: shmem is at 0x00080040, shmemsz is 0x0010
> > IVSHMEM: bar0 is at 0xfc10
> > IVSHMEM: mapped shmem and bars, got position 0x0001 - bdf = 0
> > IVSHMEM: memcpy(0x00080040, Hello from bare-metal ivshmem-demo 
> > inmate!!!  , 64)
> > IVSHMEM: 0x00080040:Hello from bare-metal ivshmem-demo inmate!!!  
> > IVSHMEM: Enabled IRQ:0x88
> > IVSHMEM: Enabling IVSHMEM_IRQs registers = 0xfc10
> > IVSHMEM: Done setting up...
>

Re: Share memory among cells on arm64

[Giovani Gracioli]

I added .vpci_irq_base = 136-32 to the bare-metal cell config (.cell) and now I 
can see the interrupts generated by uio_send through the output of cat 
/proc/interrupts:

   CPU0   CPU1   CPU2
 38: 38  0  0 GICv2 136 Edge  uio_ivshmem, 
uio_ivshmem

When uio_send writes, the interrupt handler of the uio driver is called twice 
per write (why??).

.vpci_irq_base = 136-32 has the same value as defined in the root-cell. 

I added some prints to the 
hypervisor/arch/arm-common/ivshmem.c:arch_ivshmem_trigger_interrupt and 
hypervisor/arch/arm-common/irqchip.c as described in another thread 
(https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg02605.html), 
but I can't see any output.

So, something is still not well configured.

The bare-metal inmate outputs the following when it starts:

IVSHMEM: Found 1af4:1110 at 00:00.0
IVSHMEM ERROR: device is not MSI-X capable
IVSHMEM: shmem is at 0x00080040, shmemsz is 0x0010
IVSHMEM: bar0 is at 0xfc10
IVSHMEM: mapped shmem and bars, got position 0x0001 - bdf = 0
IVSHMEM: memcpy(0x00080040, Hello from bare-metal ivshmem-demo 
inmate!!!  , 64)
IVSHMEM: 0x00080040:Hello from bare-metal ivshmem-demo inmate!!!  
IVSHMEM: Enabled IRQ:0x88
IVSHMEM: Enabling IVSHMEM_IRQs registers = 0xfc10
IVSHMEM: Done setting up...
IVSHMEM: Hello from bare-metal ivshmem-demo inmate!!!  
IVSHMEM: sending interrupt.
IVSHMEM: sending IRQ - addr = 0xfc1c
IVSHMEM: waiting for interrupt.

I'm not sure about the IVSHMEM_IRQ number, that's why I inserted the prints.

Is there any other parameter that must be add to the bare-metal cell to allow 
it to handle ivshmem interrupts?

Giovani

> 
> Both values aren't evaluated by the current Jailhouse implementation
> (see hypervisor/ivshmem.c, ivshmem_register_mmio, IVSHMEM_REG_DBELL
> handling). There is only one vector per device (currently), and that is
> send to the SPI configured in the peer cell's config.
> 
> Maybe there is a mismatch in what is configured and what the device tree
> of that cell tells the Linux guest.
> 
> Jan

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

Hello guys,

Now I am trying to send an interrupt from the root-cell (Linux) to the 
bare-metal cell. The other way around, from the bare-metal to Linux, is 
working. Whenever I start the bare-metal cell, I can see that an interrupt has 
arrived by checking the /proc/interrupts (number 38 is the uio_ivshmem in my 
case).

The problem is that when I try to issue an interrupt from the root-cell, the 
interrupt counter is no incremented and the interrupt does not arrive in the 
bare-metal cell.

I used the uio_send and I program that I implemented, which maps the pci region 
(0xfc10) using mmap and write 1 to the doorbell. However, in both cases I 
can't see an interrupt in the /proc/interrupts.

Looking at the ivshmem documentation 
(https://github.com/qemu/qemu/blob/master/docs/specs/ivshmem-spec.txt), the 
doorbell is divided in two spaces:

"The written value's high 16 bits are the ID of the peer to interrupt,
and its low 16 bits select an interrupt vector."

How can I find the bare-metal peer's ID? What should be the interrupt vector 
value?

Best,
Giovani

> I found the error. It was missing a config parameter in the non-root cell:
> 
> .num_pci_devices = 2 (in the .cell struct)
> 
> After inserting this, I got the correct output:
> 
> Shared memory connection established: "gic-demo-ivshmem" <--> "ZynqMP-ZCU102"
> 
> Thanks for the help.
> 
> > Yes, this is strange and it is not happening.
> > 
> > I added some prints in the pci_cell_init function (hypervisor/pci.c) and 
> > ivhsmem_init function (hypervisor/arm-common/ivhsmem.c). When I enable the 
> > root cell, I got the following:
> > 
> > pci_cell_init - cell ZynqMP-ZCU102, id = 0, num_pci_devices 2
> > pci_cell_init - pci type = 3
> > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
> > %% ivshmem_init() iv->bdf=0
> > pci_cell_init - pci type = 3
> > Adding virtual PCI device 00:00.1 to cell "ZynqMP-ZCU102" Cell Id = 0
> > iv->bdf = 0, dev_info->bdf = 1, id = 1
> > , shmem_region = 3, dev_info->shmem_region = 4
> > ,shmem_protocol = 0, dev protocol = 0
> > , peer_mem->phys_start = 34363932672, mem->phys_start = 34364981248
> > , peer_mem->size = 1048576, mem->size = 1048576
> > , cell->config->name = ZynqMP-ZCU102, peer config name = ZynqMP-ZCU102
> > %% ivshmem_init() iv->bdf=1
> > 
> > When I create the bare metal cell, I got the following output:
> > 
> > # jailhouse cell create zynqmp-zcu102-gic-demo-ivshmem.cell
> > Created cell "gic-demo-ivshmem"
> > Page pool usage after cell creation: mem 56/995, remap 69/131072
> > 
> > It seems that pci_cell_init() is not being called for the non-root cell.
> > 
> > > Hi,
> > > at a first look the configuration looks ok for me.
> > > 
> > > Il giorno giovedì 12 aprile 2018 19:49:09 UTC+2, Giovani Gracioli ha 
> > > scritto:
> > > > Hello,
> > > > 
> > > > Thank you for the answers.
> > > > 
> > > > My comments are below.
> > > > 
> > > > 1) The root config is defined as follows (copied only the relevant 
> > > > parts):
> > > > 
> > > > .platform_info = {
> > > > .pci_mmconfig_base = 0xfc00,
> > > > .pci_mmconfig_end_bus = 0,
> > > > .pci_is_virtual = 1,
> > > > .arm = {
> > > > .gic_version = 2,
> > > > .gicd_base = 0xf901,
> > > > .gicc_base = 0xf902f000,
> > > > .gich_base = 0xf904,
> > > > .gicv_base = 0xf906f000,
> > > > .maintenance_irq = 25,
> > > > },
> > > > },
> > > > 
> > > > .mem_regions = {
> > > > /* MMIO (permissive) */ {
> > > > .phys_start = 0xfd00,
> > > > .virt_start = 0xfd00,
> > > > .size =   0x0300,
> > > > .flags = JAILHOUSE_MEM_READ | 
> > > > JAILHOUSE_MEM_WRITE |
> > > > JAILHOUSE_MEM_IO,
> > > > },
> > > > /* RAM */ {
> > > > .phys_start

Re: Share memory among cells on arm64

[Giovani Gracioli]

I found the error. It was missing a config parameter in the non-root cell:

.num_pci_devices = 2 (in the .cell struct)

After inserting this, I got the correct output:

Shared memory connection established: "gic-demo-ivshmem" <--> "ZynqMP-ZCU102"

Thanks for the help.

> Yes, this is strange and it is not happening.
> 
> I added some prints in the pci_cell_init function (hypervisor/pci.c) and 
> ivhsmem_init function (hypervisor/arm-common/ivhsmem.c). When I enable the 
> root cell, I got the following:
> 
> pci_cell_init - cell ZynqMP-ZCU102, id = 0, num_pci_devices 2
> pci_cell_init - pci type = 3
> Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
> %% ivshmem_init() iv->bdf=0
> pci_cell_init - pci type = 3
> Adding virtual PCI device 00:00.1 to cell "ZynqMP-ZCU102" Cell Id = 0
> iv->bdf = 0, dev_info->bdf = 1, id = 1
> , shmem_region = 3, dev_info->shmem_region = 4
> ,shmem_protocol = 0, dev protocol = 0
> , peer_mem->phys_start = 34363932672, mem->phys_start = 34364981248
> , peer_mem->size = 1048576, mem->size = 1048576
> , cell->config->name = ZynqMP-ZCU102, peer config name = ZynqMP-ZCU102
> %% ivshmem_init() iv->bdf=1
> 
> When I create the bare metal cell, I got the following output:
> 
> # jailhouse cell create zynqmp-zcu102-gic-demo-ivshmem.cell
> Created cell "gic-demo-ivshmem"
> Page pool usage after cell creation: mem 56/995, remap 69/131072
> 
> It seems that pci_cell_init() is not being called for the non-root cell.
> 
> > Hi,
> > at a first look the configuration looks ok for me.
> > 
> > Il giorno giovedì 12 aprile 2018 19:49:09 UTC+2, Giovani Gracioli ha 
> > scritto:
> > > Hello,
> > > 
> > > Thank you for the answers.
> > > 
> > > My comments are below.
> > > 
> > > 1) The root config is defined as follows (copied only the relevant parts):
> > > 
> > > .platform_info = {
> > > .pci_mmconfig_base = 0xfc00,
> > > .pci_mmconfig_end_bus = 0,
> > > .pci_is_virtual = 1,
> > > .arm = {
> > > .gic_version = 2,
> > > .gicd_base = 0xf901,
> > > .gicc_base = 0xf902f000,
> > > .gich_base = 0xf904,
> > > .gicv_base = 0xf906f000,
> > > .maintenance_irq = 25,
> > > },
> > > },
> > > 
> > > .mem_regions = {
> > > /* MMIO (permissive) */ {
> > > .phys_start = 0xfd00,
> > > .virt_start = 0xfd00,
> > > .size =   0x0300,
> > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE 
> > > |
> > > JAILHOUSE_MEM_IO,
> > > },
> > > /* RAM */ {
> > > .phys_start = 0x0,
> > > .virt_start = 0x0,
> > > .size = 0x8000,
> > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE 
> > > |
> > > JAILHOUSE_MEM_EXECUTE,
> > > },
> > > /* RAM */ {
> > > .phys_start = 0x80060,
> > > .virt_start = 0x80060,
> > > .size = 0x7fa0,
> > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE 
> > > |
> > > JAILHOUSE_MEM_EXECUTE,
> > > },
> > > /* IVSHMEM shared memory region for 00:00.0 */ {
> > > .phys_start = 0x80040,
> > > .virt_start = 0x80040,
> > > .size = 0x10,
> > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE 
> > > | JAILHOUSE_MEM_ROOTSHARED,
> > > },
> > > /* IVSHMEM shared memory region for 00:01.0 */ {
> > > .phys_start = 0x80050,
> > > .virt_start = 0x80050,
> > > .size = 0x10,
> > >  

Re: Share memory among cells on arm64

[Giovani Gracioli]

Yes, this is strange and it is not happening.

I added some prints in the pci_cell_init function (hypervisor/pci.c) and 
ivhsmem_init function (hypervisor/arm-common/ivhsmem.c). When I enable the root 
cell, I got the following:

pci_cell_init - cell ZynqMP-ZCU102, id = 0, num_pci_devices 2
pci_cell_init - pci type = 3
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102" Cell Id = 0
%% ivshmem_init() iv->bdf=0
pci_cell_init - pci type = 3
Adding virtual PCI device 00:00.1 to cell "ZynqMP-ZCU102" Cell Id = 0
iv->bdf = 0, dev_info->bdf = 1, id = 1
, shmem_region = 3, dev_info->shmem_region = 4
,shmem_protocol = 0, dev protocol = 0
, peer_mem->phys_start = 34363932672, mem->phys_start = 34364981248
, peer_mem->size = 1048576, mem->size = 1048576
, cell->config->name = ZynqMP-ZCU102, peer config name = ZynqMP-ZCU102
%% ivshmem_init() iv->bdf=1

When I create the bare metal cell, I got the following output:

# jailhouse cell create zynqmp-zcu102-gic-demo-ivshmem.cell
Created cell "gic-demo-ivshmem"
Page pool usage after cell creation: mem 56/995, remap 69/131072

It seems that pci_cell_init() is not being called for the non-root cell.

> Hi,
> at a first look the configuration looks ok for me.
> 
> Il giorno giovedì 12 aprile 2018 19:49:09 UTC+2, Giovani Gracioli ha scritto:
> > Hello,
> > 
> > Thank you for the answers.
> > 
> > My comments are below.
> > 
> > 1) The root config is defined as follows (copied only the relevant parts):
> > 
> > .platform_info = {
> > .pci_mmconfig_base = 0xfc00,
> > .pci_mmconfig_end_bus = 0,
> > .pci_is_virtual = 1,
> > .arm = {
> > .gic_version = 2,
> > .gicd_base = 0xf901,
> > .gicc_base = 0xf902f000,
> > .gich_base = 0xf904,
> > .gicv_base = 0xf906f000,
> > .maintenance_irq = 25,
> > },
> > },
> > 
> > .mem_regions = {
> > /* MMIO (permissive) */ {
> > .phys_start = 0xfd00,
> > .virt_start = 0xfd00,
> > .size =   0x0300,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_IO,
> > },
> > /* RAM */ {
> > .phys_start = 0x0,
> > .virt_start = 0x0,
> > .size = 0x8000,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_EXECUTE,
> > },
> > /* RAM */ {
> > .phys_start = 0x80060,
> > .virt_start = 0x80060,
> > .size = 0x7fa0,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > JAILHOUSE_MEM_EXECUTE,
> > },
> > /* IVSHMEM shared memory region for 00:00.0 */ {
> > .phys_start = 0x80040,
> > .virt_start = 0x80040,
> > .size = 0x10,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> > JAILHOUSE_MEM_ROOTSHARED,
> > },
> > /* IVSHMEM shared memory region for 00:01.0 */ {
> > .phys_start = 0x80050,
> > .virt_start = 0x80050,
> > .size = 0x10,
> > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> > JAILHOUSE_MEM_ROOTSHARED,
> > },
> > },
> > 
> > 
> > .pci_devices = {
> > /* 00:00.0 */ {
> > .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> > .bdf = 0 << 3,
> > .bar_mask = {
> > 0xff00, 0x, 0x,
> > 0x, 0x, 0x,
> > },
> > .shmem_region = 3,
> > .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED, 
> >  

Re: Share memory among cells on arm64

[Giovani Gracioli]

ther inmate cell, the config is defined as follows:
> 
> .mem_regions = {
> /* UART */ {
> .phys_start = 0xff01,
> .virt_start = 0xff01,
> .size = 0x1000,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED,
> },
> /* RAM */ {
> .phys_start = 0x80060,
> .virt_start = 0,
> .size = 0x0001,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> JAILHOUSE_MEM_EXECUTE | 
> JAILHOUSE_MEM_LOADABLE,
> },
> /* IVSHMEM shared memory region for 00:00.0 */ {
> .phys_start = 0x80040,
> .virt_start = 0x80040,
> .size = 0x10,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> JAILHOUSE_MEM_ROOTSHARED,
> },
> },
> 
> .pci_devices = {
> /* 00:00.0 */ {
> .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> .bdf = 0 << 3,
> .bar_mask = {
> 0xff00, 0x, 0x,
> 0x, 0x, 0x,
> },
> .shmem_region = 2,
> .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
> .num_msix_vectors = 0,
> .domain = 0x,
> },
> },
> 
> When I load the uio_ivshmem driver after enabling the root cell, I got the 
> output from the driver which says "using jailhouse mode" (note that I added 
> some prints):
> 
> OF: PCI: host bridge /vpci@0 ranges:
> OF: PCI:   MEM 0xfc10..0xfc103fff -> 0xfc10
> pci-host-generic fc00.vpci: ECAM at [mem 0xfc00-0xfc0f]
> pci-host-generic fc00.vpci: PCI host bridge to bus :00
> pci_bus :00: root bus resource [bus 00]
> pci_bus :00: root bus resource [mem 0xfc10-0xfc103fff]
> pci :00:00.0: [1af4:1110] type 00 class 0xff
> pci :00:00.0: reg 0x10: [mem 0x-0x00ff 64bit]
> pci :00:01.0: [1af4:1110] type 00 class 0xff
> pci :00:01.0: reg 0x10: [mem 0x-0x00ff 64bit]
> pci :00:00.0: BAR 0: assigned [mem 0xfc10-0xfc1000ff 6]
> pci :00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 6]
> **ivshmem_pci_probe()
> uio_ivshmem :00:00.0: enabling device ( -> 0002)
> uio_ivshmem :00:00.0: using jailhouse mode
> **ivshmem_pci_probe() mem[1] addr = 00080040, 6
> **ivshmem_pci_probe()
> uio_ivshmem :00:01.0: enabling device ( -> 0002)
> uio_ivshmem :00:01.0: using jailhouse mode
> **ivshmem_pci_probe() mem[1] addr = 00080050, 6
> 
> The two vPCIs regions are mapped and appear in the lspci command:
> 
> 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> Subsystem: Red Hat, Inc Inter-VM shared memory
> Flags: bus master, fast devsel, latency 0, IRQ 38
> Memory at fc10 (64-bit, non-prefetchable) [size=256]
> Kernel driver in use: uio_ivshmem
> 
> 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> Subsystem: Red Hat, Inc Inter-VM shared memory
> Flags: bus master, fast devsel, latency 0, IRQ 39
> Memory at fc100100 (64-bit, non-prefetchable) [size=256]
> Kernel driver in use: uio_ivshmem
> 
> Also, I can see the addresses in the /proc/iomem:
> 
> fc00-fc0f : PCI ECAM
> fc10-fc103fff : //vpci@0
>   fc10-fc1000ff : :00:00.0
> fc10-fc1000ff : ivshmem
>   fc100100-fc1001ff : :00:01.0
> fc100100-fc1001ff : ivshmem
> 
> However, I don't see any output (regarding the PCI) when I load the non-root 
> inmate cell. 
> 
> 2) I use #define PCI_CFG_BASE0xfc00 in inmates/lib/arm-common/pci.c, 
> which matches the .pci_mmconfig_base = 0xfc00 in root cell config.
> 
> 3) The output from the /sys/class/uio/uio0/maps[0,1] is:
> 
> cat /sys/class/uio/uio0/maps/map0/addr 
> 0xfc10
> cat /sys/class/uio/uio0/maps/map0/offset 
> 0x0
> cat /sys/class/uio/uio0/maps/map0/size
> 0x1000
> cat /sys/class/uio/uio0/maps/map0/name
> registers
> 
> cat /sys/class/uio/uio0/maps/map1/addr  

Re: Share memory among cells on arm64

[Giovani Gracioli]

00, 0x,
},
.shmem_region = 2,
.shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED,
.num_msix_vectors = 0,
.domain = 0x,
},
},

When I load the uio_ivshmem driver after enabling the root cell, I got the 
output from the driver which says "using jailhouse mode" (note that I added 
some prints):

OF: PCI: host bridge /vpci@0 ranges:
OF: PCI:   MEM 0xfc10..0xfc103fff -> 0xfc10
pci-host-generic fc00.vpci: ECAM at [mem 0xfc00-0xfc0f]
pci-host-generic fc00.vpci: PCI host bridge to bus :00
pci_bus :00: root bus resource [bus 00]
pci_bus :00: root bus resource [mem 0xfc10-0xfc103fff]
pci :00:00.0: [1af4:1110] type 00 class 0xff
pci :00:00.0: reg 0x10: [mem 0x-0x00ff 64bit]
pci :00:01.0: [1af4:1110] type 00 class 0xff
pci :00:01.0: reg 0x10: [mem 0x-0x00ff 64bit]
pci :00:00.0: BAR 0: assigned [mem 0xfc10-0xfc1000ff 6]
pci :00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 6]
**ivshmem_pci_probe()
uio_ivshmem :00:00.0: enabling device ( -> 0002)
uio_ivshmem :00:00.0: using jailhouse mode
**ivshmem_pci_probe() mem[1] addr = 00080040, 6
**ivshmem_pci_probe()
uio_ivshmem :00:01.0: enabling device ( -> 0002)
uio_ivshmem :00:01.0: using jailhouse mode
**ivshmem_pci_probe() mem[1] addr = 00080050, 6

The two vPCIs regions are mapped and appear in the lspci command:

00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0, IRQ 38
Memory at fc10 (64-bit, non-prefetchable) [size=256]
Kernel driver in use: uio_ivshmem

00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0, IRQ 39
Memory at fc100100 (64-bit, non-prefetchable) [size=256]
Kernel driver in use: uio_ivshmem

Also, I can see the addresses in the /proc/iomem:

fc00-fc0f : PCI ECAM
fc10-fc103fff : //vpci@0
  fc10-fc1000ff : :00:00.0
fc10-fc1000ff : ivshmem
  fc100100-fc1001ff : :00:01.0
fc100100-fc1001ff : ivshmem

However, I don't see any output (regarding the PCI) when I load the non-root 
inmate cell. 

2) I use #define PCI_CFG_BASE0xfc00 in inmates/lib/arm-common/pci.c, 
which matches the .pci_mmconfig_base = 0xfc00 in root cell config.

3) The output from the /sys/class/uio/uio0/maps[0,1] is:

cat /sys/class/uio/uio0/maps/map0/addr 
0xfc10
cat /sys/class/uio/uio0/maps/map0/offset 
0x0
cat /sys/class/uio/uio0/maps/map0/size
0x1000
cat /sys/class/uio/uio0/maps/map0/name
registers

cat /sys/class/uio/uio0/maps/map1/addr  
0x00080040
cat /sys/class/uio/uio0/maps/map1/offset 
0x0
cat /sys/class/uio/uio0/maps/map1/size
0x0010
cat /sys/class/uio/uio0/maps/map1/name
shmem

Sorry for the long email, but I guess with this detailed info it may help 
finding the error.

> Il giorno giovedì 12 aprile 2018 08:51:39 UTC+2, Claudio Scordino ha scritto:
> > Hi Giovani,
> > 
> > 
> > 
> > 
> > 2018-04-12 8:01 GMT+02:00 Jan Kiszka <jan.k...@siemens.com>:
> > On 2018-04-11 19:40, Giovani Gracioli wrote:
> > 
> > > Here is the output of the unhandled data read:
> > 
> > >
> > 
> > > Unhandled data read at 0xfc10(2)
> > 
> > >
> > 
> > > FATAL: unhandled trap (exception class 0x24)
> > 
> > > Cell state before exception:
> > 
> > >  pc: 1828   lr: 15f0 spsr: 6005     EL1
> > 
> > >  sp: 3f30  esr: 24 1 146
> > 
> > >  x0: fc10   x1:    x2: 0002
> > 
> > >  x3: fc00   x4:    x5: 
> > 
> > >  x6: 1000   x7:    x8: 
> > 
> > >  x9:   x10:   x11: 
> > 
> > > x12:   x13:   x14: 
> > 
> > > x15:   x16:   x17: 
> > 
> > > x18:   x19: 1000  x20: 
> > 
> > > x21: 1af4  x22: 1110  x23: 1000
> > 
> > > x24: 2660  x25:   x26: 
> > 
> > > x27:   x28:   x29: 
> > 
> > >
> > 
> &

Re: Share memory among cells on arm64

[Giovani Gracioli]

Here is the output of the unhandled data read:

Unhandled data read at 0xfc10(2)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: 1828   lr: 15f0 spsr: 6005 EL1
 sp: 3f30  esr: 24 1 146
 x0: fc10   x1:    x2: 0002
 x3: fc00   x4:    x5: 
 x6: 1000   x7:    x8: 
 x9:   x10:   x11: 
x12:   x13:   x14: 
x15:   x16:   x17: 
x18:   x19: 1000  x20: 
x21: 1af4  x22: 1110  x23: 1000
x24: 2660  x25:   x26: 
x27:   x28:   x29: 

Parking CPU 3 (Cell: "gic-demo-ivshmem")

Am I missing something in the configuration?

> Hi Jan,
> 
> Thanks for the reply. I am using the uio driver and when I load it, the pci 
> devices become enabled. I suppose this part is ok:
> 
> lspci -v after the driver is loaded:
> 
> 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> Subsystem: Red Hat, Inc Inter-VM shared memory
> Flags: bus master, fast devsel, latency 0, IRQ 38
> Memory at fc10 (64-bit, non-prefetchable) [size=256]
> Kernel driver in use: uio_ivshmem
> 
> 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> Subsystem: Red Hat, Inc Inter-VM shared memory
> Flags: bus master, fast devsel, latency 0, IRQ 39
> Memory at fc100100 (64-bit, non-prefetchable) [size=256]
> Kernel driver in use: uio_ivshmem
> 
> I got the code available here 
> (https://github.com/evidence/linux-jailhouse-tx1) that has an ivshmem demo on 
> arm. I basically copied and pasted the pci_read_config and pci_write_config 
> functions in the inmates/lib/arm-common/pci.c (had to change some makefiles 
> to compile the code because of the pci.c file).
> 
> However, I am getting an Unhandled data read at 0xfc10, which is the 
> address of the 00:00.0 device, during the execution of the function 
> pci_find_device().
> 
> If a decrease the while value test in the pci_find_device() function (bdf < 
> 0x1), it does not find any PCI device (obviously) and there is no error.
> 
> start_bdf is initially defined as 0xfc00, as configured in the root cell 
> (.pci_mmconfig_base = 0xfc00).
> 
> Any suggestion?
> 
> Giovani
> 
> > On 2018-04-10 16:39, Giovani Gracioli wrote:
> > > Updating:
> > > 
> > > I added 
> > > 
> > > .num_msix_vectors = 1,
> > > .iommu = 1, 
> > > 
> > > to the root cell .pci_devices config, wrote a simple linux program to 
> > > write to the shared memory (0x80050) and a simple inmate cell code to 
> > > read from the shared memory. It was able to read the values that were 
> > > written.
> > > 
> > > The next step is how to generate interrupts among the cells. I tried to 
> > > use the pci functions (the same used in the ivshmem-demo) on the arm64 
> > > but got an error. The lib/pci.c is not compiled by arm64.
> > > 
> > > How should I discover the pci device and send/receive interrupts on arm64?
> > > 
> > 
> > ivshmem demo inmate for ARM/ARM64 is still work in progress. Henning, is
> > there any intermediate step we can point to?
> > 
> > You could start your tests between two Linux cells, meanwhile, using the
> > uio driver.
> > 
> > Jan
> > 
> > -- 
> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

Hi Jan,

Thanks for the reply. I am using the uio driver and when I load it, the pci 
devices become enabled. I suppose this part is ok:

lspci -v after the driver is loaded:

00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0, IRQ 38
Memory at fc10 (64-bit, non-prefetchable) [size=256]
Kernel driver in use: uio_ivshmem

00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0, IRQ 39
Memory at fc100100 (64-bit, non-prefetchable) [size=256]
Kernel driver in use: uio_ivshmem

I got the code available here (https://github.com/evidence/linux-jailhouse-tx1) 
that has an ivshmem demo on arm. I basically copied and pasted the 
pci_read_config and pci_write_config functions in the 
inmates/lib/arm-common/pci.c (had to change some makefiles to compile the code 
because of the pci.c file).

However, I am getting an Unhandled data read at 0xfc10, which is the 
address of the 00:00.0 device, during the execution of the function 
pci_find_device().

If a decrease the while value test in the pci_find_device() function (bdf < 
0x1), it does not find any PCI device (obviously) and there is no error.

start_bdf is initially defined as 0xfc00, as configured in the root cell 
(.pci_mmconfig_base = 0xfc00).

Any suggestion?

Giovani

> On 2018-04-10 16:39, Giovani Gracioli wrote:
> > Updating:
> > 
> > I added 
> > 
> > .num_msix_vectors = 1,
> > .iommu = 1, 
> > 
> > to the root cell .pci_devices config, wrote a simple linux program to write 
> > to the shared memory (0x80050) and a simple inmate cell code to read 
> > from the shared memory. It was able to read the values that were written.
> > 
> > The next step is how to generate interrupts among the cells. I tried to use 
> > the pci functions (the same used in the ivshmem-demo) on the arm64 but got 
> > an error. The lib/pci.c is not compiled by arm64.
> > 
> > How should I discover the pci device and send/receive interrupts on arm64?
> > 
> 
> ivshmem demo inmate for ARM/ARM64 is still work in progress. Henning, is
> there any intermediate step we can point to?
> 
> You could start your tests between two Linux cells, meanwhile, using the
> uio driver.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

Updating:

I added 

.num_msix_vectors = 1,
.iommu = 1, 

to the root cell .pci_devices config, wrote a simple linux program to write to 
the shared memory (0x80050) and a simple inmate cell code to read from the 
shared memory. It was able to read the values that were written.

The next step is how to generate interrupts among the cells. I tried to use the 
pci functions (the same used in the ivshmem-demo) on the arm64 but got an 
error. The lib/pci.c is not compiled by arm64.

How should I discover the pci device and send/receive interrupts on arm64?

Thanks
Giovani

> Thanks for the answers.
> 
> So, I have changed the config. In the root cell I have the following now:
> 
> /* IVSHMEM shared memory region for 00:00.0 */ {
>   .phys_start = 0x80040,
>   .virt_start = 0x80040,
>   .size = 0x10,
>   .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> JAILHOUSE_MEM_ROOTSHARED, //added MEM_ROOTSHARED
>   },
> 
> .pci_devices = {
>   /* 00:00.0 */ {
>   .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
>   .bdf = 0 << 3,
>   .bar_mask = {
>   0xff00, 0x, 0x,
>   0x, 0x, 0x,
>   },
>   .shmem_region = 3,
>   .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED, 
> //changed from JAILHOUSE_SHMEM_PROTO_VETH to JAILHOUSE_SHMEM_PROTO_UNDEFINED
>   },
>},
> 
> In the inmate cell, I have the following:
> 
> .mem_regions = {
>   ..
> .
> /* IVSHMEM shared memory region for 00:00.0 */ {
>   .phys_start = 0x80040,
>   .virt_start = 0x80040,
>   .size = 0x10,
>   .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> JAILHOUSE_MEM_ROOTSHARED, //added JAILHOUSE_MEM_ROOTSHARED
>   },
>   },
> 
> Considering just the configuration of the cells, are they correct?
> 
> So correct me if I am wrong. I will need to write a driver in the root-cell 
> (Linux) to send and receive interrupts to/from the shared ivshmem region. The 
> mapped vpci region contains only the registers of the device:
> 
> 00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
> Subsystem: Red Hat, Inc Inter-VM shared memory
> Flags: fast devsel, IRQ 38
> Memory at fc10 (64-bit, non-prefetchable) [size=256]
> 
> In this case, the address 0xfc10.
> 
> Is the address (0x80040) that I configured in the cell config file 
> actually the payload memory region?
> 
> Also, in the another inmate, how can I access the mapped region? I noticed 
> that there is an ivshmem-demo for x86, but there isn't for arm64. 
> 
> Thanks
> 
> > Hi,
> > 
> > what you see there are the registers of the ivshmem device, not the
> > payload memory region. You will have to map the paddr you have in your
> > cell-config. That paddr can be found in the config space as well, is
> > just not a regular PCI BAR.
> > 
> > I would suggest to write a uio driver and base it on
> > https://github.com/henning-schild-work/ivshmem-guest-code/tree/jailhouse
> > 
> > Here is the bit that finds the region
> > https://github.com/henning-schild-work/ivshmem-guest-code/blob/jailhouse/kernel_module/uio/uio_ivshmem.c#L96
> > 
> > Also see:
> > https://github.com/henning-schild-work/ivshmem-guest-code/blob/jailhouse/README.jailhouse
> > 
> > And you should not use JAILHOUSE_SHMEM_PROTO_VETH because you are not
> > running a network-device on top of your shmem. Use _UNDEFINED or
> > _CUSTOM instead.
> > 
> > Henning
> > 
> > Am Tue, 3 Apr 2018 10:47:31 -0700
> > schrieb Giovani Gracioli <>:
> > 
> > > Just another info, after enabling the root cell, I can see the
> > > virtual pci devices with lspci -v:
> > > 
> > > 00:00.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> > > Subsystem: Red Hat, Inc Inter-VM shared memory
> > > Flags: fast devsel
> > > Memory at fc10 (64-bit, non-prefetchable) [disabled]
> > > [size=256] Memory at  (64-bit, non-prefetchable)
> > > [disabled] Capabilities: [50] MSI-X: Enable- Count=1 Masked-
> > > 
> > > 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> > > Subsystem: Red Hat, Inc Inter-VM shared m

Re: Share memory among cells on arm64

[Giovani Gracioli]

Thanks for the answers.

So, I have changed the config. In the root cell I have the following now:

/* IVSHMEM shared memory region for 00:00.0 */ {
.phys_start = 0x80040,
.virt_start = 0x80040,
.size = 0x10,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
JAILHOUSE_MEM_ROOTSHARED, //added MEM_ROOTSHARED
},

.pci_devices = {
/* 00:00.0 */ {
.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
.bdf = 0 << 3,
.bar_mask = {
0xff00, 0x, 0x,
0x, 0x, 0x,
},
.shmem_region = 3,
.shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED, 
//changed from JAILHOUSE_SHMEM_PROTO_VETH to JAILHOUSE_SHMEM_PROTO_UNDEFINED
},
   },

In the inmate cell, I have the following:

.mem_regions = {
..
.
/* IVSHMEM shared memory region for 00:00.0 */ {
.phys_start = 0x80040,
.virt_start = 0x80040,
.size = 0x10,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
JAILHOUSE_MEM_ROOTSHARED, //added JAILHOUSE_MEM_ROOTSHARED
},
},

Considering just the configuration of the cells, are they correct?

So correct me if I am wrong. I will need to write a driver in the root-cell 
(Linux) to send and receive interrupts to/from the shared ivshmem region. The 
mapped vpci region contains only the registers of the device:

00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: fast devsel, IRQ 38
Memory at fc10 (64-bit, non-prefetchable) [size=256]

In this case, the address 0xfc10.

Is the address (0x80040) that I configured in the cell config file actually 
the payload memory region?

Also, in the another inmate, how can I access the mapped region? I noticed that 
there is an ivshmem-demo for x86, but there isn't for arm64. 

Thanks

> Hi,
> 
> what you see there are the registers of the ivshmem device, not the
> payload memory region. You will have to map the paddr you have in your
> cell-config. That paddr can be found in the config space as well, is
> just not a regular PCI BAR.
> 
> I would suggest to write a uio driver and base it on
> https://github.com/henning-schild-work/ivshmem-guest-code/tree/jailhouse
> 
> Here is the bit that finds the region
> https://github.com/henning-schild-work/ivshmem-guest-code/blob/jailhouse/kernel_module/uio/uio_ivshmem.c#L96
> 
> Also see:
> https://github.com/henning-schild-work/ivshmem-guest-code/blob/jailhouse/README.jailhouse
> 
> And you should not use JAILHOUSE_SHMEM_PROTO_VETH because you are not
> running a network-device on top of your shmem. Use _UNDEFINED or
> _CUSTOM instead.
> 
> Henning
> 
> Am Tue, 3 Apr 2018 10:47:31 -0700
> schrieb Giovani Gracioli <>:
> 
> > Just another info, after enabling the root cell, I can see the
> > virtual pci devices with lspci -v:
> > 
> > 00:00.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Flags: fast devsel
> > Memory at fc10 (64-bit, non-prefetchable) [disabled]
> > [size=256] Memory at  (64-bit, non-prefetchable)
> > [disabled] Capabilities: [50] MSI-X: Enable- Count=1 Masked-
> > 
> > 00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
> > Subsystem: Red Hat, Inc Inter-VM shared memory
> > Flags: fast devsel
> > Memory at fc100100 (64-bit, non-prefetchable) [disabled]
> > [size=256]
> > 
> > 
> > 
> > > Hello,
> > > 
> > > I would like to share buffers among cells on arm64 (Xilinx
> > > ultrascale+). The documentation suggests the use of the ivshmem.
> > > 
> > > In order to use ivshmem, I changed the root cell config as follows
> > > (it is based on the zynqmp-zcu102.c original file):
> > > 
> > > - In the .mem_regions:
> > > 
> > > /* IVSHMEM shared memory region for 00:00.0 */ {
> > >   .phys_start = 0x80040,
> > >   .virt_start = 0x80040,
> > >   .size = 0x10,
> > >   .flags = JAILHOUSE_MEM_READ |
> > > JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, //added
> > > JAILHOUSE_MEM_RO

How to send an IPI on arm64 from the root-cell to an inmate cell

[Giovani Gracioli]

Hello,

I would like to send an IPI from the root cell (Linux) to an inmate cell on 
arm64. I know Jailhouse isolates the physical cores and would not allow to send 
an IPI to a CPU that does not belong to the cell. However, I would like to test 
that anyway. 

I have a kernel module (Linux running on the root cell) that sends an IPI to a 
core. As I do not have a deep knowledge in Jailhouse, it is not clear to me how 
Jailhouse intercepts or/and avoids the IPI, when it is sent to a core that does 
not belong to the root cell.

Where in the code Jailhouse checks this? What should I change to allow the IPI 
to be sent?

On the inmate side, I am running the gic-demo.c and I am just printing the irqn 
in the handle_IRQ function. Will the IPI be delivered to the handle_IRQ 
function as well?

Best regards,

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Share memory among cells on arm64

[Giovani Gracioli]

Just another info, after enabling the root cell, I can see the virtual pci 
devices with lspci -v:

00:00.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: fast devsel
Memory at fc10 (64-bit, non-prefetchable) [disabled] [size=256]
Memory at  (64-bit, non-prefetchable) [disabled]
Capabilities: [50] MSI-X: Enable- Count=1 Masked-

00:01.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: fast devsel
Memory at fc100100 (64-bit, non-prefetchable) [disabled] [size=256]



> Hello,
> 
> I would like to share buffers among cells on arm64 (Xilinx ultrascale+). The 
> documentation suggests the use of the ivshmem.
> 
> In order to use ivshmem, I changed the root cell config as follows (it is 
> based on the zynqmp-zcu102.c original file):
> 
> - In the .mem_regions:
> 
> /* IVSHMEM shared memory region for 00:00.0 */ {
>   .phys_start = 0x80040,
>   .virt_start = 0x80040,
>   .size = 0x10,
>   .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
> JAILHOUSE_MEM_ROOTSHARED, //added JAILHOUSE_MEM_ROOTSHARED
>   },
> 
> - In the .pci_devices:
> 
> /* 00:00.0 */ {
>   .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
>   .bdf = 0 << 3,
>   .bar_mask = {
>   0xff00, 0x, 0x,
>   0x, 0x, 0x,
>   },
>   .shmem_region = 3,
>   .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
> .num_msix_vectors = 1,
> .iommu = 1,
>   },
> 
> Then, I load the cell:
> 
> jailhouse enable zynqmp-zcu102-ivshmem.cell 
> 
> Initializing Jailhouse hypervisor v0.8 (37-g1fa9001) on CPU 1
> Code location: 0xc0200060
> Page pool usage after early setup: mem 33/995, remap 64/131072
> Initializing processors:
>  CPU 1... OK
>  CPU 2... OK
>  CPU 0... OK
>  CPU 3... OK
> Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> Page pool usage after late setup: mem 42/995, remap 69/131072
> Activating hypervisor
> 
> Then, I wrote a simple user-space program that maps the virtual PCI ivshmem 
> region to user using mmap:
> 
> memfd = open("/dev/mem", O_RDWR | O_SYNC);
> 
> mapped_base = mmap(0, MEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, 
> 0xfc10 & ~MEM_MASK);
> 
> mapped_dev_base = mapped_base + (dev_base & MEM_MASK);
> 
> for(int i = 0; i < 8; i += 4) {
> *((volatile unsigned int *) (mapped_dev_base + i)) = 1;
> printf("Address: %p, Read valeu = %d\n", (void *)(mapped_dev_base + 
> i), *((unsigned int *) (mapped_dev_base + i)));
> }
> 
> When I ran this program in the root cell, I got the following Unhandled trap:
> 
> ./ivshmem_test 
> /dev/mem opened.
> Memory mapped at address 0x7f8dd87000.
> Unhandled data write at 0xfc10(4)
> 
> FATAL: unhandled trap (exception class 0x24)
> Cell state before exception:
>  pc: 00400904   lr: 004008d8 spsr: 8000 EL0
>  sp: 007fceff8df0  esr: 24 1 1810046
>  x0: 007f8dd87000   x1: 0001   x2: 0001
>  x3:    x4: 40100401   x5: 5404
>  x6: 1aec1037   x7:    x8: 0040
>  x9: ff80ffc8  x10: 007fceff8df0  x11: 007fceff8df0
> x12: 03f3  x13:   x14: 
> x15: 007f8dd8ecc0  x16:   x17: 007f8dc53240
> x18: 0a03  x19: 004009a8  x20: 
> x21:   x22:   x23: 
> x24:   x25:   x26: 
> x27:   x28:   x29: 007fceff8df0
> 
> Parking CPU 2 (Cell: "ZynqMP-ZCU102")
> 
> Does that make sense? Am I missing something in the root cell config or the 
> user-space program is wrong?
> 
> Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Share memory among cells on arm64

[Giovani Gracioli]

Hello,

I would like to share buffers among cells on arm64 (Xilinx ultrascale+). The 
documentation suggests the use of the ivshmem.

In order to use ivshmem, I changed the root cell config as follows (it is based 
on the zynqmp-zcu102.c original file):

- In the .mem_regions:

/* IVSHMEM shared memory region for 00:00.0 */ {
.phys_start = 0x80040,
.virt_start = 0x80040,
.size = 0x10,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | 
JAILHOUSE_MEM_ROOTSHARED, //added JAILHOUSE_MEM_ROOTSHARED
},

- In the .pci_devices:

/* 00:00.0 */ {
.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
.bdf = 0 << 3,
.bar_mask = {
0xff00, 0x, 0x,
0x, 0x, 0x,
},
.shmem_region = 3,
.shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
.num_msix_vectors = 1,
.iommu = 1,
},

Then, I load the cell:

jailhouse enable zynqmp-zcu102-ivshmem.cell 

Initializing Jailhouse hypervisor v0.8 (37-g1fa9001) on CPU 1
Code location: 0xc0200060
Page pool usage after early setup: mem 33/995, remap 64/131072
Initializing processors:
 CPU 1... OK
 CPU 2... OK
 CPU 0... OK
 CPU 3... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 42/995, remap 69/131072
Activating hypervisor

Then, I wrote a simple user-space program that maps the virtual PCI ivshmem 
region to user using mmap:

memfd = open("/dev/mem", O_RDWR | O_SYNC);

mapped_base = mmap(0, MEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, 
0xfc10 & ~MEM_MASK);

mapped_dev_base = mapped_base + (dev_base & MEM_MASK);

for(int i = 0; i < 8; i += 4) {
*((volatile unsigned int *) (mapped_dev_base + i)) = 1;
printf("Address: %p, Read valeu = %d\n", (void *)(mapped_dev_base + i), 
*((unsigned int *) (mapped_dev_base + i)));
}

When I ran this program in the root cell, I got the following Unhandled trap:

./ivshmem_test 
/dev/mem opened.
Memory mapped at address 0x7f8dd87000.
Unhandled data write at 0xfc10(4)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: 00400904   lr: 004008d8 spsr: 8000 EL0
 sp: 007fceff8df0  esr: 24 1 1810046
 x0: 007f8dd87000   x1: 0001   x2: 0001
 x3:    x4: 40100401   x5: 5404
 x6: 1aec1037   x7:    x8: 0040
 x9: ff80ffc8  x10: 007fceff8df0  x11: 007fceff8df0
x12: 03f3  x13:   x14: 
x15: 007f8dd8ecc0  x16:   x17: 007f8dc53240
x18: 0a03  x19: 004009a8  x20: 
x21:   x22:   x23: 
x24:   x25:   x26: 
x27:   x28:   x29: 007fceff8df0

Parking CPU 2 (Cell: "ZynqMP-ZCU102")

Does that make sense? Am I missing something in the root cell config or the 
user-space program is wrong?

Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

Found the error. I removed this line from the config.h:

#define CONFIG_INMATE_BASE 0x9000

The entry address is correct now:

 <__reset_entry>:
   0:   58007c40ldr x0, f88 <vectors+0x788>
   4:   d518c000msr vbar_el1, x0
   8:   58007c40ldr x0, f90 <vectors+0x790>
   c:   911fmov sp, x0
  10:   d2a00600mov x0, #0x30   // #3145728
  14:   d5181040msr cpacr_el1, x0
  18:   d51b423fmsr daif, xzr
  1c:   d5033fdfisb
  20:   1400047db   1214 

Thanks!


> The __reset_entry is in the wrong memory address:
> 
> 9000 <__reset_entry>:
> 9000: 58007c40ldr x0, 9f88 <vectors+0x788>
> 9004: d518c000msr vbar_el1, x0
> 9008: 58007c40ldr x0, 9f90 <vectors+0x790>
> 900c: 911fmov sp, x0
> 9010: d2a00600mov x0, #0x30   // 
> #3145728
> 9014: d5181040msr cpacr_el1, x0
> 9018: d51b423fmsr daif, xzr
> 901c: d5033fdfisb
> 9020: 14000470b   900011e0 
> 
> 9f88: 9800.word   0x9800
> 9f8c: .word   0x
> 9f90: 90004000.word   0x90004000
> 9f94: .word   0x
> 
> 
> 900011e0 :
> 900011e0: a9be53f3stp x19, x20, [sp,#-32]!
> 900011e4: b000adrpx0, 90002000 
> 900011e8: 91038800add x0, x0, #0xe2
> 900011ec: 9013adrpx19, 90001000 
> 900011f0: f9000bfestr x30, [sp,#16]
> 900011f4: 94c0bl  900014f4 
> 900011f8: 9000adrpx0, 90001000 
> 900011fc: 9104d000add x0, x0, #0x134
> 90001200: 942abl  900012a8 
> 
> What is your toolchain version?
> 
> Mine is:
> 
> aarch64-linux-gnu-gcc --version
> aarch64-linux-gnu-gcc (Linaro GCC Snapshot 6.2-2016.11) 6.2.1 20161114
> Copyright (C) 2016 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> > On 2017-12-12 16:20, Giovani Gracioli wrote:
> > > Hello,
> > > 
> > > I am trying to run the Jailhouse gic-demo on the Xilinx ZCU102 dev board. 
> > > However, when I start the demo, I got an unhandled trap. Here are my 
> > > steps:
> > > 
> > > 1) modprobe jailhouse
> > > 
> > > syslog output:
> > > 
> > > Dec 11 18:39:54 linaro-gnome kernel: [   25.666781] jailhouse: loading 
> > > out-of-tree module taints kernel.
> > > 
> > > 2) jailhouse enable jailhouse/configs/zynqmp-zcu102.cell
> > > 
> > > Initializing Jailhouse hypervisor v0.7 (157-g2149299-dirty) on CPU 1
> > > Code location: 0xc0200060
> > > Page pool usage after early setup: mem 33/979, remap 64/131072
> > > Initializing processors:
> > >  CPU 1... OK
> > >  CPU 0... OK
> > >  CPU 2... OK
> > >  CPU 3... OK
> > > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> > > Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> > > Page pool usage after late setup: mem 42/979, remap 69/131072
> > > Activating hypervisor
> > > [   54.655832] OF: ERROR: Bad of_node_put() on /amba/xilinx_drm
> > > [   54.662352] xilinx-drm-dp fd4a.dp: failed to get phy lane
> > > [   54.668376] ahci-ceva fd0c.ahci: couldn't get PHY in node ahci: 
> > > -517
> > > 
> > > 3) jailhouse cell create jailhouse/configs/zynqmp-zcu102-gic-demo.cell
> > > Created cell "gic-demo"
> > > Page pool usage after cell creation: mem 54/979, remap 69/131072
> > > 
> > > Syslog output:
> > > 
> > > Dec 11 18:46:55 linaro-gnome kernel: [  446.433244] CPU3: shutdown
> > > Dec 11 18:46:55 linaro-gnome kernel: [  446.433251] psci: CPU3 killed.
> > > Dec 11 18:46:55 linaro-gnome kernel: [  446.473758] Created Jailhouse 
> > > cell "gic-demo
> > > 
> > > 4) jailhouse cell load gic-demo jailhouse/inmates/demos/arm64/gic-demo.bin
> > > 
> > > Output: Cell "gic-demo" can be loaded
> > > 
> > > 5)  jailhouse cell start gic-demo
> > > 
> > > The output below is repeated forever:
> > > 
> > > Parking CPU 3 (Cell: "gic-demo")

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

The __reset_entry is in the wrong memory address:

9000 <__reset_entry>:
9000:   58007c40ldr x0, 9f88 <vectors+0x788>
9004:   d518c000msr vbar_el1, x0
9008:   58007c40ldr x0, 9f90 <vectors+0x790>
900c:   911fmov sp, x0
9010:   d2a00600mov x0, #0x30   // 
#3145728
9014:   d5181040msr cpacr_el1, x0
9018:   d51b423fmsr daif, xzr
901c:   d5033fdfisb
9020:   14000470b   900011e0 

9f88:   9800.word   0x9800
9f8c:   .word   0x
9f90:   90004000.word   0x90004000
9f94:   .word   0x


900011e0 :
900011e0:   a9be53f3stp x19, x20, [sp,#-32]!
900011e4:   b000adrpx0, 90002000 
900011e8:   91038800add x0, x0, #0xe2
900011ec:   9013adrpx19, 90001000 
900011f0:   f9000bfestr x30, [sp,#16]
900011f4:   94c0bl  900014f4 
900011f8:   9000adrpx0, 90001000 
900011fc:   9104d000add x0, x0, #0x134
90001200:   942abl  900012a8 

What is your toolchain version?

Mine is:

aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Linaro GCC Snapshot 6.2-2016.11) 6.2.1 20161114
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

> On 2017-12-12 16:20, Giovani Gracioli wrote:
> > Hello,
> > 
> > I am trying to run the Jailhouse gic-demo on the Xilinx ZCU102 dev board. 
> > However, when I start the demo, I got an unhandled trap. Here are my steps:
> > 
> > 1) modprobe jailhouse
> > 
> > syslog output:
> > 
> > Dec 11 18:39:54 linaro-gnome kernel: [   25.666781] jailhouse: loading 
> > out-of-tree module taints kernel.
> > 
> > 2) jailhouse enable jailhouse/configs/zynqmp-zcu102.cell
> > 
> > Initializing Jailhouse hypervisor v0.7 (157-g2149299-dirty) on CPU 1
> > Code location: 0xc0200060
> > Page pool usage after early setup: mem 33/979, remap 64/131072
> > Initializing processors:
> >  CPU 1... OK
> >  CPU 0... OK
> >  CPU 2... OK
> >  CPU 3... OK
> > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> > Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> > Page pool usage after late setup: mem 42/979, remap 69/131072
> > Activating hypervisor
> > [   54.655832] OF: ERROR: Bad of_node_put() on /amba/xilinx_drm
> > [   54.662352] xilinx-drm-dp fd4a.dp: failed to get phy lane
> > [   54.668376] ahci-ceva fd0c.ahci: couldn't get PHY in node ahci: -517
> > 
> > 3) jailhouse cell create jailhouse/configs/zynqmp-zcu102-gic-demo.cell
> > Created cell "gic-demo"
> > Page pool usage after cell creation: mem 54/979, remap 69/131072
> > 
> > Syslog output:
> > 
> > Dec 11 18:46:55 linaro-gnome kernel: [  446.433244] CPU3: shutdown
> > Dec 11 18:46:55 linaro-gnome kernel: [  446.433251] psci: CPU3 killed.
> > Dec 11 18:46:55 linaro-gnome kernel: [  446.473758] Created Jailhouse cell 
> > "gic-demo
> > 
> > 4) jailhouse cell load gic-demo jailhouse/inmates/demos/arm64/gic-demo.bin
> > 
> > Output: Cell "gic-demo" can be loaded
> > 
> > 5)  jailhouse cell start gic-demo
> > 
> > The output below is repeated forever:
> > 
> > Parking CPU 3 (Cell: "gic-demo")
> > Unhandled data write at 0x90003fe0(1)
> > 
> > FATAL: unhandled trap (exception class 0x24)
> > Cell state before exception:
> >  pc: 1214   lr:  spsr: 0005 EL1
> 
> If we are looking at the same code and disassembly (I just rebuild mine
> from next), this address is the first line in inmate_main and stores
> something on the stack.
> 
> >  sp: 90004000  esr: 24 1 045
> 
> But the stack pointer is totally off. It should be 0x4000, see the code,
> and it should be loaded from a constant variable (stack_top) by
> __reset_entry that is located at 0xf90 in my code and correctly contains
> 0x4000 in my binary.
> 
> Please check your compilation output (objdump -d), maybe also try a
> different toolchain if that is broken.
> 
> Here are the relevant parts as they should look like:
> 
>  <__reset_entry>:
>0:   58007c40ldr x0, f88 <vectors+0x788>
>4:   d518c000  

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

0x90003fe0 is not listed (mapped?).

$ cat /proc/iomem 

-7fff : System RAM
  0008-00d7 : Kernel code
  00e8-00ffcfff : Kernel data
fc00-fc0f : PCI ECAM
fc10-fc103fff : //vpci@0
  fc10-fc1000ff : :00:00.0
  fc100100-fc1001ff : :00:01.0
fd50-fd500fff : /amba/dma@fd50
fd51-fd510fff : /amba/dma@fd51
fd52-fd520fff : /amba/dma@fd52
fd53-fd530fff : /amba/dma@fd53
fd54-fd540fff : /amba/dma@fd54
fd55-fd550fff : /amba/dma@fd55
fd56-fd560fff : /amba/dma@fd56
fd57-fd570fff : /amba/dma@fd57
fd6e9000-fd6edfff : /amba/cci@fd6e/pmu@9000
ff00-ff000fff : xuartps
ff01-ff010fff : xuartps
ff02-ff020fff : /amba/i2c@ff02
ff03-ff030fff : /amba/i2c@ff03
ff07-ff070fff : /amba/can@ff07
ff0a-ff0a0fff : /amba/gpio@ff0a
ff0e-ff0e0fff : /amba/ethernet@ff0e
ff0f-ff0f0fff : /amba/spi@ff0f
ff17-ff170fff : /amba/sdhci@ff17
ffa6-ffa600ff : /amba/rtc@ffa6
8-8003f : Jailhouse hypervisor
84000-877ff : System RAM


Em terça-feira, 12 de dezembro de 2017 13:34:52 UTC-5, J. Kiszka  escreveu:
> On 2017-12-12 19:18, Giovani Gracioli wrote:
> > After I rebooted, the error is back. 
> > 
> 
> Then check what is behind 0x90003fe0 (/proc/iomem) and possibly add some
> permission to the root cell config. It might be a device we didn't use,
> thus didn't cover in the upstream config.
> 
> Jan
> 
> >> On 2017-12-12 18:55, Giovani Gracioli wrote:
> >>> Thanks Jan.
> >>>
> >>> Recompiled with kernel 4.15.4 (the last stable from mainline) and the 
> >>> error disappeared. 
> >>>
> >>> However, I am not able to see the gic-demo output. I am connected in the 
> >>> terminal /dev/ttyUSB0 (ttyPS0). There are four ttyUSB[0-3] mounted.
> >>>
> >>> Where should gic-demo be printing to?
> >>
> >> On the one that is behind the address 0xff01 - IIRC, the second UART
> >> line.
> >>
> >> Jan
> >>
> >> -- 
> >> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> >> Corporate Competence Center Embedded Linux
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

After I rebooted, the error is back. 

> On 2017-12-12 18:55, Giovani Gracioli wrote:
> > Thanks Jan.
> > 
> > Recompiled with kernel 4.15.4 (the last stable from mainline) and the error 
> > disappeared. 
> > 
> > However, I am not able to see the gic-demo output. I am connected in the 
> > terminal /dev/ttyUSB0 (ttyPS0). There are four ttyUSB[0-3] mounted.
> > 
> > Where should gic-demo be printing to?
> 
> On the one that is behind the address 0xff01 - IIRC, the second UART
> line.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

Thanks Jan.

Recompiled with kernel 4.15.4 (the last stable from mainline) and the error 
disappeared. 

However, I am not able to see the gic-demo output. I am connected in the 
terminal /dev/ttyUSB0 (ttyPS0). There are four ttyUSB[0-3] mounted.

Where should gic-demo be printing to?

Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

Em terça-feira, 12 de dezembro de 2017 10:20:09 UTC-5, Giovani Gracioli  
escreveu:
> Hello,
> 
> I am trying to run the Jailhouse gic-demo on the Xilinx ZCU102 dev board. 
> However, when I start the demo, I got an unhandled trap. Here are my steps:
> 
> 1) modprobe jailhouse
> 
> syslog output:
> 
> Dec 11 18:39:54 linaro-gnome kernel: [   25.666781] jailhouse: loading 
> out-of-tree module taints kernel.
> 
> 2) jailhouse enable jailhouse/configs/zynqmp-zcu102.cell
> 
> Initializing Jailhouse hypervisor v0.7 (157-g2149299-dirty) on CPU 1
> Code location: 0xc0200060
> Page pool usage after early setup: mem 33/979, remap 64/131072
> Initializing processors:
>  CPU 1... OK
>  CPU 0... OK
>  CPU 2... OK
>  CPU 3... OK
> Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> Page pool usage after late setup: mem 42/979, remap 69/131072
> Activating hypervisor
> [   54.655832] OF: ERROR: Bad of_node_put() on /amba/xilinx_drm
> [   54.662352] xilinx-drm-dp fd4a.dp: failed to get phy lane
> [   54.668376] ahci-ceva fd0c.ahci: couldn't get PHY in node ahci: -517
> 
> 3) jailhouse cell create jailhouse/configs/zynqmp-zcu102-gic-demo.cell
> Created cell "gic-demo"
> Page pool usage after cell creation: mem 54/979, remap 69/131072
> 
> Syslog output:
> 
> Dec 11 18:46:55 linaro-gnome kernel: [  446.433244] CPU3: shutdown
> Dec 11 18:46:55 linaro-gnome kernel: [  446.433251] psci: CPU3 killed.
> Dec 11 18:46:55 linaro-gnome kernel: [  446.473758] Created Jailhouse cell 
> "gic-demo
> 
> 4) jailhouse cell load gic-demo jailhouse/inmates/demos/arm64/gic-demo.bin
> 
> Output: Cell "gic-demo" can be loaded
> 
> 5)  jailhouse cell start gic-demo
> 
> The output below is repeated forever:
> 
> Parking CPU 3 (Cell: "gic-demo")
> Unhandled data write at 0x90003fe0(1)
> 
> FATAL: unhandled trap (exception class 0x24)
> Cell state before exception:
>  pc: 1214   lr:  spsr: 0005 EL1
>  sp: 90004000  esr: 24 1 045
>  x0: 0030   x1:    x2: 
>  x3:    x4:    x5: 
>  x6:    x7:    x8: 
>  x9:   x10:   x11: 
> x12:   x13:   x14: 
> x15:   x16:   x17: 
> x18:   x19:   x20: 
> x21:   x22:   x23: 
> x24:   x25:   x26: 
> x27:   x28:   x29: 
> 
> It seems the generated interrupt is not being correctly handled. The question 
> is why? Any ideas? 
> 
> The kernel I am using is Linux linaro-gnome 4.9.0 and Jailhouse v0.7.
> 
> Best
> Giovani

Quick correction: the current the Linux kernel is the Xilinx one, 4.9.0.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Running gic-demo on Xilinx ZCU102 (unhandled trap problem)

[Giovani Gracioli]

Hello,

I am trying to run the Jailhouse gic-demo on the Xilinx ZCU102 dev board. 
However, when I start the demo, I got an unhandled trap. Here are my steps:

1) modprobe jailhouse

syslog output:

Dec 11 18:39:54 linaro-gnome kernel: [   25.666781] jailhouse: loading 
out-of-tree module taints kernel.

2) jailhouse enable jailhouse/configs/zynqmp-zcu102.cell

Initializing Jailhouse hypervisor v0.7 (157-g2149299-dirty) on CPU 1
Code location: 0xc0200060
Page pool usage after early setup: mem 33/979, remap 64/131072
Initializing processors:
 CPU 1... OK
 CPU 0... OK
 CPU 2... OK
 CPU 3... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 42/979, remap 69/131072
Activating hypervisor
[   54.655832] OF: ERROR: Bad of_node_put() on /amba/xilinx_drm
[   54.662352] xilinx-drm-dp fd4a.dp: failed to get phy lane
[   54.668376] ahci-ceva fd0c.ahci: couldn't get PHY in node ahci: -517

3) jailhouse cell create jailhouse/configs/zynqmp-zcu102-gic-demo.cell
Created cell "gic-demo"
Page pool usage after cell creation: mem 54/979, remap 69/131072

Syslog output:

Dec 11 18:46:55 linaro-gnome kernel: [  446.433244] CPU3: shutdown
Dec 11 18:46:55 linaro-gnome kernel: [  446.433251] psci: CPU3 killed.
Dec 11 18:46:55 linaro-gnome kernel: [  446.473758] Created Jailhouse cell 
"gic-demo

4) jailhouse cell load gic-demo jailhouse/inmates/demos/arm64/gic-demo.bin

Output: Cell "gic-demo" can be loaded

5)  jailhouse cell start gic-demo

The output below is repeated forever:

Parking CPU 3 (Cell: "gic-demo")
Unhandled data write at 0x90003fe0(1)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
 pc: 1214   lr:  spsr: 0005 EL1
 sp: 90004000  esr: 24 1 045
 x0: 0030   x1:    x2: 
 x3:    x4:    x5: 
 x6:    x7:    x8: 
 x9:   x10:   x11: 
x12:   x13:   x14: 
x15:   x16:   x17: 
x18:   x19:   x20: 
x21:   x22:   x23: 
x24:   x25:   x26: 
x27:   x28:   x29: 

It seems the generated interrupt is not being correctly handled. The question 
is why? Any ideas? 

The kernel I am using is Linux linaro-gnome 4.9.0 and Jailhouse v0.7.

Best
Giovani

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.