Re: Tested x86-64 SBC or mini PC

[Chung-Fan Yang]

Thank you Jan for pointing out the old NUC will work.

For my application, newest processor might not necessary.

This might be good enough for me (if the NUC is still easily available.)

Yang
2021年4月17日土曜日 18:47:14 UTC+9 Jan Kiszka:

> On 17.04.21 10:07, Chung-Fan Yang wrote:
> > Hello,
> >
> > I have been developing on top of Jailhuose for quite some time.
> > Just become curious and have a question.
> >
> > Are there any officially tested x86-64 based SBC or mini PC?
> > Will a Intel NUC work?
> >
> > I do have a working Xeon Broadwell build at hand it it costly to make a
> > duplicate machine.
> > I did try to run on other commercial or self-built x86-64 PCs, but the
> > on-board hardware varies quite a lot and I did have a hard time getting
> > things to work from time to time.
> >
> > I would like to find a more cheap and stable platform for x86-64.
> >
> > Thank in advance for any comments or answers on this.
> >
>
> I'm not up-to-date with low-cost x86 boards. We have pre-integrated the
> older NUC6CAY in jailhouse-images, and similar Apollo Lake designs
> should be fine as well (though surely not as fast and low-jitter as Xeon
> targets).
>
> Maybe other users can share more recent experiences.
>
> Jan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/362fa4be-5085-49c5-8def-5224d7d91723n%40googlegroups.com.

Tested x86-64 SBC or mini PC

[Chung-Fan Yang]

Hello,

I have been developing on top of Jailhuose for quite some time.
Just become curious and have a question.

Are there any officially tested x86-64 based SBC or mini PC?
Will a Intel NUC work?

I do have a working Xeon Broadwell build at hand it it costly to make a 
duplicate machine.
I did try to run on other commercial or self-built x86-64 PCs, but the 
on-board hardware varies quite a lot and I did have a hard time getting 
things to work from time to time.

I would like to find a more cheap and stable platform for x86-64. 

Thank in advance for any comments or answers on this.


Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/2bd89d4f-6959-4701-b661-3d3e98a2e260n%40googlegroups.com.

Re: Jailhouse on R-Pi4 with Debian rootfs

[Chung-Fan Yang]

I made some progress.

I successful applied the device tree overlay and BL31 firmware.
Now, the hypervisor did see the reserved memory and starts loading with a 
mainline 5.9 PREEMPT_RT patched kernel.
(I did add the extra EXPORT_SYMBOLS for this to work.)
Not yet tested with any additional cell. 

Thank you guys for the hints on the device tree fix.

I also tried the jailhouse-image repository, that works fine too.

I would like to ask an additional question:

Why is it necessary to load a custom BL31 firmware instead of the in EEPROM 
one?
Is it for the PSCI CPU on/offlining?


---
Yang

2020年12月7日月曜日 16:17:44 UTC+9 Chung-Fan Yang:

>
>
> 2020年12月7日月曜日 15:14:50 UTC+9 j.kiszka...@gmail.com:
>
>> On 07.12.20 02:29, Chung-Fan Yang wrote: 
>> > Thanks you for the suggestion of using an approved image. 
>> > I will try it out. 
>> > 
>> > However, I really like to know the root cause and get the current 
>> Debian 
>> > setup working. 
>>
>> [1] officially supports the RPi4 on buster (despite the pain that 
>> brings, hope 5.10 improves the situation at bit). You should derive from 
>> that, specifically the DT overlay-based memory reservation which was 
>> fixed not so long ago. 
>>
>
> Thank you.
> I see the commit now.
> So basically I have to include an additional dtbo the the boot process, am 
> I right?
> Will it work only on the device-tree from rpi-firmware or also the in 
> kernel-tree one?
> I am currently using the in-tree device tree from  kernel 5.9 without any 
> overlay.
>
> By the way, by pain, what do you mean?
>
> ---
> Yang
>  
>
>>
>> For the kernel, I didn't do a rebase of your patch series yet. [2] is 
>> the latest upstream based queue. 
>>
>> Jan 
>>
>> [1] https://github.com/siemens/jailhouse-images 
>> [2] 
>>
>> http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse 
>>
>> > 
>> > Yang 
>> > 
>> > 
>> > 2020年12月7日月曜日 5:01:26 UTC+9 jsmo...@linuxfoundation.org: 
>> > 
>> > Check our GSoC project on Automotive Grade Linux: 
>> > 
>> https://git.automotivelinux.org/AGL/meta-agl-devel/tree/meta-agl-jailhouse 
>> > <
>> https://git.automotivelinux.org/AGL/meta-agl-devel/tree/meta-agl-jailhouse> 
>>
>> > 
>> > It can build for PI. 
>> > 
>> > An it can serve as inspiration for the values needed. 
>> > 
>> >   
>> > 
>> > 
>> > js 
>> > 
>> > Chung-Fan Yang  schrieb am So., 6. Dez. 2020, 
>> 17:34: 
>> > 
>> > Hi, 
>> > 
>> > I am working to get jailhouse work with my R-Pi4. 
>> > 
>> > I have been using a 64bit Debian buster rootfs with 
>> > a custom compiled 5.9 preempt-rt kernel. 
>> > 
>> > I have successful reserved >736M for jailhouse and inserted the 
>> > jailhouse.ko, but when I do jailhouse enable I got the following 
>> > error on uart. 
>> > 
>> > Any suggestions are appreciated. 
>> > 
>> > Initializing Jailhouse hypervisor v0.12 (223-g097bed0f) on CPU 1 
>> > Code location: 0xc0200800 
>> > Page pool usage after early setup: mem 39/994, remap 0/131072 
>> > Initializing processors: 
>> >  CPU 1... 
>> > FATAL: Unhandled HYP exception: synchronous abort from EL2 
>> >  pc: c0203864   lr: c0203850 spsr: 23c9 EL2 
>> >  sp: c0222e40  elr: c0203864  esr: 00 1 000 
>> >  x0: 8400   x1:    x2: 80003580 
>> >  x3: 0014   x4: 0002   x5: 0001 
>> >  x6: 0029   x7: c0219ec0   x8: 002a 
>> >  x9:   x10:   x11: 0001 
>> > x12: 0015  x13: 0001  x14: c0219000 
>> > x15: c0015040  x16: c020da50  x17: af45951e7518 
>> > x18: 0001  x19: c0222000  x20: c0219000 
>> > x21: c020  x22: c0219000  x23: 0001 
>> > x24: 0001  x25: c0222000  x26: c0223000 
>> > x27: c020f000  x28: c0218000  x29: c0222e40 
>> > 
>> > Hypervisor stack before exception Stopping CPU 1 (Cell: 
>> > "Raspberry-Pi4") 
>> > 
>> > PS. I did noticed there is a similar post, but it has no solution. 
>> > 
>> > -- 
>> > You received this message because you are 

Re: Jailhouse on R-Pi4 with Debian rootfs

[Chung-Fan Yang]

2020年12月7日月曜日 15:14:50 UTC+9 j.kiszka...@gmail.com:

> On 07.12.20 02:29, Chung-Fan Yang wrote: 
> > Thanks you for the suggestion of using an approved image. 
> > I will try it out. 
> > 
> > However, I really like to know the root cause and get the current Debian 
> > setup working. 
>
> [1] officially supports the RPi4 on buster (despite the pain that 
> brings, hope 5.10 improves the situation at bit). You should derive from 
> that, specifically the DT overlay-based memory reservation which was 
> fixed not so long ago. 
>

Thank you.
I see the commit now.
So basically I have to include an additional dtbo the the boot process, am 
I right?
Will it work only on the device-tree from rpi-firmware or also the in 
kernel-tree one?
I am currently using the in-tree device tree from  kernel 5.9 without any 
overlay.

By the way, by pain, what do you mean?

---
Yang
 

>
> For the kernel, I didn't do a rebase of your patch series yet. [2] is 
> the latest upstream based queue. 
>
> Jan 
>
> [1] https://github.com/siemens/jailhouse-images 
> [2] 
> http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse 
>
> > 
> > Yang 
> > 
> > 
> > 2020年12月7日月曜日 5:01:26 UTC+9 jsmo...@linuxfoundation.org: 
> > 
> > Check our GSoC project on Automotive Grade Linux: 
> > 
> https://git.automotivelinux.org/AGL/meta-agl-devel/tree/meta-agl-jailhouse 
> > <
> https://git.automotivelinux.org/AGL/meta-agl-devel/tree/meta-agl-jailhouse> 
>
> > 
> > It can build for PI. 
> > 
> > An it can serve as inspiration for the values needed. 
> > 
> >   
> > 
> > 
> > js 
> > 
> > Chung-Fan Yang  schrieb am So., 6. Dez. 2020, 
> 17:34: 
> > 
> > Hi, 
> > 
> > I am working to get jailhouse work with my R-Pi4. 
> > 
> > I have been using a 64bit Debian buster rootfs with 
> > a custom compiled 5.9 preempt-rt kernel. 
> > 
> > I have successful reserved >736M for jailhouse and inserted the 
> > jailhouse.ko, but when I do jailhouse enable I got the following 
> > error on uart. 
> > 
> > Any suggestions are appreciated. 
> > 
> > Initializing Jailhouse hypervisor v0.12 (223-g097bed0f) on CPU 1 
> > Code location: 0xc0200800 
> > Page pool usage after early setup: mem 39/994, remap 0/131072 
> > Initializing processors: 
> >  CPU 1... 
> > FATAL: Unhandled HYP exception: synchronous abort from EL2 
> >  pc: c0203864   lr: c0203850 spsr: 23c9 EL2 
> >  sp: c0222e40  elr: c0203864  esr: 00 1 000 
> >  x0: 8400   x1:    x2: 80003580 
> >  x3: 0014   x4: 0002   x5: 0001 
> >  x6: 0029   x7: c0219ec0   x8: 002a 
> >  x9:   x10:   x11: 0001 
> > x12: 0015  x13: 0001  x14: c0219000 
> > x15: c0015040  x16: c020da50  x17: af45951e7518 
> > x18: 0001  x19: c0222000  x20: c0219000 
> > x21: c020  x22: c0219000  x23: 0001 
> > x24: 0001  x25: c0222000  x26: c0223000 
> > x27: c020f000  x28: c0218000  x29: c0222e40 
> > 
> > Hypervisor stack before exception Stopping CPU 1 (Cell: 
> > "Raspberry-Pi4") 
> > 
> > PS. I did noticed there is a similar post, but it has no solution. 
> > 
> > -- 
> > You received this message because you are subscribed to the 
> > Google Groups "Jailhouse" group. 
> > To unsubscribe from this group and stop receiving emails from 
> > it, send an email to jailhouse-de...@googlegroups.com. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/jailhouse-dev/f0e8ee0a-e874-428e-a97c-91524477c7e1n%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/jailhouse-dev/f0e8ee0a-e874-428e-a97c-91524477c7e1n%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Jailhouse" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to jailhouse-de...@googlegroups.com 
> > <mailto:jailhouse-de...@googlegroups.com>. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/jailhouse-dev/126f7b0c-2fa6-4329-bb95-bddead90c9f7n%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/jailhouse-dev/126f7b0c-2fa6-4329-bb95-bddead90c9f7n%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/313f42e9-5551-4f77-afc1-498f355d3e95n%40googlegroups.com.

Re: Jailhouse on R-Pi4 with Debian rootfs

[Chung-Fan Yang]

 
Thanks you for the suggestion of using an approved image.
I will try it out.

However, I really like to know the root cause and get the current Debian 
setup working.

Yang


2020年12月7日月曜日 5:01:26 UTC+9 jsmo...@linuxfoundation.org:

> Check our GSoC project on Automotive Grade Linux:
> https://git.automotivelinux.org/AGL/meta-agl-devel/tree/meta-agl-jailhouse
>
> It can build for PI.
>
> An it can serve as inspiration for the values needed.
>
>  

>
> js
>
> Chung-Fan Yang  schrieb am So., 6. Dez. 2020, 17:34:
>
>> Hi,
>>
>> I am working to get jailhouse work with my R-Pi4.
>>
>> I have been using a 64bit Debian buster rootfs with 
>> a custom compiled 5.9 preempt-rt kernel.
>>
>> I have successful reserved >736M for jailhouse and inserted the 
>> jailhouse.ko, but when I do jailhouse enable I got the following error on 
>> uart.
>>
>> Any suggestions are appreciated.
>>
>> Initializing Jailhouse hypervisor v0.12 (223-g097bed0f) on CPU 1
>> Code location: 0xc0200800
>> Page pool usage after early setup: mem 39/994, remap 0/131072
>> Initializing processors:
>>  CPU 1... 
>> FATAL: Unhandled HYP exception: synchronous abort from EL2
>>  pc: c0203864   lr: c0203850 spsr: 23c9 EL2
>>  sp: c0222e40  elr: c0203864  esr: 00 1 000
>>  x0: 8400   x1:    x2: 80003580
>>  x3: 0014   x4: 0002   x5: 0001
>>  x6: 0029   x7: c0219ec0   x8: 002a
>>  x9:   x10:   x11: 0001
>> x12: 0015  x13: 0001  x14: c0219000
>> x15: c0015040  x16: c020da50  x17: af45951e7518
>> x18: 0001  x19: c0222000  x20: c0219000
>> x21: c020  x22: c0219000  x23: 0001
>> x24: 0001  x25: c0222000  x26: c0223000
>> x27: c020f000  x28: c0218000  x29: c0222e40
>>
>> Hypervisor stack before exception Stopping CPU 1 (Cell: "Raspberry-Pi4")
>>
>> PS. I did noticed there is a similar post, but it has no solution.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jailhouse" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jailhouse-de...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/jailhouse-dev/f0e8ee0a-e874-428e-a97c-91524477c7e1n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/jailhouse-dev/f0e8ee0a-e874-428e-a97c-91524477c7e1n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/126f7b0c-2fa6-4329-bb95-bddead90c9f7n%40googlegroups.com.

Jailhouse on R-Pi4 with Debian rootfs

[Chung-Fan Yang]

Hi,

I am working to get jailhouse work with my R-Pi4.

I have been using a 64bit Debian buster rootfs with 
a custom compiled 5.9 preempt-rt kernel.

I have successful reserved >736M for jailhouse and inserted the 
jailhouse.ko, but when I do jailhouse enable I got the following error on 
uart.

Any suggestions are appreciated.

Initializing Jailhouse hypervisor v0.12 (223-g097bed0f) on CPU 1
Code location: 0xc0200800
Page pool usage after early setup: mem 39/994, remap 0/131072
Initializing processors:
 CPU 1... 
FATAL: Unhandled HYP exception: synchronous abort from EL2
 pc: c0203864   lr: c0203850 spsr: 23c9 EL2
 sp: c0222e40  elr: c0203864  esr: 00 1 000
 x0: 8400   x1:    x2: 80003580
 x3: 0014   x4: 0002   x5: 0001
 x6: 0029   x7: c0219ec0   x8: 002a
 x9:   x10:   x11: 0001
x12: 0015  x13: 0001  x14: c0219000
x15: c0015040  x16: c020da50  x17: af45951e7518
x18: 0001  x19: c0222000  x20: c0219000
x21: c020  x22: c0219000  x23: 0001
x24: 0001  x25: c0222000  x26: c0223000
x27: c020f000  x28: c0218000  x29: c0222e40

Hypervisor stack before exception Stopping CPU 1 (Cell: "Raspberry-Pi4")

PS. I did noticed there is a similar post, but it has no solution.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/f0e8ee0a-e874-428e-a97c-91524477c7e1n%40googlegroups.com.

License of custom cell configs

[Chung-Fan Yang]

Hi, 

I want to ask a silly question.

Is there any restrictions on the cell configs source files, including those 
hand crafted and these generated by jailhouse command line tool ?

For a hand crafted configs, may I distributed them with the licenses as I 
wish?

On the other hand for generated configs, I noticed that they have BSD 
headers, so I must comply and following that, right?

BR,

Yang.


-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/d88a71e3-1269-42f6-93d4-6f66658e5da6o%40googlegroups.com.

Re: Ivshmem-2 driver for x86_64 root Linux

[Chung-Fan Yang]

> > 
> > I am curious that besides of the special ivshmem2 driver and patches to 
> > uio, are there any other significant changes to the kernel source? 
> > 
>
> You can check the commits: 
>   - uio ivshmem driver 
>   - ivshmem-net driver (lots of commits in queues/jailhouse, folded in 
> stable) 
>   - virtio-over-ivshmem (experimental, only in queues/jailhouse so far) 
>   - driver for debug console (only relevant for bring-ups) 
>   - some export-symbols, not needed with kallsyms workaround (as long as 
> that works) 
>   - some dts changes for marvell arm64 boards 
>
> > Because I am using stock 4.19 with PREEMPT_RT, i can run Nuttx and it 
> > works like a charm without any noticeable bugs. 
> > 
>
> It depends on the use case what would be missing, but things generally 
> work. At least booting. 
>
>
Thanks for the information.

Now I know how to port my old modified ivshmem driver now.

I am on x86_64 so I think I am good using the stock kernel with separately 
compiled driver.

One more question, 

As long as my user-space program don't write to the readonly region and 
trigger a fault in the hypervisor.

Without the uio readonly patch, things should work correctly, yes?

Thanks, 

Yang




 

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/7b537982-4415-44ba-a538-ffb599634633%40googlegroups.com.

Ivshmem-2 driver for x86_64 root Linux

[Chung-Fan Yang]

Hi, 

I am upgrading from v9.1 to most recent version of Jailhouse recently.

As the ivshmem version has changed, the old drivers and way of doing no 
longer works.

I am curious should I cherry-pick the commits on the jailhouse-linux git to 
my kernel source?

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse 

Or I can just use the Jailhouse enabled branches in

https://github.com/siemens/linux

Or both way it will work?

I am curious that besides of the special ivshmem2 driver and patches to 
uio, are there any other significant changes to the kernel source?

Because I am using stock 4.19 with PREEMPT_RT, i can run Nuttx and it works 
like a charm without any noticeable bugs.

BR,

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/0fabe986-8e7d-4905-89af-75ec7e4d20ff%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

2019年11月1日金曜日 16時36分16秒 UTC+9 Jan Kiszka:
>
> On 31.10.19 06:57, Chung-Fan Yang wrote: 
> > 
> > Interesting findings already, but I'm afraid we will need to dig 
> > deeper: 
> > Can you describe what all is part of your measured latency path? 
> > 
> > 
> > I measured using an oscillate scope and function generator. 
> > I am using MMIO GPIOs. The application calls a system call and waits for 
> > an interrupt on a certain GPIO. 
> > When I send a pulse to the GPIO, the IRQ handler release a semaphore, 
> > interm trigger the scheduler and wake-up the application, which send 
> > another pulse to another GPIO using MMIO. 
> > 
> > FG -> Serial -> APIC -> RTOS IRQ Hnd -> Scheduler -> Application -> 
> > Serial -> OSC 
> > 
> > The timing different of these 2 pulses are measured. 
> > 
> > Because of the waiting mechanism used, receiving the pulse involved the 
> > system call / semaphore / interrupt handling of the RTOS. 
> > On the other hand, sending doesn't use any of the RTOS feature. 
> > 
> > Do you just run code in guest mode or do you also trigger VM exits, 
> > e.g. to 
> > issue ivshmem interrupts to a remote side? 
> > 
> > 
> > I tried to instrument the system. 
> > So far there are no additional interrupts send, nor received during the 
> > whole process. 
> > VMExit do exist for EOI(systick and serial IRQ) and when I fiddle the 
> > TSC_deadline timer enable/disable bit of APIC MSR. 
> > The whole process is not related to any ivshmem operations. 
>
> Use x2APIC in your guest, and you will get rid of those VMexits (due to 
> xAPIC MMIO interception). But that's an unrelated optimization. 
>
>
Thanks for the hint.
I override the BIOS and enabled it.
There are no VM exits now.
 

> > 
> > Maybe you can sample some latencies along the critical path so that 
> > we have a better picture about  
> > 
> > where we lose time, overall or rather on specific actions. 
> > 
> > 
> > Basically, it is an overall slowdown. 
> > But code in the scheduler and application slowdown more than other 
> places. 
> > 
> > BTW, I tested the again with a partially working setup of  > 4.19/Jailhouse v0.11/old ivshmem2>. 
> > Currently, I cannot get my application running, due to some mystery, but 
> > I am observing some slowdown. 
> > Pinging the RTOS using ivshmem-net the RTT has about 2x latency: 
> >  * : ~0.060ms 
> >  * : ~0.130ms 
> > 
>
> Sound like as if we have some caching related problem. You could enable 
> access to the perf MSRs (small patch to the MSR bitmap in vmx.c) and 
> check if you see excessive cache misses in the counters. 
>
>
I am quite busy lately, so I might let this problem be as is and revisit it 
later.

I will update the thread when I made new discoveries.

Yang.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/780f0dbb-4fcf-4e0e-a247-8e9f2ff0ec76%40googlegroups.com.

Re: TSC Frequency

[Chung-Fan Yang]

I also noticed this.
I am working on a Broadwell machine with 2.2Ghz of core frequency.

Jailhouse get set the cpu_khz in comm region with the frequency probed by 
Linux during boot.
I got somthing like ~2.1997Ghz in that field.
The difference is so small that I don't have a good way to evaluate it.

It does sometimes make the process of maintain a proper systick frequency 
in custom RTOS painful.


Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/255261de-a630-4790-b91c-16c569248829%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

> Interesting findings already, but I'm afraid we will need to dig deeper: 
> Can you describe what all is part of your measured latency path? 


I measured using an oscillate scope and function generator.
I am using MMIO GPIOs. The application calls a system call and waits for an 
interrupt on a certain GPIO.
When I send a pulse to the GPIO, the IRQ handler release a semaphore, 
interm trigger the scheduler and wake-up the application, which send 
another pulse to another GPIO using MMIO.

FG -> Serial -> APIC -> RTOS IRQ Hnd -> Scheduler -> Application -> Serial 
-> OSC

The timing different of these 2 pulses are measured.

Because of the waiting mechanism used, receiving the pulse involved the 
system call / semaphore / interrupt handling of the RTOS.
On the other hand, sending doesn't use any of the RTOS feature.

Do you just run code in guest mode or do you also trigger VM exits, e.g. to 
> issue ivshmem interrupts to a remote side? 


I tried to instrument the system.
So far there are no additional interrupts send, nor received during the 
whole process.
VMExit do exist for EOI(systick and serial IRQ) and when I fiddle the 
TSC_deadline timer enable/disable bit of APIC MSR.
The whole process is not related to any ivshmem operations. 

Maybe you can sample some latencies along the critical path so that we have 
> a better picture about  
>
where we lose time, overall or rather on specific actions. 
>

Basically, it is an overall slowdown.
But code in the scheduler and application slowdown more than other places.

BTW, I tested the again with a partially working setup of .
Currently, I cannot get my application running, due to some mystery, but I 
am observing some slowdown.
Pinging the RTOS using ivshmem-net the RTT has about 2x latency:
 * : ~0.060ms
 * : ~0.130ms


Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/6705dfff-635e-4182-a684-86d0b861e94e%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

 
Alright, I did the bisect, checked the inmate library and PM.

I will summarize first, the fault seems like to be in the wip/ivshmem2.
P-state was already disabled (Silly me, forgot what I have already done.)
Code steal from inmate library seems fine.

Let me describe my setup first.
I am using the wip/ivshmem2, because my application favors unidirectional 
pipes and multiple interrupts between cells.
I have been using the v0.9.1 with an old version of ivshmem2(15ee5278), 
which has lstate/rstate, etc.

When I need to test with non-root-Linux, I upgraded to v0.11 for the new 
mmio decoder.
Along this process, I rebased the wip/ivshmem2 to v0.11 branch, which is 
the new, multi-peer ivshmem2(5c90e846).
I rewrite the drivers of RTOS, too.

Also, I changed the root cell Linux kernel version from 4.9 to 4.19. (Both 
wil PREEMPT_RT patch installed)

So I changed:
 * Linux kernel version
 * Jailhouse version
 * ivshmem2 version

Today, I cherry-picked the new multi-peer ivshmem2 onto:
 * v0.11
 * v0.10
 * v0.9.1
and tested with Linux 4.19.
All of them has a ~25.8us latency.

The baseline, Linux 4.9 /w v0.9.1 /w old ivshmem2 is 10.87us.

Then I tested Linux 4.9 /w v0.9.1 /w new multi-peer ivshmem2. It has a 
latency of 28.62us.

It seems like using the new ivshmem2 mechanism cause the execution to slow 
down.
I didn't find a specific hot-spot, so certain resource contention should be 
the cause.

BTW, I has code that "executes" on the ivshmem region, but I don't think 
this should be a problem, isn't it?

--
Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/2151b869-9732-4483-8659-90234a971f1b%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

2019年10月26日土曜日 2時09分30秒 UTC+9 Jan Kiszka:
>
> On 25.10.19 15:52, Henning Schild wrote: 
> > Well you only have soo many shared ressources and if it is not 
> > additional exits/interrupts then it is contention on shared ressources. 
> > 
> > We are probably talking about caches, TLBs and buses. 
> > 
> > You should be able to use i.e. "perf" on Linux to read out hardware 
> > performance counters. And there you might want to look for TLB and 
> > cache misses. 
> > 
> > But the bisecting might be the better idea. Jan already mentioned the 
> > "features" that could be responsible. With a bit of educated guessing 
> > you will get away with just a few tries. 
>
> BTW, does your RTOS happen to use anything of the inmate bootstrap code 
> to start in Jailhouse? That also changed. 
>
> Jan 
>

Henning,

I do think there are contentions related to memory, either TLB or 
Bus(assuming CAT is good).
I will do the bisect this month, got urgent things to do.


Jan,

Yes, I did steal some code.
I will check if they fit.


Yang
 

>
> > 
> > Henning 
> > 
> > Am Fri, 25 Oct 2019 00:04:36 -0700 
> > schrieb Chung-Fan Yang >: 
> > 
> >> Alright, I have test the latency from HW IRQ to application response. 
> >> 
> >> I found out that there aren't any additional VM-Exit or IRQ, nor RTOS 
> >> scheduling and house-keeping. 
> >> 
> >> It feels like the processor is generally slower as everything takes 
> >> longer to run. 
> >> 
> >> The IRQ epilogue takes ~7.8us and iretq ~2.x us. In addition, the 
> >> libc and syscall interface also have slow downed a bit. 
> >> 
> >> I do notice after upgrading, even with CAT, my RTOS latency are prone 
> >> to be influenced by the Linux side applications. 
> >> This was not observed during v0.9.1. 
> >> 
> >> It's strange. 
> >> 
> >> 
> >> Yang. 
> >> 
> > 
> > 
> > 
>
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE 
> Corporate Competence Center Embedded Linux 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/579d40f1-a8f4-4144-9405-3bba1ea23c14%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

2019年10月26日土曜日 18時26分57秒 UTC+9 michael...@gmail.com:
>
> Hi all,
>
> I'm not sure if this is relevant in this case, but I have noticed that on 
> Intel x86-64, if hardware p-states (HWP) are enabled in the CPU (which they 
> are by default if the CPU supports it), this introduces frequency scaling 
> coupling between cores, even when the cores are isolated in separate cells. 
> So you get this unexpected behavior where an inmate with 1 core will run 
> *faster* if the other cores are very busy, and run a lot slower if the 
> other cores are all idle. This is because the CPU itself does the frequency 
> scaling automatically in hardware and doesn't know anything about what 
> Jailhouse is doing.
>
> Passing intel_pstate=no_hwp to the Linux kernel command line disables 
> hardware p-states and gets rid of this coupling as far as I can tell. It 
> appears that HWP is a relatively new feature (last couple of years) in 
> Intel CPUs.
>

This looks like a possibility.

I did discovered that cores changing C-state in Root Cell can cause RTOS to 
become slower.
We had some disscussion a couple month before, I think.

Currently, I have disabled all the C-state, but not P-state.
Maybe the driver kicked in after I got a newer kernel?

I will check.
 

>
> -Michael
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/8aa82f84-ee18-4f6f-8d24-906528c5edcb%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

Alright, I have test the latency from HW IRQ to application response.

I found out that there aren't any additional VM-Exit or IRQ, nor RTOS 
scheduling and house-keeping.

It feels like the processor is generally slower as everything takes longer 
to run.

The IRQ epilogue takes ~7.8us and iretq ~2.x us. In addition, the libc and 
syscall interface also have slow downed a bit.

I do notice after upgrading, even with CAT, my RTOS latency are prone to be 
influenced by the Linux side applications.
This was not observed during v0.9.1.

It's strange.


Yang.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/48bb5fe2-9b9f-4ad1-872e-9eae4bdd2c43%40googlegroups.com.

Re: v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

>
> Do you mean upgrading Jailhouse (and only that) from 0.9 to 0.11? 
>
>
Yes

We refactored quite a bit between the two releases. Namely, 0.10 brought 
> per-cpu page tables. 
>

Page table might be the cause, but I am not sure.
I did noticed that my my process updating the page table during CTX in RTOS 
has dramatically slowed down after upgrade.
It was < 10us, but 250 us after upgrade.
I did some optimization by saving the 2-level and 3-level page directory 
entries and managed to reduce it to 25us.
 

>
> You could first of try to narrow the reason down a bit: Do the exit 
> statistics look different for both versions?


I don't see a large difference.

With Intel x86, you should 
> normally have no exit for an external or timer interrupt injection. 


I did noticed in both version, I have a large amount of exit due to MSR 
access.
Doesn't x2APIC EOI cause an exit?
 

> From 
> that perspective, even 20 µs is too high. Try to identify the path that 
> causes the latency. 
>
> You could also try to bisect Jailhouse between the two versions, in 
> order to identify the causing commit. But that is only plan B I would say. 
>

I will work on this, but bisect Jailhouse can be difficult considering the 
ABI and config format change from time to time.

 Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/eed4bd9a-7020-4182-9949-d529bef7b3b2%40googlegroups.com.

v0.9 vs v1.1 interrupt latency raise

[Chung-Fan Yang]

Hello,

I observed that the interrupt latency raise from 20us to 50us (measures in 
an RTOS) after I upgraded from v0.9 to v1.1.

I am working on x86_64, so I am suspecting CPU bug mitigations. 
I would like to ask that are there any CPU bugs mitigations in effect?

However, I do find out that in Root Linux, the latency is almost the same, 
comparing the 2 versions.

Are there any thing I should adjust my RTOS to adapt with?

Does anyone has ideas of the source of such difference?
Comments are welcome.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/a54a651c-13de-4aa1-9c32-475ebddc4e6f%40googlegroups.com.

Re: PoC: virtio over ivshmem for Jailhouse

[Chung-Fan Yang]

2019年10月15日火曜日 16時18分39秒 UTC+9 Jan Kiszka:
>
> On 14.10.19 19:04, Jan Kiszka wrote: 
> > Hi all, 
> > 
> > it's still basically a PoC, not yet specified and with a lot of sharp 
> > edges, but it works too well for not being pointed out: 
> > 
> > The queues/jailhouse-ivshmem2 branch on [1] contains a virtio-ivshmem 
> > transport driver [2] for Linux front-end guests and a simple virtio 
> > console backend device model [3], also for Linux. Combining that with 
> > the wip/ivshmem2 branch of Jailhouse, you can call in the root cell 
> > 
> > # echo "4a48 4a48 4a48 4a48 ffc003 ff" \ 
> >> /sys/bus/pci/drivers/uio_ivshm/new_id 
> > # virtio-ivshmem-console /dev/uio0 
> > 
> > and then start the Linux demo inmate which uses "console=hvc0". You will 
> > both see the (late) boot logs and be able to log into the guest, like it 
> > were a serial console. And all that without touching anything in 
> > Jailhouse (beyond generic ivshmem 2.0 support)! 
> > 
> > More to come, latest at [4]. 
> > 
> > Jan 
> > 
> > [1] http://git.kiszka.org/linux.git/ 
> > [2] 
> > 
> http://git.kiszka.org/?p=linux.git;a=blob;f=drivers/virtio/virtio_ivshmem.c;hb=refs/heads/queues/jailhouse-ivshmem2
>  
> > [3] 
> > 
> http://git.kiszka.org/?p=linux.git;a=blob;f=tools/virtio/virtio-ivshmem-console.c;hb=refs/heads/queues/jailhouse-ivshmem2
>  
> > [4] https://sched.co/TmxI 
> > 
>
> Was too easy: 
>
> Welcome to Buildroot 
> jailhouse login: root 
> # mount /dev/vda /mnt/ 
> [8.968669] EXT4-fs (vda): mounted filesystem with ordered data mode. 
> Opts: (null) 
> # ls -l /mnt/ 
> total 12 
> drwx--2 root root 12288 Oct 14  2019 lost+found 
>
> You just need [5] and this to make it happen: 
>
> # dd if=/dev/zero of=disk.img bs=1M count=64 
> # mkfs.ext4 disk.img 
> # echo "4a48 4a48 4a48 4a48 ffc002 ff" \ 
>> /sys/bus/pci/drivers/uio_ivshm/new_id 
> # virtio-ivshmem-block /dev/uio0 disk.img 
>
> I've updated the Jailhouse wip/ivshmem2 branch with corresponding cell 
> configurations. 
>
> Jan 
>
> [5] 
> http://git.kiszka.org/?p=linux.git;a=blob;f=tools/virtio/virtio-ivshmem-block.c;hb=refs/heads/queues/jailhouse-ivshmem2
>  
>
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE 
> Corporate Competence Center Embedded Linux 
>

This will be very handy.

I am current working on project require custom pipes and serial console 
between cells.
I have handcrafted the drivers, but if the generic virtio model is adapted, 
it will be much easier, at least for the Linux side.

Yang
 

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/e67df2f1-0967-453f-97a9-5bc2ea58a790%40googlegroups.com.

Re: Invalid MMIO access during PCI device initialization

[Chung-Fan Yang]

2019年9月20日金曜日 18時57分27秒 UTC+9 Jan Kiszka:
>
> On 20.09.19 10:49, Chung-Fan Yang wrote: 
> > Hi everyone, 
> > 
> > I am having some problem with a physical PCI-e serial card and non-root 
> Linux. 
> > 
> > I have been using the serial card with the root Linux and non-root RTOS 
> for a 
> > while whthout any problem. 
> > 
> > Recently, I decided to try this card in a non-root Linux. 
> > I compiled the Jailhouse enabled kernel and prepared a rootfs. 
> > The non-root Linux work fine with an motherboard built-in 8250 serial. 
> > 
> > I have a custom Linux driver for this PCI-e serial card. 
> > Therefore, this card should be immune from the 8250.n_uart problem. 
> > However, because the card is still in the COMMUNICATION_SERIAL class, 
> 8250_pci 
> > driver still will initialize the card during boot. 
> > 
> > During this initialization, it will try to setup the virtual channels. 
> > When it read the VC capability structures, the system hangs with the 
> following 
> > log(I had made jailhouse to be more verbose). 
> > 
> > More specifically, 
> > 
> > when it do pci_vc_do_save_buffer() in drivers/pci/vc.c and call 
> > 
> > pci_read_config_word(dev, pos + PCI_VC_PORT_CTRL, (u16 *)buf); 
> > 
> > it will in turn call read_pci_config() in arch/x86/pci/early.c, 
> resulting a fault. 
> > 
> > 
> > Removing PCI device 02:00.0 from cell "RootCell" 
> > Freeing 8 interrupt(s) for device 0200 at index 74 
> > Adding PCI device 02:00.0 to cell "linux-x86-demo" 
> > Reserving 1 interrupt(s) for device 0200 at index 72 
> > Removing PCI device 02:00.1 from cell "RootCell" 
> > Freeing 8 interrupt(s) for device 0201 at index 119 
> > Adding PCI device 02:00.1 to cell "linux-x86-demo" 
> > Reserving 1 interrupt(s) for device 0201 at index 73 
> > Removing PCI device 02:00.2 from cell "RootCell" 
> > Freeing 1 interrupt(s) for device 0202 at index 71 
> > Adding PCI device 02:00.2 to cell "linux-x86-demo" 
> > Reserving 1 interrupt(s) for device 0202 at index 71 
> > Created cell "linux-x86-demo" 
> > Page pool usage after cell creation: mem 468/16329, remap 65711/131072 
> > Cell "linux-x86-demo" can be loaded 
> > CPU 7 received SIPI, vector 100 
> > Started cell "linux-x86-demo" 
> > FATAL: unsupported instruction (0x66 0x00 0x00 0x00) 
> > FATAL: Invalid MMIO/RAM read, addr: 0x8020010c size: 0 
> > Name: linux-x86-demo 
> > RIP: 0xb2550a08 RSP: 0xad0cc003bc08 FLAGS: 10286 
> > RAX: 0xad0cd020010c RBX: 0xad0cd020 RCX: 0x010c 
> > RDX: 0x00ff RSI: 0x0002 RDI: 0x 
> > CS: 10 BASE: 0x AR-BYTES: a09b EFER.LMA 1 
> > RBP: 0x010c 
> > DS: 0 
> > SS: 0 
> > GDTR_BASE: 0xfe001000 GDTR_LIMIT: 0x007f 
> > IDTR_BASE: 0xfe00 IDTR_LIMIT: 0x0fff 
> > CR0: 0x80050033 CR3: 0x06c0a001 CR4: 0x003626f0 
> > EFER: 0x0d01 
> > CPL: 0 RPL: 0 
> > Parking CPU 7 (Cell: "linux-x86-demo") 
> > 
> > I do not understand why only this specific read cause a fault, but 
> others don't. 
> > Did I misconfig anything in the cell config file(as an attachment). 
> > 
>
> I suspect your are not using current master but rather the laster release, 
> right? Could you retry with master, specifically because of [1]? 
>
>
Yeah, I am using an old version. I want to make my system as stable as 
possible during my research.

Using the current master solve the problem.

Thanks for the help.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/8a0921f1-11ce-4231-a9ec-c115a059cfee%40googlegroups.com.

Ivshmem, MSIX, irq_find_mapping return -EEXIST

[Chung-Fan Yang]

Hello,

I am looking for some help.
I am using ivshmem-v2 with virtio.
I am planning to implement a virtual serial driver on top of it.
I ripped off the ivshmem-net driver, and obtain a queue which I should able to 
communicate with other cells from host.

The problem is that during root cell's PCI probing, pci_alloc_irq_vectors 
returned -NOSPCE.
I went down the rabbit hole with ftrace and found out that in the MSIX 
initialization, irq_find_mapping returned -EEXIST.

I have cross check the code with ivshmem-net, there is no difference except for 
the NAPI registration.

I am filling quite helpless down this rabbithole, and can really use some help.
I can provide the trace log and driver code if it is necessary.

Thanks,

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/deca16ad-b539-4447-b865-f0fbcb5abe3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Power management state is vulnerable to Linux

[Chung-Fan Yang]

> Was your test running on separate physical cores?

Yes, separate physical cores, same package.

> Yes, see our TODO list: power management access on x86 is currently 
> unmoderated, 
> thus you can influence other cells if they share a centrally managed power 
> domain.
> 
> This is fixable, but the major effort is first of all in modeling those 
> aspects 
> correctly so that we can decide what to do when cell X requests to modify 
> some 
> power parameters.

This explains the phenomenon I observed.
Thanks.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Power management state is vulnerable to Linux

[Chung-Fan Yang]

Hi,

I had previously posted question on the power management(PM) on x86.
I would like to share some of my discoveries and discuss this strange 
phenomenon.

In the previous discussion, I was acknowledge that newly created cells are 
running in full power. They have to manage the power themselves.

I had written a simple benchmark which busy loop and poke the RTS of an UART 
(lacking GPIO on my x86 platform). Loading this into a cell, it generated a 
"perfect" square wave. However there were some mysterious 20us jitter spikes.

I confirmed that no interrupt occurred and UART hardware is not vicious. After 
I do 'cat 0 > /dev/cpu_dma_latency', the jitter spikes disappeared.

I would like to suspect that the power management part on the hardware are some 
how shared across cells, but I am not a x86 pro so I would like to confirm this.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remapping MSI interrupts of pci-e serial card

[Chung-Fan Yang]

> > Hmm, this fails when passing it to jailhouse config create. Could it be 
> > that you 
> > ran the collect job as normal user, not as root?
> 
> Seems like I failed to copy it from the remote host via scp.
> I have attached a new version.
> It has been run with current working condition.

I noticed that the file "might" be identical.
I am already running as root, so any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remapping MSI interrupts of pci-e serial card

[Chung-Fan Yang]

2019年1月15日火曜日 21時31分39秒 UTC+9 J. Kiszka:
> On 15.01.19 13:09, Chung-Fan Yang wrote:
> > 
> >> Did you regenerate the root cell config after adding the card? Or manually 
> >> added
> >> the additional entries needed for it? Something went wrong in that area.
> > 
> > I regenerated the config and cross checked with the working one.
> > Nothing vicious was found.
> > 
> > I did notice that the IRQ was an IO-APIC IRQ in the /proc/interrupts.
> > I think this is the problem. I am working on a MSI version of the driver.
> 
> Conceptually, IO-APIC-based interrupts should work as well (and do in other 
> setups).
In this case, do I need to add irq_chips to the sysconfig?

> And if they do not work, you should at least see some error messages on 
> the hypervisor console. Do you?

No I got nothing. The initialization finfshed happily without errors.

-
Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remapping MSI interrupts of pci-e serial card

[Chung-Fan Yang]

> Did you regenerate the root cell config after adding the card? Or manually 
> added 
> the additional entries needed for it? Something went wrong in that area.

I regenerated the config and cross checked with the working one.
Nothing vicious was found.

I did notice that the IRQ was an IO-APIC IRQ in the /proc/interrupts.
I think this is the problem. I am working on a MSI version of the driver.

-
Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Remapping MSI interrupts of pci-e serial card

[Chung-Fan Yang]

Hello,

I recently added a PCI-e serial card to my known working x86-64 Jailhouse box.
It got bdf 02:00.0 on the bus and Linux without Jailhouse works fine.

However the port ceased to work when I enable Jailhouse.
It seems like that during IOMMU remapping, pci_enabled_msi_vectors reported 0 
for the card.
The card should have at least one MSI interrupt vector.

I observed that the Linux driver only register and enable the device IRQ when 
it's in used by some application, but I am not sure this is the cause.

Any suggestions or guideline to add a PCI-e serial card to a x86-64 machine?

-
Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年11月13日火曜日 18時17分03秒 UTC+9 J. Kiszka:
> On 13.11.18 08:55, Chung-Fan Yang wrote:
> > 2018年11月12日月曜日 16時34分08秒 UTC+9 Chung-Fan Yang:
> >> 2018年11月12日月曜日 15時10分30秒 UTC+9 Chung-Fan Yang:
> >>> 2018年11月10日土曜日 16時25分57秒 UTC+9 Jan Kiszka:
> >>>> On 09.11.18 09:02, Chung-Fan Yang wrote:
> >>>>> 2018年10月8日月曜日 19時23分52秒 UTC+9 J. Kiszka:
> >>>>>> On 05.10.18 09:52, Chung-Fan Yang wrote:
> >>>>>>> 2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> >>>>>>>> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> >>>>>>>>> On 04.10.18 10:46, Chung-Fan Yang wrote:
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 
> >>>>>>>>>> machine.
> >>>>>>>>>> I am currently trying to implement interrupt driven seril output. 
> >>>>>>>>>> Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to 
> >>>>>>>>>> the RTOS.
> >>>>>>>>>>
> >>>>>>>>>> I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to 
> >>>>>>>>>> remap IRQ4.
> >>>>>>>>>> The IOAPIC demo works fine, but not when I use the same technique 
> >>>>>>>>>> to remap IRQ4, the following error message is printed on the 
> >>>>>>>>>> jailhouse console(ttyS1).
> >>>>>>>>>>
> >>>>>>>>>> VT-d fault event reported by IOMMU 1:
> >>>>>>>>>>  Source Identifier (bus:dev.func): f0:1f.7
> >>>>>>>>>>  Fault Reason: 0x26 Fault Info: 6a0 Type 0
> >>>>>>>>>>
> >>>>>>>>>> I try to read the VT-d manual, but I can't quite get the point why 
> >>>>>>>>>> this is happening. So I would kindly to ask help here. I have 
> >>>>>>>>>> attached the sysconfig and RTOS's config.
> >>>>>>>>>
> >>>>>>>>> You need to align the ID of the IOAPIC in the non-root cell config 
> >>>>>>>>> with the root
> >>>>>>>>> cell. That's 0x1f0ff in your case.
> >>>>>>>>>
> >>>>>>>>> Jan
> >>>>>>>>
> >>>>>>>> Thank you. This do the trick. The error is gone.
> >>>>>>>>
> >>>>>>>> However, the interrupt is not triggered on CPU when 16550 asserted 
> >>>>>>>> it (polling IIR register reveals there was an interrupt).
> >>>>>>>> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the 
> >>>>>>>> same technique in ioapic-demo.
> >>>>>>>> It seems like pin 4 was always high, although 16550 should be 
> >>>>>>>> sending edge triggering signals.
> >>>>>>>>
> >>>>>>>> I am wondering that my board has a different connection from 16550 
> >>>>>>>> to the IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> >>>>>>
> >>>>>> Does Linux (as non-root cell) work with that UART normally? Testing 
> >>>>>> that would
> >>>>>> allow us to exclude the Jailhouse cell configuration from the equation.
> >>>>>>
> >>>>>>>>
> >>>>>>>> Yang
> >>>>>>>
> >>>>>>> I didn't find the MADT table in the /sys/firmware/acpi, assuming my 
> >>>>>>> baord firmware is missing it.
> >>>>>>
> >>>>>> The non-root cell does not have ACPI - or is that when looking at the 
> >>>>>> root cell?
> >>>>>>
> >>>>>>> According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
> >>>>>>> I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
> >>>>>>> This is starnge. Where did that UART IRQ go?
> >>>>>>
> >>>>>> Unless you have some special embedded SoC, it will be IRQ 4. Again, 
> >>&g

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年11月12日月曜日 16時34分08秒 UTC+9 Chung-Fan Yang:
> 2018年11月12日月曜日 15時10分30秒 UTC+9 Chung-Fan Yang:
> > 2018年11月10日土曜日 16時25分57秒 UTC+9 Jan Kiszka:
> > > On 09.11.18 09:02, Chung-Fan Yang wrote:
> > > > 2018年10月8日月曜日 19時23分52秒 UTC+9 J. Kiszka:
> > > >> On 05.10.18 09:52, Chung-Fan Yang wrote:
> > > >>> 2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> > > >>>> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> > > >>>>> On 04.10.18 10:46, Chung-Fan Yang wrote:
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 
> > > >>>>>> machine.
> > > >>>>>> I am currently trying to implement interrupt driven seril output. 
> > > >>>>>> Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to 
> > > >>>>>> the RTOS.
> > > >>>>>>
> > > >>>>>> I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to 
> > > >>>>>> remap IRQ4.
> > > >>>>>> The IOAPIC demo works fine, but not when I use the same technique 
> > > >>>>>> to remap IRQ4, the following error message is printed on the 
> > > >>>>>> jailhouse console(ttyS1).
> > > >>>>>>
> > > >>>>>> VT-d fault event reported by IOMMU 1:
> > > >>>>>> Source Identifier (bus:dev.func): f0:1f.7
> > > >>>>>> Fault Reason: 0x26 Fault Info: 6a0 Type 0
> > > >>>>>>
> > > >>>>>> I try to read the VT-d manual, but I can't quite get the point why 
> > > >>>>>> this is happening. So I would kindly to ask help here. I have 
> > > >>>>>> attached the sysconfig and RTOS's config.
> > > >>>>>
> > > >>>>> You need to align the ID of the IOAPIC in the non-root cell config 
> > > >>>>> with the root
> > > >>>>> cell. That's 0x1f0ff in your case.
> > > >>>>>
> > > >>>>> Jan
> > > >>>>
> > > >>>> Thank you. This do the trick. The error is gone.
> > > >>>>
> > > >>>> However, the interrupt is not triggered on CPU when 16550 asserted 
> > > >>>> it (polling IIR register reveals there was an interrupt).
> > > >>>> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the 
> > > >>>> same technique in ioapic-demo.
> > > >>>> It seems like pin 4 was always high, although 16550 should be 
> > > >>>> sending edge triggering signals.
> > > >>>>
> > > >>>> I am wondering that my board has a different connection from 16550 
> > > >>>> to the IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> > > >>
> > > >> Does Linux (as non-root cell) work with that UART normally? Testing 
> > > >> that would
> > > >> allow us to exclude the Jailhouse cell configuration from the equation.
> > > >>
> > > >>>>
> > > >>>> Yang
> > > >>>
> > > >>> I didn't find the MADT table in the /sys/firmware/acpi, assuming my 
> > > >>> baord firmware is missing it.
> > > >>
> > > >> The non-root cell does not have ACPI - or is that when looking at the 
> > > >> root cell?
> > > >>
> > > >>> According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
> > > >>> I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
> > > >>> This is starnge. Where did that UART IRQ go?
> > > >>
> > > >> Unless you have some special embedded SoC, it will be IRQ 4. Again, 
> > > >> testing with
> > > >> Linux as non-root guest can help to isolate reasons.
> > > >>
> > > >> Jan
> > > >>
> > > >> -- 
> > > >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > >> Corporate Competence Center Embedded Linux
> > > > 
> > > > Hi Jan,
> > > > 
> > > > Finally have some time back to this issue.
> > > > 
> &

Re: x86_64 Power management

[Chung-Fan Yang]

2018年11月8日木曜日 2時29分15秒 UTC+9 J. Kiszka:
> On 07.11.18 18:11, Henning Schild wrote:
> > Am Wed, 7 Nov 2018 17:17:32 +0100
> > schrieb Jan Kiszka :
> > 
> >> On 07.11.18 16:13, Henning Schild wrote:
> >>> Am Tue, 6 Nov 2018 17:02:34 -0800
> >>> schrieb Chung-Fan Yang :
> >>>
> >>>> Hi,
> >>>>
> >>>> I am having some quextions about power management(PM) on x86.
> >>>> As afr as I understand, there are C-states and P-states,
> >>>> controlling the sleeping depth and frequency scale accrodingly, on
> >>>> an x86 processor.
> >>>>
> >>>> My question is that when I use jailhouse house to create a non-root
> >>>> cell, which state it will fall into?
> >>>
> >>> You will get the CPU in its full power mode, if you want any of the
> >>> other states your inmate needs to support that (PM code).
> >>>
> >>>> Will it be reset to C0 and full frequency? or inherit the root
> >>>> state when cell is created?
> >>>>
> >>>> Is it necessary to write some PM code in the inmates?
> >>>
> >>> Only if you want to safe power, or if you need to throttle so your
> >>> BIOS does not hit you with an SMI.
> >>
> >> Power management - better: power management moderation - is not
> >> provided by Jailhouse so far. That means, as Henning described,
> >> inmates are responsible to do "the right thing". That's surely not
> >> optimal, can lead to mistakes and has to be improved (see TODO). E.g,
> >> we just learned that calling "hlt" in the apic-demo is likely a
> >> source of undesired latency, we should do polling or mwait with the
> >> right level (there will be patches soonish).
> > 
> > Interesting. Will you switch the test to mwait or will you trap hlt and
> > emulate it with mwait?
> 
> No idea yet, and some tests are still pending. But it's fairly likely that we 
> will have to trap mwait and manage which maximum C-state the cell can request 
> this way. With hlt being no longer a reasonable alternative, we will likely 
> not 
> be able to provide an exit-free way to wait for interrupts without burning 
> cycles.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

2018年11月8日木曜日 2時29分15秒 UTC+9 J. Kiszka:
> On 07.11.18 18:11, Henning Schild wrote:
> > Am Wed, 7 Nov 2018 17:17:32 +0100
> > schrieb Jan Kiszka :
> > 
> >> On 07.11.18 16:13, Henning Schild wrote:
> >>> Am Tue, 6 Nov 2018 17:02:34 -0800
> >>> schrieb Chung-Fan Yang :
> >>>
> >>>> Hi,
> >>>>
> >>>> I am having some quextions about power management(PM) on x86.
> >>>> As afr as I understand, there are C-states and P-states,
> >>>> controlling the sleeping depth and frequency scale accrodingly, on
> >>>> an x86 processor.
> >>>>
> >>>> My question is that when I use jailhouse house to create a non-root
> >>>> cell, which state it will fall into?
> >>>
> >>> You will get the CPU in its full power mode, if you want any of the
> >>> other states your inmate needs to support that (PM code).
> >>>
> >>>> Will it be reset to C0 and full frequency? or inherit the root
> >>>> state when cell is created?
> >>>>
> >>>> Is it necessary to write some PM code in the inmates?
> >>>
> >>> Only if you want to safe power, or if you need to throttle so your
> >>> BIOS does not hit you with an SMI.
> >>
> >> Power management - better: power management moderation - is not
> >> provided by Jailhouse so far. That means, as Henning described,
> >> inmates are responsible to do "the right thing". That's surely not
> >> optimal, can lead to mistakes and has to be improved (see TODO). E.g,
> >> we just learned that calling "hlt" in the apic-demo is likely a
> >> source of undesired latency, we should do polling or mwait with the
> >> right level (there will be patches soonish).
> > 
> > Interesting. Will you switch the test to mwait or will you trap hlt and
> > emulate it with mwait?
> 
> No idea yet, and some tests are still pending. But it's fairly likely that we 
> will have to trap mwait and manage which maximum C-state the cell can request 
> this way. With hlt being no longer a reasonable alternative, we will likely 
> not 
> be able to provide an exit-free way to wait for interrupts without burning 
> cycles.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Thanks you Jan and Henning for providing such detail.

Now I can be sure that the system is not throttled.
I removed the hlt in my idle loop. 
Non-root cell's working fine so far, but I will keep an eye on SMI to be sure.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting up the Serial Port

[Chung-Fan Yang]

2018年11月13日火曜日 8時45分39秒 UTC+9 Michael Hinton:
> Hello,
> 
> I'm struggling to figure out how to read out the serial port output of 
> Jailhouse from my Kubuntu 18.04 Intel x86-64 box. Apparently, this is the 
> only way to read out Jailhouse console logs in a cell in real hardware. Does 
> anyone have any instructions for how to do this?
> 
> Here is what I tried:
> 
> I purchased these cables and connected my Linux machine to my Windows machine:
> https://www.amazon.com/gp/product/B001Y1F0HW/ref=oh_aui_detailpage_o02_s00?ie=UTF8=1
> https://www.amazon.com/gp/product/B0758B6MK6/ref=oh_aui_detailpage_o02_s00?ie=UTF8=1

For the RS-232 back plane bracket, there exist 2 different pin header layout on 
the motherboard.
Did you confirm that the one you buy is compatible to you motherboard?

Ref: http://pinoutguide.com/Motherboard/rs232_header_pinout.shtml

> 
> In the Linux terminal, I ran `dmesg | grep tty` to see the baud rate (115200 
> and 8N1).
> 
> I used minicom -s to set up /dev/ttyS0 to be 115200 8N1 and no flow control.
> 
> On my Windows machine, I installed the drivers for the usb-serial adapter, 
> and set the baud rate and other serial connection information in the driver 
> settings.
> 
> I then used Putty to listen to the output, setting the serial information to 
> match the above, with no success. I have yet to see any output anywhere.
> 
> This is really frustrating, and I'm not sure how to debug this further or 
> where to go from here.
> 

Not a frequent Windows user, but the software parts seems fine to me.

> Some resources I've tried to follow:
> https://help.ubuntu.com/community/Minicom
> https://unix.stackexchange.com/questions/117037/how-to-send-data-to-a-serial-port-and-see-any-answer
> 
> Thanks,
> Michael


-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年11月12日月曜日 15時10分30秒 UTC+9 Chung-Fan Yang:
> 2018年11月10日土曜日 16時25分57秒 UTC+9 Jan Kiszka:
> > On 09.11.18 09:02, Chung-Fan Yang wrote:
> > > 2018年10月8日月曜日 19時23分52秒 UTC+9 J. Kiszka:
> > >> On 05.10.18 09:52, Chung-Fan Yang wrote:
> > >>> 2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> > >>>> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> > >>>>> On 04.10.18 10:46, Chung-Fan Yang wrote:
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 
> > >>>>>> machine.
> > >>>>>> I am currently trying to implement interrupt driven seril output. 
> > >>>>>> Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to 
> > >>>>>> the RTOS.
> > >>>>>>
> > >>>>>> I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to 
> > >>>>>> remap IRQ4.
> > >>>>>> The IOAPIC demo works fine, but not when I use the same technique to 
> > >>>>>> remap IRQ4, the following error message is printed on the jailhouse 
> > >>>>>> console(ttyS1).
> > >>>>>>
> > >>>>>> VT-d fault event reported by IOMMU 1:
> > >>>>>> Source Identifier (bus:dev.func): f0:1f.7
> > >>>>>> Fault Reason: 0x26 Fault Info: 6a0 Type 0
> > >>>>>>
> > >>>>>> I try to read the VT-d manual, but I can't quite get the point why 
> > >>>>>> this is happening. So I would kindly to ask help here. I have 
> > >>>>>> attached the sysconfig and RTOS's config.
> > >>>>>
> > >>>>> You need to align the ID of the IOAPIC in the non-root cell config 
> > >>>>> with the root
> > >>>>> cell. That's 0x1f0ff in your case.
> > >>>>>
> > >>>>> Jan
> > >>>>
> > >>>> Thank you. This do the trick. The error is gone.
> > >>>>
> > >>>> However, the interrupt is not triggered on CPU when 16550 asserted it 
> > >>>> (polling IIR register reveals there was an interrupt).
> > >>>> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the 
> > >>>> same technique in ioapic-demo.
> > >>>> It seems like pin 4 was always high, although 16550 should be sending 
> > >>>> edge triggering signals.
> > >>>>
> > >>>> I am wondering that my board has a different connection from 16550 to 
> > >>>> the IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> > >>
> > >> Does Linux (as non-root cell) work with that UART normally? Testing that 
> > >> would
> > >> allow us to exclude the Jailhouse cell configuration from the equation.
> > >>
> > >>>>
> > >>>> Yang
> > >>>
> > >>> I didn't find the MADT table in the /sys/firmware/acpi, assuming my 
> > >>> baord firmware is missing it.
> > >>
> > >> The non-root cell does not have ACPI - or is that when looking at the 
> > >> root cell?
> > >>
> > >>> According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
> > >>> I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
> > >>> This is starnge. Where did that UART IRQ go?
> > >>
> > >> Unless you have some special embedded SoC, it will be IRQ 4. Again, 
> > >> testing with
> > >> Linux as non-root guest can help to isolate reasons.
> > >>
> > >> Jan
> > >>
> > >> -- 
> > >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > >> Corporate Competence Center Embedded Linux
> > > 
> > > Hi Jan,
> > > 
> > > Finally have some time back to this issue.
> > > 
> > > I have built a bzImage using the patched linux source tree.
> > > Following:  jailhouse/Documentation/non-root-linux.txt
> > > 
> > > I use it to boot the non-root linux with the appended cell config.
> > > 
> > > The kernel boot dmesg did print on tty0, but it crashes at some point.
> > > 
> > > Hypervisor give the following message (I did some modification on the 
>

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年11月10日土曜日 16時25分57秒 UTC+9 Jan Kiszka:
> On 09.11.18 09:02, Chung-Fan Yang wrote:
> > 2018年10月8日月曜日 19時23分52秒 UTC+9 J. Kiszka:
> >> On 05.10.18 09:52, Chung-Fan Yang wrote:
> >>> 2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> >>>> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> >>>>> On 04.10.18 10:46, Chung-Fan Yang wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 machine.
> >>>>>> I am currently trying to implement interrupt driven seril output. 
> >>>>>> Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to the 
> >>>>>> RTOS.
> >>>>>>
> >>>>>> I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to remap 
> >>>>>> IRQ4.
> >>>>>> The IOAPIC demo works fine, but not when I use the same technique to 
> >>>>>> remap IRQ4, the following error message is printed on the jailhouse 
> >>>>>> console(ttyS1).
> >>>>>>
> >>>>>> VT-d fault event reported by IOMMU 1:
> >>>>>> Source Identifier (bus:dev.func): f0:1f.7
> >>>>>> Fault Reason: 0x26 Fault Info: 6a0 Type 0
> >>>>>>
> >>>>>> I try to read the VT-d manual, but I can't quite get the point why 
> >>>>>> this is happening. So I would kindly to ask help here. I have attached 
> >>>>>> the sysconfig and RTOS's config.
> >>>>>
> >>>>> You need to align the ID of the IOAPIC in the non-root cell config with 
> >>>>> the root
> >>>>> cell. That's 0x1f0ff in your case.
> >>>>>
> >>>>> Jan
> >>>>
> >>>> Thank you. This do the trick. The error is gone.
> >>>>
> >>>> However, the interrupt is not triggered on CPU when 16550 asserted it 
> >>>> (polling IIR register reveals there was an interrupt).
> >>>> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the 
> >>>> same technique in ioapic-demo.
> >>>> It seems like pin 4 was always high, although 16550 should be sending 
> >>>> edge triggering signals.
> >>>>
> >>>> I am wondering that my board has a different connection from 16550 to 
> >>>> the IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> >>
> >> Does Linux (as non-root cell) work with that UART normally? Testing that 
> >> would
> >> allow us to exclude the Jailhouse cell configuration from the equation.
> >>
> >>>>
> >>>> Yang
> >>>
> >>> I didn't find the MADT table in the /sys/firmware/acpi, assuming my baord 
> >>> firmware is missing it.
> >>
> >> The non-root cell does not have ACPI - or is that when looking at the root 
> >> cell?
> >>
> >>> According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
> >>> I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
> >>> This is starnge. Where did that UART IRQ go?
> >>
> >> Unless you have some special embedded SoC, it will be IRQ 4. Again, 
> >> testing with
> >> Linux as non-root guest can help to isolate reasons.
> >>
> >> Jan
> >>
> >> -- 
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
> > 
> > Hi Jan,
> > 
> > Finally have some time back to this issue.
> > 
> > I have built a bzImage using the patched linux source tree.
> > Following:  jailhouse/Documentation/non-root-linux.txt
> > 
> > I use it to boot the non-root linux with the appended cell config.
> > 
> > The kernel boot dmesg did print on tty0, but it crashes at some point.
> > 
> > Hypervisor give the following message (I did some modification on the 
> > verboseness of the debug message):
> > 
> > Cell "linux-x86-demo" can be loaded
> > CPU 1 received SIPI, vector 100
> > CPU 2 received SIPI, vector 100
> > CPU 3 received SIPI, vector 100
> > Started cell "linux-x86-demo"
> > CPU 2 received SIPI, vector 9a
> > CPU 3 received SIPI, vector 9a
> > FATAL: Invalid PIO read, port: 87 size: 1
> 
> That's likely due to CONFIG_ISA_DMA_API being enabled in your non-root kernel.
&

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年10月8日月曜日 19時23分52秒 UTC+9 J. Kiszka:
> On 05.10.18 09:52, Chung-Fan Yang wrote:
> > 2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> >> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> >>> On 04.10.18 10:46, Chung-Fan Yang wrote:
> >>>> Hello,
> >>>>
> >>>> I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 machine.
> >>>> I am currently trying to implement interrupt driven seril output. 
> >>>> Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to the 
> >>>> RTOS.
> >>>>
> >>>> I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to remap 
> >>>> IRQ4.
> >>>> The IOAPIC demo works fine, but not when I use the same technique to 
> >>>> remap IRQ4, the following error message is printed on the jailhouse 
> >>>> console(ttyS1).
> >>>>
> >>>> VT-d fault event reported by IOMMU 1:
> >>>>Source Identifier (bus:dev.func): f0:1f.7
> >>>>Fault Reason: 0x26 Fault Info: 6a0 Type 0
> >>>>
> >>>> I try to read the VT-d manual, but I can't quite get the point why this 
> >>>> is happening. So I would kindly to ask help here. I have attached the 
> >>>> sysconfig and RTOS's config.
> >>>
> >>> You need to align the ID of the IOAPIC in the non-root cell config with 
> >>> the root
> >>> cell. That's 0x1f0ff in your case.
> >>>
> >>> Jan
> >>
> >> Thank you. This do the trick. The error is gone.
> >>
> >> However, the interrupt is not triggered on CPU when 16550 asserted it 
> >> (polling IIR register reveals there was an interrupt).
> >> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the same 
> >> technique in ioapic-demo.
> >> It seems like pin 4 was always high, although 16550 should be sending edge 
> >> triggering signals.
> >>
> >> I am wondering that my board has a different connection from 16550 to the 
> >> IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> 
> Does Linux (as non-root cell) work with that UART normally? Testing that 
> would 
> allow us to exclude the Jailhouse cell configuration from the equation.
> 
> >>
> >> Yang
> > 
> > I didn't find the MADT table in the /sys/firmware/acpi, assuming my baord 
> > firmware is missing it.
> 
> The non-root cell does not have ACPI - or is that when looking at the root 
> cell?
> 
> > According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
> > I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
> > This is starnge. Where did that UART IRQ go?
> 
> Unless you have some special embedded SoC, it will be IRQ 4. Again, testing 
> with 
> Linux as non-root guest can help to isolate reasons.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Hi Jan,

Finally have some time back to this issue.

I have built a bzImage using the patched linux source tree.
Following:  jailhouse/Documentation/non-root-linux.txt

I use it to boot the non-root linux with the appended cell config.

The kernel boot dmesg did print on tty0, but it crashes at some point.

Hypervisor give the following message (I did some modification on the 
verboseness of the debug message):

Cell "linux-x86-demo" can be loaded
CPU 1 received SIPI, vector 100
CPU 2 received SIPI, vector 100
CPU 3 received SIPI, vector 100
Started cell "linux-x86-demo"
CPU 2 received SIPI, vector 9a
CPU 3 received SIPI, vector 9a
FATAL: Invalid PIO read, port: 87 size: 1
Name: linux-x86-demo
RIP: 0x8354b6ee RSP: 0x9acec003beb0 FLAGS: 246  

RAX: 0x RBX: 0x8354b6ee RCX: 0x 

RDX: 0x9863c24d RSI: 0x RDI: 0x8354b6ee 

CS: 10 BASE: 0x AR-BYTES: a09b EFER.LMA 1   

RBP: 0x
DS: 0
SS: 0
GDTR_BASE: 0xfe001000 GDTR_LIMIT: 0x007f

IDTR_BASE: 0xfe00 IDTR_LIMIT: 0x0fff

CR0: 0x80050033 CR3: 0x0400a001 CR4: 0x003626f0 

EFER: 0x0d01
CPL: 0 RPL: 0
Parking CPU 1 (Cell: "linux-x86-demo")

No idea what happened. Any suggestions?

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse&quo

x86_64 Power management

[Chung-Fan Yang]

Hi,

I am having some quextions about power management(PM) on x86.
As afr as I understand, there are C-states and P-states, controlling the 
sleeping depth and frequency scale accrodingly, on an x86 processor.

My question is that when I use jailhouse house to create a non-root cell, which 
state it will fall into?

Will it be reset to C0 and full frequency? or inherit the root state when cell 
is created?

Is it necessary to write some PM code in the inmates?

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Interrupt with UIO and ivshmem2

[Chung-Fan Yang]

2018年11月2日金曜日 17時42分54秒 UTC+9 J. Kiszka:
> On 01.11.18 13:44, Chung-Fan Yang wrote:
> > 2018年11月1日木曜日 18時41分23秒 UTC+9 Chung-Fan Yang:
> >> Hello,
> >>
> >> I am working on ivshmem-net to achieve communication between host and cell.
> >> Therefore, I applied the ivshmem2 patch both on the jailhouse and 
> >> root-cell linux kernel (4.9).
> >>
> >> The root side's ivhsmem-net driver is correctly started and I can setup an 
> >> ip address without problem. The non-root side is laking a Ethernet 
> >> deriver. Thus, not checked.
> >>
> >> Together I wish that my old implementation on uio_ivshmem and code ported 
> >> from ivshmem_demo can still work on the new model.
> >>
> >> Therefore, both root and non-root have 2 ivshmem PCI device. One isfor 
> >> supporting the old model, an other for ivshmem-net.
> >>
> >> I read the commits, found out that the memory locations in ivehmem2 has 
> >> expanded to 3 per PCI device. I adapted it and changed both the 
> >> uio_ivshmem and non-root ivshmem driver. I can read and write the shared 
> >> memory without problem.
> >>
> >> But I am current having problem on send and reviving IPIs between root and 
> >> non-root cells. I noticed the following things in the ivshmem2 commits, 
> >> which might affect interrupts:
> >>
> >> 1. The MMIO registers, including the doorbell register's location had 
> >> changed .(64f5b8fe)
> >> 2. MMIO region is expanded to 4K. (27d3ad63)
> >> 3. BAR4 is move to BAR2. (b4e9474c)
> >>
> >> For 1, I changed the index used to access MMIO door bell register. It was 
> >> 12 I changed to 4. But it's still not working.
> >>
> >> For 2, I don't really think it's a problem, because the mapped region is 
> >> larger than before.
> >>
> >> For 3, I read the code of ivshmem_demo. It used to use BAR2 to setup 
> >> msix_table for APIC and MIS-X mapping. I am suspecting that moving the 
> >> BAR4 to BAR2 makes this process some how broken, but I have no idea what 
> >> happened and how to fix it.
> >>
> >> Anyone had used the old uio driver on ivshmem2 and successful talked with 
> >> other cells before?
> >> The configs are attached, I can attach part of the code if requested.
> >> I could really use some help.
> >>
> >> Yang
> > 
> > Hello everyone,
> > 
> > I am glad that I figured it out my self.
> > 
> > Here are the steps. Unless specified, apply only to custom non-root driver.
> > 
> > First, the share memory address and size are not fixed 0x40 and 0x48 
> > anymore.
> > You need to read and decide which regions you want to use by reading out 
> > the PCI Vendor CAPs. Anything 0 in size is a none populated memory region. 
> > Do this for both sides. (Reference commit ID: d2e21a50)
> > 
> > Second, the misx-table is popluated at bar2 not bar4 as before. The uio 
> > driver can handle this, but the code in ivshmem_demo cannot. You have to 
> > change the process of "map_shmem_and_bars." Msix-table address should be 
> > written to PCI_CFG_CAP + 8, which is BAR2 (was hard coded 16 to point to 
> > BAR4).
> > 
> > Third, the MMIO registers changed index, ivpos, a.k.a ID should be 0 
> > instead 8.
> > The doorbell register should be 4 instead 12 (counting in bytes). Change 
> > the non-root driver accordingly. On root Linux, mmaped regions are 
> > typically used to poke the mmio region. Therefore, change where you poke 
> > accordingly.
> > 
> > Finally, on non-root side don't forget to make the probing process to 
> > recognize the right ivshmem version. Class and reversion read from CAP 
> > would have been ored with additional 0x2. On root Linux, you can just force 
> > the uio_ivshmem driver to use jailhouse mode by commenting out some codes.
> > 
> > Hope someone can benifit from this.
> > If you had any problem making the change, I would be happy to help.
> > 
> 
> Note that "ivshmem 2.0" is a prototype, nothing officially supported yet. It 
> is 
> supposed to demonstrate potential improvements of the current ivshmem variant 
> we 
> have in Jailhouse, provided we follow that path (not unlikely, just not 
> decided 
> yet).
> 
> What is your motivation to use this model, rather than the current one?
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

I would like to make the RTOS tal

Re: Interrupt with UIO and ivshmem2

[Chung-Fan Yang]

2018年11月1日木曜日 18時41分23秒 UTC+9 Chung-Fan Yang:
> Hello,
> 
> I am working on ivshmem-net to achieve communication between host and cell.
> Therefore, I applied the ivshmem2 patch both on the jailhouse and root-cell 
> linux kernel (4.9).
> 
> The root side's ivhsmem-net driver is correctly started and I can setup an ip 
> address without problem. The non-root side is laking a Ethernet deriver. 
> Thus, not checked.
> 
> Together I wish that my old implementation on uio_ivshmem and code ported 
> from ivshmem_demo can still work on the new model.
> 
> Therefore, both root and non-root have 2 ivshmem PCI device. One isfor 
> supporting the old model, an other for ivshmem-net.
> 
> I read the commits, found out that the memory locations in ivehmem2 has 
> expanded to 3 per PCI device. I adapted it and changed both the uio_ivshmem 
> and non-root ivshmem driver. I can read and write the shared memory without 
> problem.
> 
> But I am current having problem on send and reviving IPIs between root and 
> non-root cells. I noticed the following things in the ivshmem2 commits, which 
> might affect interrupts:
> 
> 1. The MMIO registers, including the doorbell register's location had changed 
> .(64f5b8fe)
> 2. MMIO region is expanded to 4K. (27d3ad63)
> 3. BAR4 is move to BAR2. (b4e9474c)
> 
> For 1, I changed the index used to access MMIO door bell register. It was 12 
> I changed to 4. But it's still not working.
> 
> For 2, I don't really think it's a problem, because the mapped region is 
> larger than before.
> 
> For 3, I read the code of ivshmem_demo. It used to use BAR2 to setup 
> msix_table for APIC and MIS-X mapping. I am suspecting that moving the BAR4 
> to BAR2 makes this process some how broken, but I have no idea what happened 
> and how to fix it.
> 
> Anyone had used the old uio driver on ivshmem2 and successful talked with 
> other cells before?
> The configs are attached, I can attach part of the code if requested.
> I could really use some help.
> 
> Yang

Hello everyone,

I am glad that I figured it out my self.

Here are the steps. Unless specified, apply only to custom non-root driver.

First, the share memory address and size are not fixed 0x40 and 0x48 anymore.
You need to read and decide which regions you want to use by reading out the 
PCI Vendor CAPs. Anything 0 in size is a none populated memory region. Do this 
for both sides. (Reference commit ID: d2e21a50)

Second, the misx-table is popluated at bar2 not bar4 as before. The uio driver 
can handle this, but the code in ivshmem_demo cannot. You have to change the 
process of "map_shmem_and_bars." Msix-table address should be written to 
PCI_CFG_CAP + 8, which is BAR2 (was hard coded 16 to point to BAR4).

Third, the MMIO registers changed index, ivpos, a.k.a ID should be 0 instead 8.
The doorbell register should be 4 instead 12 (counting in bytes). Change the 
non-root driver accordingly. On root Linux, mmaped regions are typically used 
to poke the mmio region. Therefore, change where you poke accordingly.

Finally, on non-root side don't forget to make the probing process to recognize 
the right ivshmem version. Class and reversion read from CAP would have been 
ored with additional 0x2. On root Linux, you can just force the uio_ivshmem 
driver to use jailhouse mode by commenting out some codes.

Hope someone can benifit from this.
If you had any problem making the change, I would be happy to help.

Yang




-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Interrupt with UIO and ivshmem2

[Chung-Fan Yang]

Hello,

I am working on ivshmem-net to achieve communication between host and cell.
Therefore, I applied the ivshmem2 patch both on the jailhouse and root-cell 
linux kernel (4.9).

The root side's ivhsmem-net driver is correctly started and I can setup an ip 
address without problem. The non-root side is laking a Ethernet deriver. Thus, 
not checked.

Together I wish that my old implementation on uio_ivshmem and code ported from 
ivshmem_demo can still work on the new model.

Therefore, both root and non-root have 2 ivshmem PCI device. One isfor 
supporting the old model, an other for ivshmem-net.

I read the commits, found out that the memory locations in ivehmem2 has 
expanded to 3 per PCI device. I adapted it and changed both the uio_ivshmem and 
non-root ivshmem driver. I can read and write the shared memory without problem.

But I am current having problem on send and reviving IPIs between root and 
non-root cells. I noticed the following things in the ivshmem2 commits, which 
might affect interrupts:

1. The MMIO registers, including the doorbell register's location had changed 
.(64f5b8fe)
2. MMIO region is expanded to 4K. (27d3ad63)
3. BAR4 is move to BAR2. (b4e9474c)

For 1, I changed the index used to access MMIO door bell register. It was 12 I 
changed to 4. But it's still not working.

For 2, I don't really think it's a problem, because the mapped region is larger 
than before.

For 3, I read the code of ivshmem_demo. It used to use BAR2 to setup msix_table 
for APIC and MIS-X mapping. I am suspecting that moving the BAR4 to BAR2 makes 
this process some how broken, but I have no idea what happened and how to fix 
it.

Anyone had used the old uio driver on ivshmem2 and successful talked with other 
cells before?
The configs are attached, I can attach part of the code if requested.
I could really use some help.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Minimal configuration for demo inmates, 1 CPU, 1 MB RAM, 1 serial port
 *
 * Copyright (c) Siemens AG, 2013
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 */

#include 
#include 

#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])

struct {
	struct jailhouse_cell_desc cell;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[9];
	struct jailhouse_cache cache_regions[1];
	struct jailhouse_irqchip irqchips[2];
	__u8 pio_bitmap[0x2000];
	struct jailhouse_pci_device pci_devices[2];
	struct jailhouse_pci_capability pci_caps[0];
} __attribute__((packed)) config = {
	.cell = {
		.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.name = "non-root",

		.cpu_set_size = sizeof(config.cpus),
		.num_memory_regions = ARRAY_SIZE(config.mem_regions),
		.num_cache_regions = ARRAY_SIZE(config.cache_regions),
		.num_irqchips = ARRAY_SIZE(config.irqchips),
		.pio_bitmap_size = ARRAY_SIZE(config.pio_bitmap),
		.num_pci_devices = 2,
	},

	.cpus = {
		0x8,
	},

	.mem_regions = {
		/* RAM */ {
			.phys_start = 0x410,
			.virt_start = 0,
			.size = 0x4000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
		},
		/* communication region */ {
			.virt_start = 0x4000,
			.size = 0x1000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_COMM_REGION,
		},
		/* IVSHMEM shared memory region */
		{
			.phys_start = 0x4410,
			.virt_start = 0x40001000,
			.size = 0x12,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_ROOTSHARED,
		},
{ 0 },
{ 0 },
		/* IVSHMEM-net shared memory region */
		{ 0 },
		{
			.phys_start = 0x4422,
			.virt_start = 0x4020,
			.size = 0x8,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_ROOTSHARED,
		},
		{
			.phys_start = 0x442a,
			.virt_start = 0x4028,
			.size = 0x8,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_ROOTSHARED,
		},
	},

	.cache_regions = {
		{
			.start = 0,
			.size = 10,
			.type = JAILHOUSE_CACHE_L3,
		},
	},

	.irqchips = {
		/* IOAPIC */ {
			.address = 0xfec0,
			.id = 0x1f0ff,
			.pin_bitmap = {
0x10 /* ACPI IRQ */
			},
		},
		/* IOAPIC 2, GSI base 24 */
		{
			.address = 0xfec01000,
			.id = 0x1002c,
			.pin_bitmap = {
0x00
			},
		},
	},

	.pio_bitmap = {
		[ 0/8 ...  0x2f7/8] = -1,
		[ 0x2f8/8 ...  0x2ff/8] = 0, /* serial2 */
		[ 0x300/8 ...  0x3f7/8] = -1,
		[ 0x3f8/8 ...  0x3ff/8] = 0, /* serial1 */
		[ 0x400/8 ... 0xe00f/8] = -1,
		[0xe010/8 ... 0xe017/8] = 0, /* OXPCIe952 serial1 */
		[0xe018/8 ... 0x/8] = -1,
	},
	.pci_devices = {
		{
			.type = 

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年10月5日金曜日 16時18分41秒 UTC+9 Chung-Fan Yang:
> 2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> > On 04.10.18 10:46, Chung-Fan Yang wrote:
> > > Hello,
> > > 
> > > I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 machine.
> > > I am currently trying to implement interrupt driven seril output. 
> > > Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to the 
> > > RTOS.
> > > 
> > > I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to remap 
> > > IRQ4.
> > > The IOAPIC demo works fine, but not when I use the same technique to 
> > > remap IRQ4, the following error message is printed on the jailhouse 
> > > console(ttyS1).
> > > 
> > > VT-d fault event reported by IOMMU 1:
> > >   Source Identifier (bus:dev.func): f0:1f.7
> > >   Fault Reason: 0x26 Fault Info: 6a0 Type 0
> > > 
> > > I try to read the VT-d manual, but I can't quite get the point why this 
> > > is happening. So I would kindly to ask help here. I have attached the 
> > > sysconfig and RTOS's config.
> > 
> > You need to align the ID of the IOAPIC in the non-root cell config with the 
> > root 
> > cell. That's 0x1f0ff in your case.
> > 
> > Jan
> 
> Thank you. This do the trick. The error is gone.
> 
> However, the interrupt is not triggered on CPU when 16550 asserted it 
> (polling IIR register reveals there was an interrupt).
> I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the same 
> technique in ioapic-demo.
> It seems like pin 4 was always high, although 16550 should be sending edge 
> triggering signals.
> 
> I am wondering that my board has a different connection from 16550 to the 
> IOAPIC, which is unlike traditional XT-PIC, but not so sure.
> 
> Yang

I didn't find the MADT table in the /sys/firmware/acpi, assuming my baord 
firmware is missing it.
According to the BIOS config, COM1 is at 0x3f8 with IRQ=4.
I tried to map IRQ 1~7, none of them works. IRQ 9 is indeed PM timer.
This is starnge. Where did that UART IRQ go?

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remaping IRQ of ttyS0

[Chung-Fan Yang]

2018年10月4日木曜日 18時13分50秒 UTC+9 Jan Kiszka:
> On 04.10.18 10:46, Chung-Fan Yang wrote:
> > Hello,
> > 
> > I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 machine.
> > I am currently trying to implement interrupt driven seril output. 
> > Therefore, I tried to remap the IRQ4, which is the ttyS0's IRQ, to the RTOS.
> > 
> > I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to remap IRQ4.
> > The IOAPIC demo works fine, but not when I use the same technique to remap 
> > IRQ4, the following error message is printed on the jailhouse 
> > console(ttyS1).
> > 
> > VT-d fault event reported by IOMMU 1:
> >   Source Identifier (bus:dev.func): f0:1f.7
> >   Fault Reason: 0x26 Fault Info: 6a0 Type 0
> > 
> > I try to read the VT-d manual, but I can't quite get the point why this is 
> > happening. So I would kindly to ask help here. I have attached the 
> > sysconfig and RTOS's config.
> 
> You need to align the ID of the IOAPIC in the non-root cell config with the 
> root 
> cell. That's 0x1f0ff in your case.
> 
> Jan

Thank you. This do the trick. The error is gone.

However, the interrupt is not triggered on CPU when 16550 asserted it (polling 
IIR register reveals there was an interrupt).
I mapped pin 4 of IOAPIC to vector number 36 of the LAPIC, using the same 
technique in ioapic-demo.
It seems like pin 4 was always high, although 16550 should be sending edge 
triggering signals.

I am wondering that my board has a different connection from 16550 to the 
IOAPIC, which is unlike traditional XT-PIC, but not so sure.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Remaping IRQ of ttyS0

[Chung-Fan Yang]

Hello,

I am using jailhouse to run a RTOS(nuttx) on a Xeon E5-2630 v4 machine.
I am currently trying to implement interrupt driven seril output. Therefore, I 
tried to remap the IRQ4, which is the ttyS0's IRQ, to the RTOS.

I try to mimic IOAPIC demo's procedure of reampping ACPI IRQ to remap IRQ4. 
The IOAPIC demo works fine, but not when I use the same technique to remap 
IRQ4, the following error message is printed on the jailhouse console(ttyS1).

VT-d fault event reported by IOMMU 1:
 Source Identifier (bus:dev.func): f0:1f.7
 Fault Reason: 0x26 Fault Info: 6a0 Type 0

I try to read the VT-d manual, but I can't quite get the point why this is 
happening. So I would kindly to ask help here. I have attached the sysconfig 
and RTOS's config.

Yang.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Minimal configuration for demo inmates, 1 CPU, 1 MB RAM, 1 serial port
 *
 * Copyright (c) Siemens AG, 2013
 *
 * Authors:
 *  Jan Kiszka 
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 */

#include 
#include 

#define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0])

struct {
	struct jailhouse_cell_desc cell;
	__u64 cpus[1];
	struct jailhouse_memory mem_regions[4];
	struct jailhouse_cache cache_regions[1];
	struct jailhouse_irqchip irqchips[1];
	__u8 pio_bitmap[0x2000];
	struct jailhouse_pci_device pci_devices[1];
	struct jailhouse_pci_capability pci_caps[0];
} __attribute__((packed)) config = {
	.cell = {
		.signature = JAILHOUSE_CELL_DESC_SIGNATURE,
		.revision = JAILHOUSE_CONFIG_REVISION,
		.name = "nuttx",

		.cpu_set_size = sizeof(config.cpus),
		.num_memory_regions = ARRAY_SIZE(config.mem_regions),
		.num_cache_regions = ARRAY_SIZE(config.cache_regions),
		.num_irqchips = ARRAY_SIZE(config.irqchips),
		.pio_bitmap_size = ARRAY_SIZE(config.pio_bitmap),
		.num_pci_devices = ARRAY_SIZE(config.pci_devices),
		.num_pci_caps = ARRAY_SIZE(config.pci_caps),
	},

	.cpus = {
		0x8,
	},

	.mem_regions = {
		/* RAM */ {
			.phys_start = 0x3f00,
			.virt_start = 0,
			.size = 0x0400,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE,
		},
		/* communication region */ {
			.virt_start = 0x0400,
			.size = 0x1000,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_COMM_REGION,
		},
		/* IVSHMEM shared memory region */
		{
			.phys_start = 0x4300,
			.virt_start = 0x04001000,
			.size = 0x12,
			.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_ROOTSHARED,
		},
	},

	.cache_regions = {
		{
			.start = 0,
			.size = 10,
			.type = JAILHOUSE_CACHE_L3,
		},
	},

	.irqchips = {
		/* IOAPIC */ {
			.address = 0xfec0,
			.id = 0xff01,
			.pin_bitmap = {
0x10 /* ttyS0 IRQ */
			},
		},
	},

	.pio_bitmap = {
		[ 0/8 ...  0x2f7/8] = -1,
		[ 0x2f8/8 ...  0x2ff/8] = 0, /* serial2 */
		[ 0x300/8 ...  0x3f7/8] = -1,
		[ 0x3f8/8 ...  0x3ff/8] = 0, /* serial1 */
		[ 0x400/8 ... 0xe00f/8] = -1,
		[0xe010/8 ... 0xe017/8] = 0, /* OXPCIe952 serial1 */
		[0xe018/8 ... 0x/8] = -1,
	},

	.pci_devices = {
		{
			.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
			.domain = 0x,
			.bdf = 0x0f << 3,
			.bar_mask = {
0xff00, 0x, 0x,
0x, 0xffe0, 0x,
			},
			.num_msix_vectors = 1,
			.shmem_region = 2,
		},
	},
};
/*
 * Jailhouse, a Linux-based partitioning hypervisor
 *
 * Copyright (c) Siemens AG, 2014-2017
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 *
 * Alternatively, you can use or redistribute this file under the following
 * BSD license:
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 * 1. Redistributions of source code must retain the above copyright
 *notice, this list of conditions and the following disclaimer.
 *
 * 2. Redistributions in binary form must reproduce the above copyright
 *notice, this list of conditions and the following disclaimer in the
 *documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS 

Re: Root cell not receiving ivshmem interrupt

[Chung-Fan Yang]

2018年8月17日金曜日 1時53分29秒 UTC+9 J. Kiszka:
> On 2018-08-16 18:25, Chung-Fan Yang wrote:
> > Hello,
> > 
> > I would like some help. I am trying to accomplish a root <-> non-root cell 
> > communication.
> > I followed README.jailhouse in 
> > https://github.com/henning-schild-work/ivshmem-guest-code to build to tools 
> > for the root side and fired the ivshmem-demo as the non-root cell.
> > 
> > I can see the interrupt of virtual pci interface is enumerated in 
> > /proc/interrupts
> >   34:  0  0  0  0  0  0 
> >  0  0  0  IR-PCI-MSI 245760-edge  uio_ivshmem
> > 
> > 
> > I am able to send interrupt from root cell to non-root cell. I can see 
> > responses from the debug console. Also, I can read the memory data in the 
> > root cell.
> > But I am not able to receive the interrupts from non-root cell. using the 
> > uio_read tool, it will hang for ever, and the count in /proc/interrupts 
> > wouldn't increase.
> > 
> > I am using the head of the jailhouse branch of ivshmem-guest-code.
> > The kernel version is 4.13, and 4.9, both are not working.
> > 
> 
> Are you running on real hardware or in qemu/kvm? For real hw, if there 
> are two IOMMU units, try assigning the ivshmem PCI device to the other 
> (likely unit 1).

This is exactly the reason it is not working.
I am using real hw with 2 iommu unit. After setting iommu=1 for the virtual 
device it worked.

Thank you guys,

Yang

> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Root cell not receiving ivshmem interrupt

[Chung-Fan Yang]

Hello,

I would like some help. I am trying to accomplish a root <-> non-root cell 
communication.
I followed README.jailhouse in 
https://github.com/henning-schild-work/ivshmem-guest-code to build to tools for 
the root side and fired the ivshmem-demo as the non-root cell.

I can see the interrupt of virtual pci interface is enumerated in 
/proc/interrupts
 34:  0  0  0  0  0  0  
0  0  0  IR-PCI-MSI 245760-edge  uio_ivshmem


I am able to send interrupt from root cell to non-root cell. I can see 
responses from the debug console. Also, I can read the memory data in the root 
cell.
But I am not able to receive the interrupts from non-root cell. using the 
uio_read tool, it will hang for ever, and the count in /proc/interrupts 
wouldn't increase.

I am using the head of the jailhouse branch of ivshmem-guest-code.
The kernel version is 4.13, and 4.9, both are not working.

Thank you,

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unsupported MSI/MSI-X state crash on x86 jailhouse startup

[Chung-Fan Yang]

2018年8月17日金曜日 0時02分10秒 UTC+9 Henning Schild:
> Am Thu, 16 Aug 2018 05:52:45 -0700
> schrieb Chung-Fan Yang :
> 
> > Hello,
> > 
> > I am using jailhouse on my Xeon machine with a super micro
> > motherboard. I was using kernel 4.4, which works fine, but upgraded
> > to kernel 4.13 today.
> > 
> > After the update, when I enable jailhouse, it crashed with
> > 
> > FATAL: Unsupported MSI/MSI-X state, device 0a:00.0, cap 5
> 
> You could try to see if CONFIG_TRACE_ERROR gives you more information,
> otherwise you could add printks to trace the error back to its origin.
> Looking at the code "iommu_map_interrupt" in hypervisor/arch/x86/vtd.c
> is likely the source of the error.
> 
> > which never happened with kernel version 4.4. 
> > 
> > I checked with lspci, 0a:00.0 is a PCI bridge.
> > 0a:00.0 PCI bridge: ASMedia Technology Inc. Device
> > 1182
> > 
> 
> If nothing is behind that bridge - or nothing of interest. You can just
> remove it - and every device behind it - from the root cell config. And
> then unbind all drivers from all the pci devices before "jailhouse
> enable". That would be a workaround where jailhouse would not have to
> deal with the device. But it would still be interesting to know what is
> causing your problem.

I think there isn't any interesting behind the bridge. Thus, yes, I am trying 
to do this as a workaround right now. It seems working, not 100% sure.

I am also interested about the cause of this. I will try to trace it if I have 
some spare time.

Thank You,

Yang

> 
> Henning
> 
> > I tried kernel version 4.9, 4.13.4.14, which all crashed.
> > I am seeking some help.
> > 
> > The kernel config is appended for reference.
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unsupported MSI/MSI-X state crash on x86 jailhouse startup

[Chung-Fan Yang]

2018年8月16日木曜日 22時47分26秒 UTC+9 Henning Schild:
> Am Thu, 16 Aug 2018 05:52:45 -0700
> schrieb Chung-Fan Yang :
> 
> > Hello,
> > 
> > I am using jailhouse on my Xeon machine with a super micro
> > motherboard. I was using kernel 4.4, which works fine, but upgraded
> > to kernel 4.13 today.
> > 
> > After the update, when I enable jailhouse, it crashed with
> > 
> > FATAL: Unsupported MSI/MSI-X state, device 0a:00.0, cap 5
> > 
> > which never happened with kernel version 4.4. 
> > 
> > I checked with lspci, 0a:00.0 is a PCI bridge.
> > 0a:00.0 PCI bridge: ASMedia Technology Inc. Device
> > 1182
> >   
> > 
> > I tried kernel version 4.9, 4.13.4.14, which all crashed.
> > I am seeking some help.
> > 
> > The kernel config is appended for reference.
> > 
> 
> Which version of jailhouse are you using?
> 
> Henning

I was using v0.9.1, but also tried with current master.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Jailhouse debug console on COM2 via SOL

[Chung-Fan Yang]

2018年4月20日金曜日 22時51分53秒 UTC+9 Francois Ozog:
> HI,
> 
> 
> Did you do the test with the main Linux, jailhouse not started? so that you 
> first validate that the complete communication chain works.
> 
> 
> In addition to UART_BASE you also have a UART_DIVIDER option that controls 
> setting of the baud rate.
> 
> 
> FF
> 
> 
> On 20 April 2018 at 10:39, Chung-Fan Yang <sonic...@gmail.com> wrote:
> 2018年4月20日金曜日 17時27分46秒 UTC+9 Ralf Ramsauer:
> 
> > On 04/20/2018 10:23 AM, Chung-Fan Yang wrote:
> 
> > > I can get the Jailhouse to output on SOL now, but apic-demo still prints 
> > > nothing.
> 
> > > I did see it went up into running from jailhouse's debug information.
> 
> > > 
> 
> > Did you set the correct UART_BASE in inmates/lib/x86/printk.c?
> 
> > 
> 
> >   Ralf
> 
> 
> 
> This is the problem.
> 
> I set it to 0x2f8 now, and getting the print outs.
> 
> 
> 
> Thank you, Ralf
> 
> 
> 
> Yang
> 
> 
> 
> 
> 
> -- 
> 
> You received this message because you are subscribed to the Google Groups 
> "Jailhouse" group.
> 
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jailhouse-de...@googlegroups.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> 
> 
> -- 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> François-Frédéric Ozog | Director Linaro Networking GroupT: +33.67221.6485
> franco...@linaro.org | Skype: ffozog

Hi,

I did check the communication channel without jailhouse.
The problem was that I messed up with the generated sysconfig.

And, thanks for the tip on baud rate.

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Jailhouse debug console on COM2 via SOL

[Chung-Fan Yang]

2018年4月20日金曜日 17時27分46秒 UTC+9 Ralf Ramsauer:
> On 04/20/2018 10:23 AM, Chung-Fan Yang wrote:
> > I can get the Jailhouse to output on SOL now, but apic-demo still prints 
> > nothing.
> > I did see it went up into running from jailhouse's debug information.
> > 
> Did you set the correct UART_BASE in inmates/lib/x86/printk.c?
> 
>   Ralf

This is the problem.
I set it to 0x2f8 now, and getting the print outs.

Thank you, Ralf

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Jailhouse debug console on COM2 via SOL

[Chung-Fan Yang]

I can get the Jailhouse to output on SOL now, but apic-demo still prints 
nothing.
I did see it went up into running from jailhouse's debug information.

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jailhouse debug console on COM2 via SOL

[Chung-Fan Yang]

Hello everyone,

I am trying out Jailhouse on a Xeon based machine which has IPMI SOL.

I would like to use the IPMI SOL, which is COM2, as the Jailhouse debug console.
Thus I changed the debug address to 0x2f8 in the generated sysconfig.c.

Although I can have the jailhouse enabled and apic-demo running, I can't see 
any thing coming out form the SOL channel, either Jailhouse debug nor apic-demo 
output. I can only see things being logged in /dev/jailhouse.

I have confirmed that SOL(/dev/ttyS1) address is 0x2f8 by using dmesg, and it's 
able to send message with other hosts using minicom.

Any idea where went wrong?

Best Regards,

Yang

-- 
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.