On 7/1/19 5:44 PM, Jan Kiszka wrote:
> On 01.07.19 17:11, Ralf Ramsauer wrote:
>> On 7/1/19 4:04 PM, Jan Kiszka wrote:
>>> On 01.07.19 15:52, Ralf Ramsauer wrote:
>>>>
>>>>
>>>> On 7/1/19 3:09 PM, Jan Kiszka wrote:
>>>>> On 01.07.19 14:59, Ralf Ramsauer wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 6/27/19 9:06 AM, Jan Kiszka wrote:
>>>>>>> On 25.06.19 19:25, Ralf Ramsauer wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> for the completeness sake: it's about ivshmem-net. The PCI device
>>>>>>>> shows
>>>>>>>> up in the root cell and can be discovered via lspci, but the driver
>>>>>>>> fails while probing with
>>>>>>>>
>>>>>>>> [17061.414176] ivshmem-net 0000:00:01.0: enabling device (0000 ->
>>>>>>>> 0002)
>>>>>>>> [17061.420598] ivshmem-net 0000:00:01.0: invalid IVPosition -1
>>>>>>>
>>>>>>> The register read-out failed. Maybe a mismatch between driver and
>>>>>>> Jailhouse version: Which revisions are you using on both sides?
>>>>>>
>>>>>> siemens/4.19-rt vs. jailhouse/next. Should match.
>>>>>>
>>>>>> The bar_mask was copied over from the qemu demo. Other than that, the
>>>>>> only thing that changed is the bdf. We simply chose a free one on our
>>>>>> system.
>>>>>>
>>>>>> The memory region behin ivshmem is high memory above 32-bit. I
>>>>>> instrumented and checked the code, but that shouldn't be a problem.
>>>>>
>>>>> This is rather related to the MMIO register access. Check if reading
>>>>> that ID/IVPos register actually triggers a VM exit. I suspect it
>>>>> doesn't.
>>>>
>>>> Hmm. Correct. I guess we should end up in ivshmem_register_mmio()
>>>> but we
>>>> don't.
>>>>
>>>> For bar0, jailhouse registers MMIO 0x380000000000. This is in sync with
>>>> the kernel:
>>>> [ 1416.878650] pci 0000:00:01.0: BAR 0: assigned [mem
>>>> 0x380000000000-0x3800000000ff 64bit]
>>>>
>>>> That's odd. Actually we should trap. Instrumentation of ivshmem-net
>>>> below gives me:
>>>>
>>>> [ 2044.832898] regs location: 4080053db000
>>>>
>>>> Huh? Shouldn't that be 0x380000000000?
>>>
>>> What's "regs location"? What does "lspci -vv -s 0000:00:01.0" report?
>>
>> pr_err("regs location: %llx\n", virt_to_phys(regs));
>
> Calling virt_to_phys on ioremapped memory may not work. virt_to_phys is
> primarily (if not only) thought for calculating the address for a kernel
> piece of RAM.
>
>>
>> Please find the output of lspci attached.
>>
>
> That looks consistent.
>
> Did you check that there is no accidental mapping of that virtual
> address so something else? If not, check earlier in the interception
> path if there is a VM-exit, but we just do not end up in ivshmem for it.
Got it running. The issue was that the config generator fully mapped all
PCI Bus regions:
/* MemRegion: 380000000000-380fffffffff : PCI Bus 0000:00 */
{
.phys_start = 0x380000000000,
.virt_start = 0x380000000000,
.size = 0x1000000000,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
},
[...]
So access wasn't intercepted at all as those pages were fully mapped.
After commenting out all of those regions everything works as expected
(well, not tested against another endpoint, but at least ivshmem-net
successfully probes).
The config generator created those regions. Happens on master, next and
older versions. I guess this happens as those regions don't have any
siblings -- they should probably be filtered out. Find the iomem attached.
BTW: This behavior can be reconstructed by running the config generator
on the qemu virtual target for x86.
Thanks
Ralf
>
>>>
>>> Hmm 64-bit... FWIW, I'm going to remove that "feature" from future
>>> ivshmem again, moving things back to 32-bit address space.
>>
>> But we do have:
>>
>> 380000000000-380fffffffff : PCI Bus 0000:00
>> 381000000000-381fffffffff : PCI Bus 0000:16
>> 382000000000-382fffffffff : PCI Bus 0000:64
>> 383000000000-383fffffffff : PCI Bus 0000:b2
>>
>>>
>>> That said, this constellation may have triggered an issue in ivshmem or
>>> even the MMIO dispatcher that wasn't visible so far.
>>
>> But will moving the memory region to 32-bit address space solve the
>> issue in this case?
>
> Can't tell as we do not know the root cause yet. But you can already try
> to remove PCI_BAR_64BIT from bar[0] initialization in
> hypervisor/ivshmem.c and check what changes.
>
> Jan
>
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/dd55c1a5-8863-5f94-b339-bebc221dfc6f%40oth-regensburg.de.
For more options, visit https://groups.google.com/d/optout.
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000a0000-000fffff : Reserved
000a0000-000bffff : PCI Bus 0000:00
000c4000-000cbfff : PCI Bus 0000:00
000f0000-000fffff : System ROM
00100000-5411001f : System RAM
01000000-01e071d0 : Kernel code
01e071d1-0256cd3f : Kernel data
02af4000-02bfffff : Kernel bss
54110020-5412865f : System RAM
54128660-5412901f : System RAM
54129020-5415a45f : System RAM
5415a460-541f001f : System RAM
541f0020-5422145f : System RAM
54221460-542c701f : System RAM
542c7020-542cf05f : System RAM
542cf060-56101fff : System RAM
56102000-5e109fff : Reserved
5e10a000-5f531fff : System RAM
5f532000-5f60ffff : Reserved
5f610000-611bdfff : System RAM
611be000-651bdfff : Reserved
651be000-6c2cefff : System RAM
6c2cf000-6e7cefff : Reserved
6e7cf000-6f5fefff : ACPI Non-volatile Storage
6f5ff000-6f7fefff : ACPI Tables
6f7ff000-6f7fffff : System RAM
6f800000-8fffffff : Reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
90000000-aaffffff : PCI Bus 0000:00
90000000-900fffff : PCI Bus 0000:01
90000000-90007fff : 0000:01:00.0
90100000-901fffff : PCI Bus 0000:04
90100000-9013ffff : 0000:04:00.0
90140000-9017ffff : 0000:04:00.1
91000000-91ffffff : PCI Bus 0000:02
91000000-91ffffff : PCI Bus 0000:03
91000000-91ffffff : 0000:03:00.0
91000000-912fffff : efifb
92000000-928fffff : PCI Bus 0000:02
92000000-928fffff : PCI Bus 0000:03
92000000-927fffff : 0000:03:00.0
92808000-9280bfff : 0000:03:00.0
92900000-929fffff : PCI Bus 0000:01
92900000-92907fff : 0000:01:00.0
92a00000-92afffff : PCI Bus 0000:04
92a00000-92a0ffff : 0000:04:00.1
92a00000-92a0ffff : tg3
92a10000-92a1ffff : 0000:04:00.1
92a10000-92a1ffff : tg3
92a20000-92a2ffff : 0000:04:00.1
92a20000-92a2ffff : tg3
92a30000-92a3ffff : 0000:04:00.0
92a30000-92a3ffff : tg3
92a40000-92a4ffff : 0000:04:00.0
92a40000-92a4ffff : tg3
92a50000-92a5ffff : 0000:04:00.0
92a50000-92a5ffff : tg3
92b00000-92b7ffff : 0000:00:17.0
92b00000-92b7ffff : ahci
92b80000-92bfffff : 0000:00:11.5
92b80000-92bfffff : ahci
92c00000-92c0ffff : 0000:00:14.0
92c10000-92c13fff : 0000:00:1f.2
92c14000-92c15fff : 0000:00:17.0
92c14000-92c15fff : ahci
92c16000-92c17fff : 0000:00:11.5
92c16000-92c17fff : ahci
92c18000-92c180ff : 0000:00:1f.4
92c19000-92c19fff : 0000:00:16.4
92c1a000-92c1afff : 0000:00:16.1
92c1b000-92c1bfff : 0000:00:16.0
92c1c000-92c1cfff : 0000:00:14.2
92c1e000-92c1e0ff : 0000:00:17.0
92c1e000-92c1e0ff : ahci
92c1f000-92c1f0ff : 0000:00:11.5
92c1f000-92c1f0ff : ahci
92c20000-92c20fff : 0000:00:05.4
aaffc000-aaffcfff : dmar3
ab000000-c5ffffff : PCI Bus 0000:16
ab000000-ab0fffff : PCI Bus 0000:17
ab000000-ab007fff : 0000:17:00.0
ab100000-ab100fff : 0000:16:05.4
ab200000-ab2fffff : PCI Bus 0000:17
ab200000-ab207fff : 0000:17:00.0
c5ffc000-c5ffcfff : dmar0
c6000000-e0ffffff : PCI Bus 0000:64
c6000000-c6000fff : 0000:64:05.4
e0ffc000-e0ffcfff : dmar1
e1000000-fbffffff : PCI Bus 0000:b2
e1000000-e11fffff : PCI Bus 0000:b3
e1000000-e10fffff : 0000:b3:00.0
e1100000-e110ffff : 0000:b3:00.0
e1100000-e110ffff : megasas: LSI
e1200000-e1200fff : 0000:b2:05.4
fbffc000-fbffcfff : dmar2
fd000000-fdabffff : pnp 00:04
fdad0000-fdadffff : pnp 00:04
fdb00000-fdffffff : pnp 00:04
fdc6000c-fdc6000f : iTCO_wdt
fe000000-fe010fff : Reserved
fe000000-fe00ffff : pnp 00:04
fe010000-fe010fff : PCI Bus 0000:00
fe010000-fe010fff : 0000:00:1f.5
fe011000-fe01ffff : pnp 00:04
fe036000-fe03bfff : pnp 00:04
fe03d000-fe3fffff : pnp 00:04
fe410000-fe7fffff : pnp 00:04
fec00000-fecfffff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec08000-fec083ff : IOAPIC 2
fec10000-fec103ff : IOAPIC 3
fec18000-fec183ff : IOAPIC 4
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed3ffff : pnp 00:01
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
ff000000-ffffffff : pnp 00:01
100000000-2ffffffff : Reserved
300000000-87fffffff : System RAM
380000000000-380fffffffff : PCI Bus 0000:00
381000000000-381fffffffff : PCI Bus 0000:16
382000000000-382fffffffff : PCI Bus 0000:64
383000000000-383fffffffff : PCI Bus 0000:b2
