[+Cc'ed Stefan, a colleague of mine who recently used non-root Linux on
x86 systems]
Hi
On 12/01/2024 14:07, Michele Pescapè wrote:
Hi,
The config is the correct one. Nothing from dmesg except
[ 8254.074890] smpboot: CPU 8 is now offline
[ 8254.206652] smpboot: CPU 9 is now offline
[ 8254.254775] Created Jailhouse cell "linux-2"
The hypervisor only reports:
CPU 9 received SIPI, vector 96 (this is the last line of the
output from the previous attempt)
Created cell "linux-2" (first line after issuing the cell linux
command)
Page pool usage after cell creation: mem 272/32211, remap 16392/131072
Cell "linux-2" can be loaded
I tried using the -w flag, this is what it said when using :
jailhouse cell create configs/x86/linux_guest2.cell
jailhouse cell load linux-2 linux-loader.bin -a 0x0
../buildroot-2023.11/output/images/bzImage -a 0xffbe00 parameters -a 0x1000
jailhouse cell start linux-2
I take it the kernel is loaded at 0xffbe00 which is right at the edge of
the low ram region. In fact after issuing the cell load command and
Exactly, bzImage won't fit in here - that's the culprit. And I don't
know why cell-linux places it there.
Stefan, CC'ed, recently enabled non-root Linux on one of our test
machines. Stefan, we finally got non-root Linux running, right?
We had other issues -- but why didn't we have that issue here?
Even if I try to use the qemu-demo config, I basically have the same issue:
$ ./tools/jailhouse-cell-linux \
-w test.params -a x86 ./configs/x86/linux-x86-demo.cell \
bzImage -c "console=foobar"
gives:
jailhouse cell create ./configs/x86/linux-x86-demo.cell
jailhouse cell load linux-x86-demo linux-loader.bin -a 0x0
/home/ralf/workspace/linux/O/x86-qemu/arch/x86/boot/bzImage -a 0xffc800
test.params -a 0x1000
jailhouse cell start linux-x86-demo
And same here: 0xffc800 is odd. What does our machine (dodecarya) print
here?
adjusting the path for the loader I get JAILHOUSE_CELL_LOAD: Invalid
argument.
Nothing is said from dmesg, the hypervisor only says "Cell "linux-2" can
be loaded".
At this point I tried pointing the address for the kernel in the cell
load command to the high ram region by giving -a 0x42300000 as the address.
Dont' try to work around, we need to get the loader script working, the
error is somewhere here. Manually adjusting the load address makes
things worse, as on x86, the params file contains the so-called
zero-page, and adresses that are encoded there must fit. That's why we
have the cell-linux helper script that simplifies things.
We don't have to do that ugly zero-page dance on other architectures.
I'll try to do some testing with our machine when I find some time.
Thanks
Ralf
I get the following output when starting the cell:
Created cell "linux-2"
Page pool usage after cell creation: mem 270/32211, remap 16392/131072
Cell "linux-2" can be loaded
Started cell "linux-2"
CPU 9 received SIPI, vector 100
CPU 8 received SIPI, vector 100
FATAL: unable to get MMIO instruction
FATAL: Invalid MMIO/RAM read, addr: 0x0000000001000000 size: 0
RIP: 0x0000000001000200 RSP: 0x0000000000009fe8 FLAGS: 46
RAX: 0x0000000001000200 RBX: 0x000000000000427d RCX: 0x00000000000000ff
RDX: 0x0000000000000000 RSI: 0x0000000000001000 RDI: 0x0000000000000000
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080010011 CR3: 0x0000000000005000 CR4: 0x0000000000040220
EFER: 0x0000000000001500
Parking CPU 9 (Cell: "linux-2")
FATAL: Unexpected #VMEXIT, exitcode 7f, exitinfo1 0x0000000000000000
exitinfo2 0x0000000000000000
RIP: 0x000000000000000f RSP: 0x0000000000009fe8 FLAGS: 2
RAX: 0x0000000001000201 RBX: 0x000000000000427d RCX: 0x00000000000000ff
RDX: 0x0000000000000000 RSI: 0x0000000000001000 RDI: 0x0000000000000000
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000001000201 CR3: 0x0000000000005000 CR4: 0x0000000000040220
EFER: 0x0000000000001500
And then it continues to display the vmexit error without stopping if
the cell isn't destroyed. I can avoid this by increasing the size of the
low ram region to 0x00200000.
That allows me to proceed and starting the cell with cell start, the
hypervisor output being:
Created cell "linux-2"
Page pool usage after cell creation: mem 272/32211, remap 16392/131072
Cell "linux-2" can be loaded
Started cell "linux-2"
CPU 9 received SIPI, vector 100
CPU 8 received SIPI, vector 100
but without any other sign.
I still need to decrease the kernel size, i only reduced it by 1MB.
I'll attach again the non root config, however it isn't much different
than the other I sent, i only fixed the weird alignment and changed a
bit the start addresses of the comm region and the high ram region.
Thanks,
Michele
Il giorno giovedì 11 gennaio 2024 alle 21:55:47 UTC+1 Ralf Ramsauer ha
scritto:
Hi,
On 11/01/2024 13:57, Michele Pescapè wrote:
> Hi,
>
> after changing the virt_start to match the phys_start for that
memory
> region I got an invalid argument error when starting the cell
>
> Traceback (most recent call last):
> File "/usr/local/libexec/jailhouse/jailhouse-cell-linux", line
739,
> in <module>
> cell.load(arch.kernel_image, arch.kernel_address())
> File
"/usr/local/lib/python3.10/dist-packages/pyjailhouse/cell.py",
> line 44, in load
> fcntl.ioctl(self.dev <http://self.dev>,
self.JAILHOUSE_CELL_LOAD, load)
> OSError: [Errno 22] Invalid argument
Are you sure that you created the cell with the correct config? What
does dmesg say? Any output on the hypervisor console when this error
happens?
>
> Printing the arch.kernel_address() returned 16760320 (should be
> 0xffbe00), so I thought that the bzImage was too big for the low
ram region.
jailhouse-cell-linux won't load the bzImage to the low ram region. You
can use the -w option to dump a file that prints the commands that
jailhouse-cell-linux would invoke.
> I increased the size to .size = 0x04000000 and shifted the virt and
> phys_start of the other two regions to match the change
(virt_start =
> 0x04000000 for the comm region and virt=phys_start = 0x46000000
for the
> third region) .
> No issues were reported by config check but after starting the
cell i
> got no output even using earlyprintk.
>
> I also tried using an initrd but I got a similar issue, invalid
argument
> when starting the cell, this time it was the arch.ramdisk_address()
> which was 107327488 (0x665b000).
> So i tried increasing even more the size of the first region to
size =
> 0x0a000000 (comm region now starts at .virt_start = 0x0a000000,
and the
> third region starts at virt=.phys_start = 0x4c000000 with size =
> 0x6000000).
> Again, no config check issues so I started it but no output
again, the
> hypervisor only reports:
>
> Created cell "linux-1"
> Page pool usage after cell creation: mem 268/32211, remap
16392/131072
> Cell "linux-1" can be loaded
> CPU 5 received SIPI, vector 100
> CPU 4 received SIPI, vector 100
> Started cell "linux-1"
>
> My bzImage is 12,9MB and the rootfs.cpio is 23.1MB if that matters.
Try starting without the ramdisk. We can later take care of the
ramdisk,
but first the kernel needs to start.
For a compressed image, your kernel is really huge. I bet you can boil
it down to a few MBs. With a kernel of that size, you might want to
play
around with jailhouse-cell-linux -k.
Ralf
>
> Thanks,
> Michele
>
>
>
> Il giorno mercoledì 10 gennaio 2024 alle 20:58:03 UTC+1 Ralf
Ramsauer ha
> scritto:
>
> Hi Michele,
>
> On 10/01/2024 17:40, Michele Pescapè wrote:
> > Hi,
> >
> > I'm attaching the configurations I'm currently using. I'll try
> disabling
> > KASLR.
>
> […]
> /* high RAM */
> {
> .phys_start = 0x42100000,
> .virt_start = 0x00101000,
> .size = 0x8000000,
> […]
>
> Alignment of your high ram is odd. Virt & Phys should have the same
> alignment. Try to use .phys=.virt=0x42100000, I guess that should
help.
>
> Thanks,
> Ralf
>
>
> >
> > Thank you for your help,
> > Michele
> >
> > Il giorno mercoledì 10 gennaio 2024 alle 17:24:35 UTC+1 Ralf
> Ramsauer ha
> > scritto:
> >
> > Hi,
> >
> > On 09/01/2024 19:58, Michele Pescapè wrote:
> > > Hi,
> > >
> > > Just an update, I was able to get the following output by
> adding -c
> > > "earlyprintk=ttyS0,115200" to the command line:
> > >
> > > Started cell "linux-1"
> > >
> > >
> > > Invalid physical address chosen!
> >
> > could you please share your system and non-root cell
configuration?
> >
> > It smells like you have an overlap with the communication region.
> You
> > can check that with tools/jailhouse-config-check.
> >
> > >
> > >
> > >
> > >
> > > Physical KASLR disabled: no suitable memory region!
> >
> > Disable KASLR in your kernel for the moment. You don't need it
while
> > ramping up your system.
> >
> > Thanks,
> > Ralf
> >
> > >
> > > [ 0.000000] Linux version 6.2.0-rc3 (root@mp-LINUX-DESKTOP)
> > > (x86_64-buildroot-linux-gnu-gcc.br_real (Buildroot 2023.11)
> > 12.3.0, GNU
> > > ld (GNU Binutils) 2.40) #2 SMP PREEMPT_DYNAMIC Sun Jan 7
> > 18:35:23 CET 2024
> > > [ 0.000000] Command line: earlyprintk=ttyS0,115200
> > > [ 0.000000] KERNEL supported cpus:
> > > [ 0.000000] Intel GenuineIntel
> > > [ 0.000000] AMD AuthenticAMD
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87
> > floating
> > > point registers'
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE
> > registers'
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX
> > registers'
> > > [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]:
> > 256
> > > [ 0.000000] x86/fpu: Enabled xstate features 0x7, context
size
> > is 832
> > > bytes, using 'compacted' format.
> > > [ 0.000000] signal: max sigframe size: 1776
> > > [ 0.000000] BIOS-provided physical RAM map:
> > > [ 0.000000] BIOS-e801: [mem
> > 0x0000000000000000-0x000000000009efff] usable
> > > [ 0.000000] printk: bootconsole [earlyser0] enabled
> > > [ 0.000000] NX (Execute Disable) protection: active
> > > [ 0.000000] extended physical RAM map:
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x0000000000000000-0x0000000000001fff] usable
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x0000000000002000-0x000000000000212b] usable
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x000000000000212c-0x000000000009efff] usable
> > > [ 0.000000] DMI not present or invalid.
> > > [ 0.000000] Hypervisor detected: Jailhouse
> > > [ 0.000000] tsc: Detected 3393.624 MHz processor
> > > [ 0.000019] .text .data .bss are not marked as E820_TYPE_RAM!
> > > [ 0.005751] last_pfn = 0x3630 max_arch_pfn = 0x400000000
> > > [ 0.011039] x86/PAT: Configuration [0-7]: WB WC UC- UC WB
> > WP UC- WT
> > > Memory KASLR using RDRAND RDTSC...
> > > [ 0.032374] Using GB pages for direct mapping
> > > [ 0.036743] Kernel panic - not syncing: alloc_low_pages: can
> not
> > > alloc memory
> > > [ 0.043712] CPU: 0 PID: 0 Comm: swapper Not tainted
> 6.2.0-rc3 #2
> > > [ 0.049702] Call Trace:
> > > [ 0.052134] <TASK>
> > > [ 0.054219] dump_stack_lvl+0x3a/0x58
> > > [ 0.057865] dump_stack+0x10/0x18
> > > [ 0.061162] panic+0x15c/0x308
> > > [ 0.064205] alloc_low_pages+0x7d/0x1c0
> > > [ 0.068024] phys_pmd_init+0x1f2/0x2eb
> > > [ 0.071758] phys_pud_init+0x114/0x2d5
> > > [ 0.075493] phys_p4d_init+0x4c/0x269
> > > [ 0.079135] ?
> > __raw_callee_save___native_queued_spin_unlock+0x15/0x30
> > > [ 0.085649] __kernel_physical_mapping_init+0x11e/0x29a
> > > [ 0.090859] kernel_physical_mapping_init+0xf/0x20
> > > [ 0.095631] init_memory_mapping+0x273/0x410
> > > [ 0.099890] init_range_memory_mapping+0xec/0x150
> > > [ 0.104574] init_mem_mapping+0x268/0x2e7
> > > [ 0.108567] setup_arch+0x839/0xdcf
> > > [ 0.112039] ? lockdown_lsm_init+0x21/0x2f
> > > [ 0.116120] start_kernel+0x6e/0x997
> > > [ 0.119680] x86_64_start_reservations+0x24/0x2c
> > > [ 0.124281] x86_64_start_kernel+0xec/0xff
> > > [ 0.128360] secondary_startup_64_no_verify+0xe5/0xeb
> > > [ 0.133400] </TASK>
> > > [ 0.135570] ---[ end Kernel panic - not syncing:
> > alloc_low_pages: can
> > > not alloc memory ]---
> > > Closing cell "linux-1"
> > > Page pool usage after cell destruction: mem 254/32211, remap
> > 16392/131072
> > > CPU 4 received SIPI, vector 96
> > > CPU 5 received SIPI, vector 96
> > >
> > > This happens both with and without the initrd in the command,
> > i'll try
> > > to solve this and see what happens.
> > >
> > > By the way, is it possible that the new cell starts transmitting
> > with a
> > > baud rate different than the root cell? Without specifying the
> > speed in
> > > the command above the serial console stopped receiving anything
> > until I
> > > disabled and re-enabled the hypervisor. If not I may have
> something
> > > wrong on my end to solve
> > >
> > > Il giorno martedì 9 gennaio 2024 alle 17:37:48 UTC+1 Michele
> > Pescapè ha
> > > scritto:
> > >
> > > I stand corrected, the freeze happens when i do not set
> > > JAILHOUSE_CELL_PASSIVE_COMMREG in the inmate
> > >
> > > Il giorno martedì 9 gennaio 2024 alle 17:26:57 UTC+1 Michele
> Pescapè
> > > ha scritto:
> > >
> > > Hi, thank you for replying.
> > >
> > > ttyS0 is indeed whitelisted in the inmate and the number of
> > > pio_regions is set accordingly.
> > >
> > > As for the crash after disable/shutdown, I think it was related
> > > to the PCI device, after commenting that and the corresponding
> > > regions out I no longer get that freeze. CONFIG_JAILHOUSE_GUEST
> and
> > > CONFIG_JAILHOUSE_DBCON were already set in the kernel
> > > configuration I'm using for the non root linux; still, I
> > > recompiled it just in case.
> > >
> > > Michele
> > >
> > > Il giorno martedì 9 gennaio 2024 alle 14:14:33 UTC+1 Ralf
> > > Ramsauer ha scritto:
> > >
> > > Hi,
> > >
> > > On 08/01/2024 23:56, Michele Pescapè wrote:
> > > > Hi all,
> > > >
> > > > I have an AMD Ryzen 2600 on a b450 motherboard. My goal
> > > is to run two
> > > > non root linux cells, however, as of right now, I'm not
> > > able to start
> > > > any non root linux cells.
> > > > First of all I don't receive an output from the cell (I
> > > do get output
> > > > from the apic-demo though), therefore I don't really know
> > > if they're
> > > > even crashing or not; this is the output I get from the
> > > hypervisor after
> > > > enabling the root cell and issuing the following command:
> > > >
> > > > sudo jailhouse cell linux configs/x86/linux_guest1.cell
> > > > ../buildroot-2023.11/output/images/bzImage -i
> > > > ../buildroot-2023.11/output/images/rootfs.cpio
> > >
> > > for the first few tries, simply don't load a ramdisk. Try to
> > > get the
> > > kernel booting. It will crash with "cannot mount rootfs". If
> > > we get so
> > > far, then you can continue specifying the ramdisk. But
> > > first, we have to
> > > get that far.
> > >
> > > >
> > > > Adding PCI device ff:1f.7 to cell "linux-x86-demo"
> > > > Created cell "linux-x86-demo"
> > > > Page pool usage after cell creation: mem 336/32211, remap
> > > 16392/131072
> > > > Cell "linux-x86-demo" can be loaded
> > > > CPU 4 received SIPI, vector 100
> > > > CPU 3 received SIPI, vector 100
> > > > Started cell "linux-x86-demo"
> > > >
> > > > After this the cell appears to be running in cell list,
> > > however I can't
> > > > seem to get any output to confirm that (I even set
> > > > JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE, but nothing
> > > appears on the vrtual
> > > > console accessed via "jailhouse console -f"). I also
> > > tried adding -c
> > > > "console=ttyS0,115200".
> > >
> > > ttyS0, i.e., x86 Port 0x3f8 is whitelisted in the cell
> > > configuration?
> > > Actually, your cmdline is correct.
> > >
> > > > At this point if I try to disable/shutdown the cell, the
> > > system freezes
> > > > requiring a hard reset.
> > >
> > > Okay, is the guest Linux compiled with Jailhouse guest
> > > support? Please
> > > ensure that non-root Linux is compiled with
> > > CONFIG_JAILHOUSE_GUEST and
> > > CONFIG_JAILHOUSE_DBCON. The first one is mandatory for x86
> > > Linux guests.
> > >
> > > >
> > > > Also, there seems to be a mismatch between the PCI bdf
> > > value in the log
> > > > above and the one in the cell's configuration. In both
> > > root and non root
> > > > cells i set .bdf = (0x0f << 3) for the IVSHMEM net device
> > > and the Root
> > > > cell correctly reports that ("Adding virtual PCI device
> > > 00:0f.0 to cell
> > > > "RootCell"").
> > >
> > > First things first. Let's try to get Linux kicked off before
> > > diving into
> > > PCI. Comment it out for the moment, let's do that later.
> > >
> > > Ralf
> > >
> > > >
> > > > If I understood correctly, once the cell works properly I
> > > won't need to
> > > > set the console anymore as I can assign an ip (-c "ip
> > > x.x.x.x" in the
> > > > cell linux command) and ssh to the cell, right?
> > > >
> > > > I attach the root and non root cells' configurations in
> > > case they're
> > > > needed. I can provide both the root and non root cell's
> > > kernel .conf and
> > > > the buildroot configuration I used if needed.
> > > >
> > > > Thank you for your time,
> > > > Michele
> > > >
> > > > --
> > > > You received this message because you are subscribed to
> > > the Google
> > > > Groups "Jailhouse" group.
> > > > To unsubscribe from this group and stop receiving emails
> > > from it, send
> > > > an email to [email protected]
> > > > <mailto:[email protected]>.
> > > > To view this discussion on the web visit
> > > >
> > >
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com>>>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/824bacc4-b7ae-47c9-878e-7203214b4fc3n%40googlegroups.com?utm_medium=email&utm_source=footer>>>>>.
> > >
> > > --
> > > You received this message because you are subscribed to the
Google
> > > Groups "Jailhouse" group.
> > > To unsubscribe from this group and stop receiving emails from
it,
> > send
> > > an email to [email protected]
> > > <mailto:[email protected]>.
> > > To view this discussion on the web visit
> > >
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com>>>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/2202bf30-5b2c-45ea-9f9f-699daf338843n%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Jailhouse" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to [email protected]
> > <mailto:[email protected]>.
> > To view this discussion on the web visit
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/9a021e3f-caca-4ebe-9351-be2c3e999446n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/jailhouse-dev/23a9c30e-5507-449e-9589-697904becac4n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/23a9c30e-5507-449e-9589-697904becac4n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/23a9c30e-5507-449e-9589-697904becac4n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/23a9c30e-5507-449e-9589-697904becac4n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
--
You received this message because you are subscribed to the Google
Groups "Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/bbd32aa2-053e-40a2-b4d5-2326238d0793n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/bbd32aa2-053e-40a2-b4d5-2326238d0793n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/0b19f9a0-4f13-46d1-a851-3a0b57deeefb%40oth-regensburg.de.
