Thanks,
Michele
Il giorno mercoledì 17 gennaio 2024 alle 11:52:18 UTC+1 Ralf Ramsauer ha
scritto:
Hi Michele,
On 16/01/2024 20:44, Michele Pescapè wrote:
> Hi,
>
> I'm trying to give network access to the non root cell, however
it seems
> that I'm not able to pass the right PCI device. It always
defaults to
> taking away the 00:00.0 pci device from the root cell causing a
crash
> for the root cell.
>
> In both the root and non root cell's configs I added the shared
memory
> regions using the macro JAILHOUSE_SHMEM_NET_REGIONS(0x60000000,
0) for
> the root and using dev_id=1 for the non root. The resulting
regions do
> not overlap other memory regions, config check does not return
any errors.
>
> I also added the following PCI device in both cells, where
> .shmem_regions_start is assigned to the correct index for the first
> shared memory region created using the aforementioned macro.
>
> /* IVSHMEM: 00:05.0 */
> {
> .type = JAILHOUSE_PCI_TYPE_IVSHMEM,
> .domain = 0x0,
> .bdf = 0x28,
[…]
> .bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_MSIX,
> .num_msix_vectors = 2,
> .shmem_regions_start = 0,
> .shmem_dev_id = 0, (1 in the non root cell)
> .shmem_peers = 2,
> .shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
> },
>
> I tried using different values for the .bdf, none of which were
already
> used by other devices but none worked. The root cell always sees
the new
> device correctly when starting the hypervisor, I can also see it
with
> lspci and ifconfig, however once I issue the cell linux command
to start
> the non root cell, I get the following output followed by a
streak of
> errors from the hypervisor, forcing a manual reset.
>
> Removing PCI device 00:00.0 from cell "RootCell"
> Adding PCI device 00:00.0 to cell "linux-1"
.bdf = 0x28 doesn't correspond with 00:00.0. Would you please test your
current configs against jailhouse-config-check, and in case of no
errors, would you please attach the current versions again?
Thanks,
Ralf
> FATAL: unable to get MMIO instruction
> FATAL: Invalid MMIO/RAM write, addr: 0x0000000106d8c020 size: 0
> RIP: 0xffffffffa910ff4c RSP: 0xffffab85400f7de0 FLAGS: 93
> RAX: 0x0000000000000000 RBX: 0x000000000002f170 RCX:
0x0000000000000040
> RDX: 0x0000000000000414 RSI: 0xffff8b6000955400 RDI:
0x0000000000000001
> CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
> CR0: 0x0000000080050033 CR3: 0x0000000106d8c000 CR4:
0x00000000003506a0
> EFER: 0x0000000000001d01
> Parking CPU 4 (Cell: "RootCell")
> (etc..)
>
> I'm not sure what I'm missing here, CONFIG_IVSHMEM_NET is enabled in
> both the root and non root kernel config, I also enabled
> CONFIG_PCI_HOST_GENERIC in the root linux as suggested in one of the
> conversations on this site.
>
> Thanks,
> Michele
>
>
> Il giorno domenica 14 gennaio 2024 alle 15:49:40 UTC+1 Ralf
Ramsauer ha
> scritto:
>
> Hi Michele,
>
> On 14/01/2024 02:18, Michele Pescapè wrote:
> > Hi,
> >
> > There were PIO writes to ports 4e,4f,2e and 2f, each of size 1
> and each
> > one on subsequent restarts. I'm not sure how to figure out what
they
> > belong to.
>
> Easy and straight forward: Read the jailhouse crash dump.
>
> Look at the program counter, and maybe the return address.
Disassemble
> your kernel (objdump -d vmlinux) and lookup that address. Now you
know
> the name of the routine in the kernel that caused the crash.
>
> Don't simply whitelist ports. Only whitelist them, if you're sure
what
> you're doing.
>
> >
> > CONFIG_ISA_DMA_API is already disabled.
> >
> > I missed the irqchip for the uart. However as of right now I just
> copied
> > the whole fragment from the root configuration which means I
> could be
> > taking away other interrupts from the root cell, I still have to
> figure
> > out how to tune that
> >
> > /* IOAPIC 13, GSI base 0 */
> > {
> > .address = 0xfec00000,
> > .id = 0xa0,
> > .pin_bitmap = {
> > 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
>
> With this, you take away ALL interrupts from the root cell, which
you
> don't want to do, you only want to take away the interrupt that
> corresponds to ttyS0 (i.e. Port 0x3f8).
>
> See here: https://en.wikipedia.org/wiki/COM_(hardware_interface)
<https://en.wikipedia.org/wiki/COM_(hardware_interface)>
> <https://en.wikipedia.org/wiki/COM_(hardware_interface)
<https://en.wikipedia.org/wiki/COM_(hardware_interface)>>
>
> COM1: I/O port 0x3F8, IRQ 4
>
> So you want to only assign IRQ 4 to your non-root cell. See
> linux-x86-demo.c to see how that works.
>
> > },
> > },
> >
> > Still, now I can finally login into the non root linux.
>
> Excellent!
>
> Ralf
>
> >
> > Thanks,
> > Michele
> >
> > Il giorno sabato 13 gennaio 2024 alle 21:09:28 UTC+1 Ralf
> Ramsauer ha
> > scritto:
> >
> > Hi Michele,
> >
> > On 13/01/2024 18:08, Michele Pescapè wrote:
> > > Hi,
> > >
> > > The problem was that the ram regions in the non root
configuration
> > > weren't detected as such because of the
missing JAILHOUSE_MEM_DMA
> > flag.
> >
> > Yikes, makes sense.
> >
> > > After adding that, I also had to add two pio_regions because
> the non
> > > root linux was crashing because of two ports which aren't
> present in
> > > /proc/ioports.
> >
> > Uhm - which PIO ports? Don't simply assign PIO ports. There must
> be a
> > reason for them. What ports did crash?
> >
> > Did you disable(!) CONFIG_ISA_DMA_API? Please disable it.
Therefore,
> > you
> > have to activate CONFIG_EXPERT first.
> >
> > I *bet* it was i8237…
> >
> > > I also had to add mce=off to the command line because i had an
> > unhandled
> >
> > or disable CONFIG_X86_MCE.
> >
> > > MSR error, I'll have to disable that in the kernel config.
> > > At this point the non root linux seems to start, I see the boot
> > log. No
> >
> > Excellent!
> >
> > > way of interacting with it as there is not a login prompt or
> > anything, I
> > > think I need to ssh to the cell at this point, right?
> >
> > Was the initramdisk loaded correctly?
> >
> > Did you assign - in your non-root cell config - the irqchip and
the
> > corresponding interrupts of the uart?
> >
> > > That means I'll now need to work on those ivshmem net devices.
> >
> > If you need ivshmem, then this would be the next step.
> >
> > Ralf
> >
> > >
> > > Thanks,
> > > Michele
> > >
> > > Il giorno sabato 13 gennaio 2024 alle 15:13:12 UTC+1 Ralf
> > Ramsauer ha
> > > scritto:
> > >
> > > Hi,
> > >
> > > On 13/01/2024 12:28, Michele Pescapè wrote:
> > > > Hi,
> > > >
> > > > You are right, I got confused about those addresses, my
bad. At
> > > least
> > > > now I know that config is the correct one and I don't have to
> > tinker
> > > > with it.
> > > > I'm back to a kernel panic from the inmate when booting the
> cell.
> > > It's
> > > > similar to the one I had earlier, not sure yet of what the
> > > problem may be.
> > >
> > > Great, we're a step further.
> > >
> > > >
> > > > Created cell "linux-2"
> > > > Page pool usage after cell creation: mem 406/32211, remap
> > > 16392/131072
> > > > Cell "linux-2" can be loaded
> > > > CPU 9 received SIPI, vector 100
> > > > Started cell "linux-2"
> > > > CPU 8 received SIPI, vector 100
> > > > No EFI environment detected.
> > > > early console in extract_kernel
> > > > input_data: 0x000000000275c308
> > > > input_len: 0x00000000008b0981
> > > > output: 0x0000000001000000
> > > > output_len: 0x0000000001fccb30
> > > > kernel_total_size: 0x0000000001e28000
> > > > needed_size: 0x0000000002000000
> > > > trampoline_32bit: 0x000000000009d000
> > > >
> > > > Decompressing Linux... Parsing ELF... done.
> > > > Booting the kernel.
> > > > [ 0.000000] Linux version 6.2.0-rc3 (root@mp-LINUX-DESKTOP)
> > > > (x86_64-buildroot -linux-gnu-gcc.br_real (Buildroot
> 2023.11)
> > > > 12.3.0, GNU ld (GNU Binutils) 2.40) # 2 SMP
> PREEMPT_DYNAMIC
> > > Fri
> > > > Jan 12 17:36:57 CET 2024
> > > > [ 0.000000] Command line: earlyprintk=ttyS0,115200
> > > > [ 0.000000] KERNEL supported cpus:
> > > > [ 0.000000] Intel GenuineIntel
> > > > [ 0.000000] AMD AuthenticAMD
> > > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87
> > > floating
> > > > point regi sters'
> > > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE
> > > registers'
> > > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX
> > > registers'
> > > > [ 0.000000] x86/fpu: xstate_offset[2]: 576,
xstate_sizes[2]:
> > > 256
> > > > [ 0.000000] x86/fpu: Enabled xstate features 0x7, context
> size
> > > is 832
> > > > bytes, using 'compacted' format.
> > > > [ 0.000000] signal: max sigframe size: 1360
> > > > [ 0.000000] BIOS-provided physical RAM map:
> > > > [ 0.000000] BIOS-e801: [mem
> > > 0x0000000000000000-0x000000000009efff] usable
> > >
> > > Okay, here should be all memory regions listed. My non-root
Linux
> > in my
> > > Qemu VM, for example, shows here:
> > >
> > > [ 0.000000] BIOS-provided physical RAM map:
> > > [ 0.000000] BIOS-e820: [mem
0x0000000000000000-0x00000000000fffff]
> > > usable
> > > [ 0.000000] BIOS-e820: [mem
0x0000000000100000-0x0000000000100fff]
> > > reserved
> > > [ 0.000000] BIOS-e820: [mem
0x0000000000200000-0x00000000048fffff]
> > > usable
> > >
> > > Are you absolutely sure, that you have no further
modifications in
> > > Jailhouse or the Linux loader?
> > >
> > > Actually, in your case, there should be e820 as well (instead of
> > e801).
> > > Go to the Linux kernel sources, have a look at
> > > arch/x86/kernel/e820.c:1279
> > >
> > > and Jailhouse's jailhouse-cell-linux:638.
> > >
> > > jailhouse-cell-linux fills information of all memory regions
into
> > the
> > > zero page. Would you please instrument code (Linux/Jailhouse)
> to see
> > > where those regions are not parsed?
> > >
> > > > [ 0.000000] printk: bootconsole [earlyser0] enabled
> > > > [ 0.000000] NX (Execute Disable) protection: active
> > > > [ 0.000000] extended physical RAM map:
> > > > [ 0.000000] reserve setup_data: [mem
> > > > 0x0000000000000000-0x0000000000001fff] u sable
> > > > [ 0.000000] reserve setup_data: [mem
> > > > 0x0000000000002000-0x000000000000212b] u sable
> > > > [ 0.000000] reserve setup_data: [mem
> > > > 0x000000000000212c-0x000000000009efff] u sable
> > > > [ 0.000000] DMI not present or invalid.
> > > > [ 0.000000] Hypervisor detected: Jailhouse
> > >
> > > Just guessing loud: Hmm, you have guest support enabled, that's
> > not the
> > > issue.
> > >
> > > > [ 0.000000] tsc: Detected 3393.624 MHz processor
> > > > [ 0.000017] .text .data .bss are not marked as
E820_TYPE_RAM!
> > >
> > > Here's the next error that makes me curious, why you system
falls
> > back
> > > to E801...
> > >
> > > > [ 0.005745] last_pfn = 0x2e28 max_arch_pfn = 0x400000000
> > > > [ 0.011025] x86/PAT: PAT support disabled because
> > > CONFIG_X86_PAT is
> > > > disabled in the kernel.
> > >
> > > Please enable CONFIG_X86_PAT and MTRR in your kernel.
> > >
> > > > [ 0.019362] x86/PAT: Configuration [0-7]: WB WT UC- UC
WB
> > > WT UC- UC
> > > > [ 0.034867] Using GB pages for direct mapping
> > > > [ 0.039193] Kernel panic - not syncing: alloc_low_pages:
can
> > not
> > > > alloc memory
> > >
> > > Yeah, that's the logical aftereffect after the errors above.
> > >
> > > Thanks,
> > > Ralf
> > >
> > > > [ 0.046183] CPU: 0 PID: 0 Comm: swapper Not tainted
> > 6.2.0-rc3 #2
> > > > [ 0.052176] Call Trace:
> > > > [ 0.054606] <TASK>
> > > > [ 0.056691] ? dump_stack_lvl+0x33/0x4e
> > > > [ 0.060510] ? panic+0x157/0x303
> > > > [ 0.063723] ? sprintf+0x56/0x80
> > > > [ 0.066936] ? alloc_low_pages+0x70/0x1a0
> > > > [ 0.070930] ? phys_pmd_init+0x1fc/0x2eb
> > > > [ 0.074839] ? phys_pud_init+0x116/0x2d3
> > > > [ 0.078744] ? __kernel_physical_mapping_init+0x11a/0x290
> > > > [ 0.084128] ? init_memory_mapping+0x25e/0x3b0
> > > > [ 0.088558] ? init_range_memory_mapping+0xe7/0x145
> > > > [ 0.093417] ? init_mem_mapping+0x242/0x298
> > > > [ 0.097585] ? setup_arch+0x74e/0xcbd
> > > > [ 0.101231] ? start_kernel+0x66/0x8b7
> > > > [ 0.104965] ? load_ucode_bsp+0x43/0x11b
> > > > [ 0.108873] ? secondary_startup_64_no_verify+0xe0/0xeb
> > > > [ 0.114085] </TASK>
> > > > [ 0.116255] ---[ end Kernel panic - not syncing:
> > > alloc_low_pages: can
> > > > not all oc memory ]---
> > > >
> > > >
> > > > Thank you for your continuous support,
> > > > Michele
> > > >
> > > > Il giorno sabato 13 gennaio 2024 alle 00:05:43 UTC+1 Ralf
> > > Ramsauer ha
> > > > scritto:
> > > >
> > > > Hi Michele,
> > > >
> > > > On 12/01/2024 14:07, Michele Pescapè wrote:
> > > > > jailhouse cell load linux-2 linux-loader.bin -a 0x0
> > > > > ../buildroot-2023.11/output/images/bzImage -a 0xffbe00
> > parameters
> > > > -a 0x1000
> > > > > jailhouse cell start linux-2
> > > > >
> > > > > I take it the kernel is loaded at 0xffbe00 which is right
> at the
> > > > edge of
> > > > > the low ram region. In fact after issuing the cell load
> command
> > > and
> > > > > adjusting the path for the loader I get JAILHOUSE_CELL_LOAD:
> > > Invalid
> > > > > argument.
> > > >
> > > > Just tested cell-linux in a qemu machine, there it works, with
> > > pretty
> > > > similar addresses, which got me confused.
> > > >
> > > > After double-checking it: 0xffb.e00 is *not* at the edge of
low
> > mem:
> > > >
> > > > Low mem is 0x000.000 -- 0x0ff.fff
> > > > Comm region is 0x100.000 -- 0x100.fff
> > > >
> > > > 0xffb.e00 is (obviously) way above.
> > > >
> > > > Please try to set your high mem's .virt_start to 0x200000.
Then,
> > > > 0xffbe00 is offsetted ~16MB inside your highmem, and it should
> > work!
> > > >
> > > > IOW:
> > > >
> > > > /* high RAM */
> > > > {
> > > > .phys_start = 0x42300000,
> > > > .virt_start = 0x200000,
> > > > .size = 0x11000000,
> > > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > > > JAILHOUSE_MEM_EXECUTE |
> > > > JAILHOUSE_MEM_LOADABLE,
> > > > },
> > > >
> > > > Thanks
> > > > Ralf
> > > >
> > > > --
> > > > You received this message because you are subscribed to the
> Google
> > > > Groups "Jailhouse" group.
> > > > To unsubscribe from this group and stop receiving emails from
> it,
> > > send
> > > > an email to [email protected]
> > > > <mailto:[email protected]>.
> > > > To view this discussion on the web visit
> > > >
> > >
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>>>>.
> > >
> > > --
> > > You received this message because you are subscribed to the
Google
> > > Groups "Jailhouse" group.
> > > To unsubscribe from this group and stop receiving emails from
it,
> > send
> > > an email to [email protected]
> > > <mailto:[email protected]>.
> > > To view this discussion on the web visit
> > >
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>>>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Jailhouse" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to [email protected]
> > <mailto:[email protected]>.
> > To view this discussion on the web visit
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
--
You received this message because you are subscribed to the Google
Groups "Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/0f1154fe-f680-47e6-a3da-9e1a37fd1f55n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/0f1154fe-f680-47e6-a3da-9e1a37fd1f55n%40googlegroups.com?utm_medium=email&utm_source=footer>.
