Ralf/Jan, Are you sure you remove the device from the kernel cmdline? *Yes, there is no ttyS* device in my root linux cmdline* - Are you sure that there doesn't spawn a tty? Check with systemctl. *No tty as far as I can tell, verified with "lsof | grep ttyS". The lsof command stops working after the crash though* - lsof | grep ttyS - The root-cell might receive the serial device's interrupt, while its ports are assigned to the non-root cell. Does the crash happen when you type in a character to the serial line? *This is possible, do you have an example of how I would accomplish assigning the interrupt (IRQ 4 for ttyS0) to the non-root? To rule out the interrupt concern.*
In this case, assign the interrupt to the non-root cell. You will need it in any case there once you have a initrd and want to type in stuff. So there must be some reason why the crash happens. The hard way is to add 8250.nr_uarts=0 in your kernel cmdline of the root cell. *I can use this method for now to continue booting the non-root linux.* > > if I leave out the pio mapping in the non-root linux, and > alolothat block in the root cell instead then the non-root linux gets a > PIO read fault on "0x3F9" while booting. > Ralf mentioned that Linux would not try enumerate the device if > its set to -1 ( [0x3f8/8 ... 0x3ff/8] = -1) , but the non-root seems to > attempt to do something with it anyway: > > "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled" > "FATAL: Invalid PIO read, port: 3f9: size 1" Hmm. At least that's what I thought. I don't know how enumeration of platform devices works exactly, but reconsidering this, the behaviour above makes sense: How would it know that a platform device is not present, depending of the configuration of the pio_bitmap. * Since I do technically have two serial ports, is there a way (via command line or kernel config for ex) to hardcode what the non-root would enumerate for the UART when its booting up? Meaning, force it to "see" the 0x2f8 serial port as ttyS0 instead of 0x3f8 when booting up and** CONFIG_SERIAL_8250_RUNTIME_UARTS = 1**. That way if you had say three serial ports and three cells (1 root, 2 non-root linux) then each could have their own separate dedicated ttyS0.* As far as just booting the non-root besides the serial concern, I haven't provided an initramfs to my non-root linux yet. I would like to probably boot it with a simple ramdisk if possible to get a point where I can log in. Thanks, Wayne On Wed, Jun 12, 2019 at 6:43 AM Ralf Ramsauer < [email protected]> wrote: > Hi, > > On 6/11/19 11:05 PM, Wayne wrote: > > Jan/Ralf, > > > > I was able to get past issues 1 and 3 above by using Jan's kernel config > > from this thread: > > > > > https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ > ". > > > > The DMA issue was resolved by disabling kernel option CONFIG_ISA_DMA_API > > as stated in another thread, and reflected in the config file above. > > > > So now I think i've almost got the non-root linux booting. I'm stuck at > > the point where its trying to mount a filesystem and since i'm not > > providing an initramfs it panics. > > Perfect! > > > > > Similar to the output below: > > > > [ 1.080178] VFS: Cannot open root device "(null)" or unknown-block(0,0): > > error -6 > > [ 1.087662] Please append a correct "root=" boot option; here are the > > available partitions: > > [ 1.096013] Kernel panic - not syncing: VFS: Unable to mount root fs on > > unknown-block(0,0) > > > > I have just a couple more questions: > > 1. In general, is it true that an inmate will takeover control from the > > root of any resource in the following arrays, or are there some cases > > where they get "shared" between the root and inmate? > > Depends. > > > A. ".mem_regions" -> Shared? > > May be shared with the root cell (i.e., if the region has the flag > JAILHOUSE_MEM_ROOTSHARED set). > > > B. ".irqchips" -> Shared? > > Shared with whom? Only one guest runs on a CPU. This guest will have > full access to the irqchip. Some accesses must be intercepted. > > > C. ".pio_bitmap" -> Inmate takes control of these if matching > > between inmate and root > > Exclusive control. > > > D. ".pci_devices" -> Inmate takes control? > > Exclusive control. > > > E. ".pci_caps" -> Inmate takes control? > > Exclusive control. > > > > > 2. Along the same lines, going back to my ttyS0 issue mentioned above > > while trying to share it: > > > > If I allocate the following to the linux non root cell (and set > > to -1 in the root) > > BTW, there's generally no need to set -1 in the root cell -- it will be > taken away when the non-root cell is created. > > > .pio_bitmap = { > > [0x3f8/8 ... 0x3ff/8] = 0x00 /* serial 2*/ > > } > > > > Then on the root cell I randomly get a PIO read fault on "0x3FE" > > when the non-root is booting, i'm not sure what process is causing this. > > - Are you sure you remove the device from the kernel cmdline? > - Are you sure that there doesn't spawn a tty? Check with systemctl. > - lsof | grep ttyS > - The root-cell might receive the serial device's interrupt, while its > ports are assigned to the non-root cell. Does the crash happen when > you type in a character to the serial line? > > In this case, assign the interrupt to the non-root cell. You will need > it in any case there once you have a initrd and want to type in stuff. > > So there must be some reason why the crash happens. The hard way is to > add 8250.nr_uarts=0 in your kernel cmdline of the root cell. > > > > > if I leave out the pio mapping in the non-root linux, and > > alolothat block in the root cell instead then the non-root linux gets a > > PIO read fault on "0x3F9" while booting. > > Ralf mentioned that Linux would not try enumerate the device if > > its set to -1 ( [0x3f8/8 ... 0x3ff/8] = -1) , but the non-root seems to > > attempt to do something with it anyway: > > > > "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled" > > "FATAL: Invalid PIO read, port: 3f9: size 1" > > Hmm. At least that's what I thought. I don't know how enumeration of > platform devices works exactly, but reconsidering this, the behaviour > above makes sense: How would it know that a platform device is not > present, depending of the configuration of the pio_bitmap. > > Anyway, 8250 drivers have some cmdline parameters that can be used to > control initialisation (e.g., see the nr_uarts parameters above). > > > > > Is it possible to map the same pio block in both the root and > > non-root with a different mask? Or does the non-root just override it? > > It seems like the latter is true. > > Exactly. That's not a) possible, and b) does not really make sense. > > Ralf > > > > > Thanks again Jan and Ralf for your help getting to this point. > > > > > > On Tue, Jun 11, 2019 at 2:13 PM Wayne <[email protected] > > <mailto:[email protected]>> wrote: > > > > Jan & Ralf: > > > > I have a little good news, I have successfully gotten the non-root > > linux to display some startup output based on your suggestions. > > Also, there must have been some small difference between the stock > > linux-x86-demo cell config and the one I tweaked. I went back to > > the original and made some minor changes and it started displaying > > the serial output now for the non-root node. > > > > I ended up making the root cell the primary user of the ttyS0 > > device, and just having the non-root linux output to the hypervisor > > virtual console (console=jailhouse). In my setup, there appears to > > be some process still attempting to use the ttyS0 randomly on the > > root linux, and that causes the hypervisor to halt it if I don't > > leave it in the PIO array. Not sure who is doing it. > > > > > > This is where i'm at now. The non-root linux keeps running into > > points where it tries to access PIO and gets parked: > > > > 1. First crash: > > > > [ 0.424925]kworker/u6:0 (27) used greated stack depth: 14656 > > bytes left > > [ 0.425807]futex has table entries: 1024 (order: 4, 65536 bytes) > > "FATAL: Invalid PIO write, port: 70: size 1" > > ... > > Parking CPU 3 (Cell: "linux-x86-demo") > > > > According to /proc/ioports, 70 has to do with the Realtime clock > > (rtc0). I was able to temporarily add the 70 mapping to the > > non-root linux, but should it be there at all? I tried to disable > > RTC support in the guest kernel config. > > > > > > 2. On the next attempt, I then got further with a crash trying to > > initialize Ser device ttyS0: > > > > "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing > enabled" > > "FATAL: Invalid PIO read, port: 3f9: size 1" > > ... > > Parking CPU 3 (Cell: "linux-x86-demo") > > > > I can get around that if I allow the non-root access in its pio > > table, but then I have a problem above with the root linux randomly > > trying to use it. > > > > > > 3. Now with the temporary adjustments to the non-root pio table > > above I get here: > > > > "[ 0.972399]clocksource:Switched to clocksource tsc" > > "FATAL: Invalid PIO read, port: 87: size 1" > > ... > > Parking CPU 3 (Cell: "linux-x86-demo") > > > > According to proc/ioports this one has to do with "dma page > > request". It isn't mapped in my root linux PIO or non-root linux > array. > > > > > > Aside from the serial conflict, it seems like these should remain > > controlled by the root linux or hypervisor. Do you use a > > particular bare minimum x86 kernel config on your machine for the > > guest to get around these? I know Jan mentioned the > > jailhouse-images project might have something, but I couldn't find > > it digging around quick. > > > > Thanks to you both for your responses and patience. I got the root > > linux up and going on my own, but the non-root guest is proving to > > be more difficult. > > > > Wayne > > > > > > > > On Tue, Jun 11, 2019 at 11:31 AM Ralf Ramsauer > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, > > > > On 6/10/19 7:45 PM, Wayne wrote: > > > Thank you for your responses Ralf and Jan. > > > > > > Yes, I have successfully gotten both the apic-demo and > > tiny-demo cells > > > to work on my system before attempting to load the non-root > > linux. I > > > can see serial output from them. > > > > > > However, if I try to share the UART between root and inmate > > for them I > > > run into a crash on the hypervisor if the root linux attempts > > to write > > > to the UART (after creating/starting a demo). Does the root > > linux lose > > > access once an inmate is created/started? Or am I missing > > something > > > from my System config or tiny-demo config that allows them to > > share? I > > > attached the crash in hypervisor_output.txt. > > > > > > Also, I noticed the tiny-demo output on the UART gets mirrored > > on the > > > > Yes, according to your config, > > JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE is > > set. This means that the guest will use the virtual console. The > > virtual > > console uses the hypervisor output which depends whatever is set > > in your > > system configuration. In your case the hypervisor uses 3f8. > > > > And additionally, the inmate also a valid .console set. Again, > > 3f8. This > > means, it will use the console *and* the hypervisor debug > > hypercall. In > > your case, both outputs are routed to the same device which > > explains the > > mirroring. > > > > > Jailhouse virtual console. Does that mean that its writing to > the > > > serial through the hypervisor and not directly accessing ttyS0 > > itself? > > > > Exactly. > > > > > If so, is there a way to create the cell to write ttyS0 > > directly as a > > > test? To see if it exhibits same behavior as the non-root > linux. > > > > Sure, simply align the .address field of .console, and allow > > access to > > that port via .pio_bitmap. > > > > > > > > FYI, I create the tiny-demo like this: > > > > > > jailhouse cell create configs/x86/tiny-demo.cell > > > jailhouse cell load tiny-demo inmates/demos/x86/tiny-demo.cell > > > > jailhouse cell start tiny-demo > > > > > > Going back to my original non-root linux question, I don't see > any > > > output when connected over ttyS0. Its address is 0x3f8 > > ("serial 1") in > > > the PIO, and that seems to work fine for the other simple > > demos, minus > > > the hypervisor crash issue above when sharing. I know there > > is the > > > other "serial 2" port but I haven't tried to use that one. > > > > I see that port on our Dell server as well. It's present, but I > > don't > > know where it's connected to. > > > > Let me just summarise your issue: > > > > You only have one serial line available, right? From the > root-cell's > > POV, it's ttyS0 on 3f8. > > > > I would propose to share ttyS0 between the hypervisor and > non-root > > Linux. This means: > > > > - Don't let the root-cell use ttyS0. Remove any console=ttyS0 > > parameters from you commandline. Ensure that noone else > accesses > > ttyS0 (e.g., getty@ttyS0 or other friends) > > > > - Set 3f8 as debug_console in your master.c (seems you already > > have) > > > > - Allow 3f8 access in the linux-demo.c (already set in the > default > > linux-demo > > > > - Disallow 2f8 access in linux-demo.c: > > - [ 0x2f8/8 ... 0x2ff/8] = 0, /* serial2 */ > > + [ 0x2f8/8 ... 0x2ff/8] = -1, /* serial2 */ > > > > I don't know how Linux enumerates serial devices, but this > > ensures > > that Linux won't see the unconnected serial line and map > > ttyS0 to > > 3f8 > > > > > > > > non root linux launched with: > > > /tools/jailhouse cell linux configs/x86/linux-x86.cell > > > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200" > > > > Then, this should actually work... > > > > And if not, then console=jailhouse0 should work. > > > > > > > > I can try the 4.19 siemens kernel and "next" branch for > > jailhouse you > > > suggested. Do you think that combination will resolve the > missing > > > serial for the non-root linux? Otherwise, did my kernel > > config, system > > > config, and non-root linux cell configs look OK? > > > > I didn't look at it, but we should at least - even if some flags > are > > still incorrect - see the "Uncompressing kernel..." thing. > > > > Wait -- one thing you could try: Deactivate CONFIG_EFI and > > especially > > CONFIG_EFI_STUBS. I'm not sure, but it could be that EFI_STUBS > > change > > the header format that our bootloader patches. > > > > > > > > Thanks again for your help. > > > > No problem. But it's not yet working. ;-) > > > > Ralf > > > > > > > > > > > On Fri, Jun 7, 2019 at 5:30 PM Ralf Ramsauer > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > > > Hi Wayne, > > > > > > On 6/7/19 10:04 PM, Wayne wrote: > > > > Hi Ralf, > > > > > > > > Thank you for your responses. I have attached the > > requested config > > > > files. I am now just trying to use the same bzImage for > > the root and > > > > the non-root linux node. > > > > > > One question that you didn't answer: do apic-demo or > > tiny-demo work? > > > They need to to have correct .console parameter set, so > > ensure that > > > parameters are set if you run those demos in your > > linux-demo cell. Just > > > compare it to tiny-demo.c or apic-demo.c > > > > > > > > > > > Yes, I am using vanilla 4.16 because the documentation > > indicated > > > that it > > > > is the first mainline release to officially support > > Jailhouse Guest > > > > images. Do you suggest that I use something else? > > > > > > As Jan already wrote: Try to use Siemens' 4.19 branch. > > > > > > > > > > > Right now i'm just trying to get some serial output from > > the non-root > > > > Linux. Hopefully, I have that configured correctly. > > Ideally, I would > > > > like to share the same serial console for the Root node > > and non-root > > > > node eventually if possible. I have been OK with using > > the virtual > > > > console (jailhouse console -f) for the Hypervisor output. > > > > > > In addition to Jan's comment: > > > > > > Currently you only have access to the jailhouse console > > via 'jailhouse > > > console' from your root cell. Imagine your root cell > > crashes for some > > > reason. You will never see the fault reason, which makes > > things hard to > > > debug. > > > > > > I would suggest to configure the hypervisor to use the > > serial console. > > > You can share the device with the non-root Linux, that's > > no problem. > > > > > > In this case, the non-root kernel's output will always be > > printed to the > > > serial console. Directly, if you choose console=ttySx, and > > indirectly > > > via the hypervisor if you choose console=jailhouse. > > > > > > BTW: According to your config, your system is a PowerEdge, > > and the > > > non-root cell gets both, 0x2f8 and 0x3f8. Are you sure > > that ttyS1 is the > > > correct console that you are connected to? > > > > > > Just mentioning this as we have a similar machine here, > > and, afair, both > > > platform serial device are 'usable', but one ends in the > > nirvana while > > > it is accessible. > > > > > > Ah, and one last thing: try to switch to the current next > > branch for > > > jailhouse. Jan just integrated a few patches from me that > > might also be > > > relevant for your machine. > > > > > > HTH > > > Ralf > > > > > > > > > > > Wayne > > > > > > > > On Fri, Jun 7, 2019 at 9:10 AM Ralf Ramsauer > > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>> wrote: > > > > > > > > Hi, > > > > > > > > On 6/7/19 2:28 PM, Wayne wrote: > > > > > Hello, > > > > > > > > > > I am new to Linux development and Jailhouse. I > have > > > successfully > > > > booted > > > > > the Jailhouse Hypervisor and root cell on a bare > > metal X86 Linux > > > > system > > > > > (No QEMU). I am now trying to load a non-root > > Linux cell > > > and I have a > > > > > few questions. Jailhouse accepts and starts my > > non-root > > > linux cell > > > > > configuration and I see it as "running" through the > > > "jailhouse cell > > > > > list" command. However, I don't see any serial > > output from the > > > > > "non-root linux" cell booting up. I’m not sure > > what the > > > non-root node > > > > > is doing at this point. > > > > > > > > Ok, first of all, can you see any output from > > Jailhouse's demo > > > inmates > > > > (e.g., tiny-demo or apic-demo)? > > > > > > > > > > > > > > My root node is a 4.16 kernel > > configured this > > > way: > > > > > 1. > > CONFIG_JAILHOUSE_GUEST is > > > not set > > > > > 2. CONFIG_SERIO=y > > > > > 3. > > > CONFIG_SERIAL_8250_RUNTIME_UARTS=4 > > > > > > > > > > My non-root node is a 4.16 kernel > > configured > > > this way: > > > > > 1. > > CONFIG_JAILHOUSE_GUEST=y > > > > > 2. CONFIG_SERIO=m > > (can't seem > > > to disable > > > > > completely in my config for 4.16) > > > > > 3. > > > CONFIG_SERIAL_8250_RUNTIME_UARTS=1 > > > > > > > > Could you please provide a full .config? What branch > > are you > > > exactly > > > > using? Vanilla 4.16? > > > > > > > > Please attach your system config and your non-root > linux > > > config as well. > > > > > > > > You may also want to try > > https://github.com/siemens/linux . See > > > > jailhouse-enabling branches for some releases. > > > > > > > > > > > > > > In general, do the kernel config settings have to > > match between > > > > the root > > > > > node and non-linux or is the above fine? > > > > > > > > No, they do not have to be the same, but they can. > > Still, an > > > analysis > > > > requires your .config. > > > > > > > > > > > > > > The vmlinux-4.1.16-Guest bzImage is approx 7MB, > > and the > > > inmate node is > > > > > allocated ~75MB of RAM. > > > > > > > > > > I have a single UART, so I have configured the > > root cell > > > system config > > > > > to output to the virtual hypervisor console: > > > > > > > > Wait... You configured your root-cell to use the > > hypervisor debug > > > > console? How? It's only available once the > hypervisor is > > > enabled. How > > > > should this work? > > > > > > > > But let's have a look at your configuration first. > > > > > > > > So you want to: > > > > - Have the UART (0x3f8) available in the non-root > cell > > > > - Use 0x3f8 as hypervisor debug console as well > > > > - No console for root-cell > > > > > > > > Did I get this right? > > > > > > > > Ralf > > > > > > > > > > > > > > .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE, > > > > > > > > > > .debug_console = { > > > > > .type = JAILHOUSE_CON_TYPE_NONE, > > > > > }, > > > > > > > > > > and I have configured the non-root linux cell to > > output to > > > the UART: > > > > > > > > > > (Added serial 0x3f8 to pio bitmap for non-root > > linux) and > > > started the > > > > > node with this: > > > > > > > > > > ./tools/jailhouse cell linux > > configs/x86/linux-x86.cell > > > > > /boot/vmlinux-4.1.16-Guest -c > "console=ttyS0,115200" > > > > > (Note I also tried "console=jailhouse" in the > > command above, > > > and that > > > > > produces the same result) > > > > > > > > > > I then see the following on my hypervisor console > > > (./tools/jailhouse > > > > > console -f): > > > > > > > > > > Created cell "linux-x86-demo" > > > > > ... > > > > > Cell "linux-x86-demo" can be loaded > > > > > Started cell "linux-x86-demo" > > > > > > > > > > After a little while I do get a parked CPU error > > on the root > > > node, > > > > looks > > > > > like its trying to do something with the UART as > well: > > > > > FATAL: Invalid PIO read, port: 3fe size: 1 > > > > > > > > > > I would expect something to pop out on the UART > > from the > > > non-root > > > > linux > > > > > node first. Note that root node has serial 0x3f8 > > disabled > > > in its pio > > > > > bitmap. > > > > > > > > > > I verifed that the UART is functioning by allowing > the > > > hypervisor to > > > > > print to it and also performed an echo test over > > ttyS0. > > > > > > > > > > I have tried several configurations of > > kernel.....including your > > > > current > > > > > "queues/jailhouse" branch head kernel for the > > non-root node, > > > along > > > > with > > > > > the kernel config for 4.7 posted in this thread > > below (but I > > > get same > > > > > result as above when I start it, no kernel output): > > > > > > > > > > > > > > > > > > > " > https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ > ". > > > > > > > > > > Any information you can provide to me will be > > helpful. I'm > > > not sure > > > > > what might be going wrong here. > > > > > > > > > > Thanks, > > > > > Wayne > > > > > > > > > > -- > > > > > You received this message because you are > > subscribed to the > > > Google > > > > > Groups "Jailhouse" group. > > > > > To unsubscribe from this group and stop receiving > > emails > > > from it, send > > > > > an email to > > [email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>> > > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>>> > > > > > <mailto:[email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>> > > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>>>>. > > > > > To view this discussion on the web visit > > > > > > > > > > > > > > > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com > > > > > > > > > > > > > > < > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com?utm_medium=email&utm_source=footer > >. > > > > > For more options, visit > > https://groups.google.com/d/optout. > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Jailhouse" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] > > <mailto:[email protected]>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0O2zSeuLLY3MaeRW7cQrWbq-6Y2BHJg%2Bx_j6nk%3DECa_A%40mail.gmail.com > > < > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0O2zSeuLLY3MaeRW7cQrWbq-6Y2BHJg%2Bx_j6nk%3DECa_A%40mail.gmail.com?utm_medium=email&utm_source=footer > >. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc36sP7ABEsR5Bp%2Bgpts0DUBENqV6eFDPazfs5kR_QRGaw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
