Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Hello everybody, I'm also facing this problem for several months, trying to solve it individually. In my case the situation is yet more complicated, as I'm using quite complex setup, connecting VM's partitions from another server via AoE. According to console logs, the problems seems to be related to networking, as the first "hang-up" messages are usually something like igb :01:00.0 eth2: Reset adapter igb :01:00.0 eth2: Reset adapter ... followed by sequence of: ata3.00: exception Emask 0x0 SAct 0x30 SErr 0x0 action 0x6 frozen ata3.00: failed command: WRITE FPDMA QUEUED ata3.00: cmd 61/01:a0:08:f0:08/00:00:00:00:00/40 tag 20 ncq dma 512 out res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout) ata3.00: status: { DRDY } ata3.00: failed command: READ FPDMA QUEUED ata3.00: cmd 60/08:a8:20:f8:55/00:00:11:00:00/40 tag 21 ncq dma 4096 in res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout) ... and then other I/O errors, like blk_update_request: I/O error, dev sdb, sector 254266864 and others. IMHO something (the networking?) breaks the I/O in kernel. Nevertheless, the system somehow keeps running, being not able to read or write to disks, but a kernel panic is not announced! I've tried all suggested actions - trying newer kernels, building custom kernel with disabled IPv6 & QoS and other potentially problematic items, disabling all suspected features in BIOS, tuning network interfaces parameters, even disabling them (and using others from another vendor)... With no difference. However I've discovered by lucky chance, that the problem is somehow related to CPU family - when using the Intel E55xx (tested at least for E5520 and E5504), the problem occurs quite frequently (~ once per day), while using E56xx (tested at least for E5606), the problem becomes very rare or disappears at all (I'm using it longer than 1 month without a problem now). I hope this information will help to solve this problem, or at least it will give viable option for other afflicted (and desparate) users. Regards, Jan.
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Thursday, December 1, 2016, 2:59:36 PM, you wrote: > On 01.12.2016 14:26, Wei Liu wrote: >> This is still the same kernel log that was sent some time ago. >> So, if you have built Xen with debug=y, could you try to set Xen log >> level to the highest and capture "xl dmesg" when guest crashes? > It's not the guest that crashes, it's dom0. So when the host crashes, > I'm not able to issue any commands anymore. >> But I think this is increasingly likely to be a Linux kernel issue >> because you've tried multiple versions of xen. Maybe it is time to try >> different versions of Dom0 kernels (sorry if you've tried that, I can't >> remember all the details over so many moons). > Yes, indeed I have tried different kernels, but I can't remember details > as well... ;/ Hi Ingo, Have you tried without enabling "ndisc" (QoS) and "ipv6" ? They are both present in your log and i assume you are using a bridged network config ? You wouldn't be the first to stumble over a more generic kernel network bug while using Xen, due to less well tested combinations. So it's worth testing if plain ipv4 and no QoS works. -- Sander
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On 01.12.2016 14:26, Wei Liu wrote: This is still the same kernel log that was sent some time ago. So, if you have built Xen with debug=y, could you try to set Xen log level to the highest and capture "xl dmesg" when guest crashes? It's not the guest that crashes, it's dom0. So when the host crashes, I'm not able to issue any commands anymore. But I think this is increasingly likely to be a Linux kernel issue because you've tried multiple versions of xen. Maybe it is time to try different versions of Dom0 kernels (sorry if you've tried that, I can't remember all the details over so many moons). Yes, indeed I have tried different kernels, but I can't remember details as well... ;/ -- Ciao... //http://blog.windfluechter.net Ingo \X/ XMPP: i...@jabber.windfluechter.net gpg pubkey: http://www.juergensmann.de/ij_public_key.asc
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Thu, Dec 01, 2016 at 02:59:36PM +0100, Ingo Jürgensmann wrote: > On 01.12.2016 14:26, Wei Liu wrote: > > >This is still the same kernel log that was sent some time ago. > >So, if you have built Xen with debug=y, could you try to set Xen log > >level to the highest and capture "xl dmesg" when guest crashes? > > It's not the guest that crashes, it's dom0. So when the host crashes, I'm > not able to issue any commands anymore. > Oh, sorry for speaking non-sense. In that case you will need to setup a serial console to capture output on the fly. Wei.
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Tue, Nov 29, 2016 at 09:20:55PM +0100, Ingo Jürgensmann wrote: > Am 29.11.2016 um 10:08 schrieb Wei Liu : > >> http://paste.debian.net/895464/ > > Entry not found -- maybe it expired... Sorry. > > Here it is: > This is still the same kernel log that was sent some time ago. So, if you have built Xen with debug=y, could you try to set Xen log level to the highest and capture "xl dmesg" when guest crashes? But I think this is increasingly likely to be a Linux kernel issue because you've tried multiple versions of xen. Maybe it is time to try different versions of Dom0 kernels (sorry if you've tried that, I can't remember all the details over so many moons). Wei.
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Am 29.11.2016 um 10:08 schrieb Wei Liu : >> http://paste.debian.net/895464/ > Entry not found -- maybe it expired... Sorry. Here it is: Nov 14 09:19:52 31.172.31.251 [39677.027813] BUG: unable to handle kernel Nov 14 09:19:52 31.172.31.251 at 880002b4c06e Nov 14 09:19:52 31.172.31.251 [39677.027868] IP: Nov 14 09:19:52 31.172.31.251 [] ndisc_send_redirect+0x44c/0x4d0 Nov 14 09:19:52 31.172.31.251 [39677.027902] PGD 1a07067 Nov 14 09:19:52 31.172.31.251 Nov 14 09:19:52 31.172.31.251 [39677.027934] Oops: [#1] Nov 14 09:19:52 31.172.31.251 Nov 14 09:19:52 31.172.31.251 [39677.027956] Modules linked in: Nov 14 09:19:52 31.172.31.251 authenc(E) Nov 14 09:19:52 31.172.31.251 echainiv(E) Nov 14 09:19:52 31.172.31.251 xfrm6_mode_tunnel(E) Nov 14 09:19:52 31.172.31.251 xfrm4_mode_tunnel(E) Nov 14 09:19:52 31.172.31.251 xt_physdev(E) Nov 14 09:19:52 31.172.31.251 br_netfilter(E) Nov 14 09:19:52 31.172.31.251 xen_netback(E) Nov 14 09:19:52 31.172.31.251 tun(E) Nov 14 09:19:52 31.172.31.251 xen_blkback(E) Nov 14 09:19:52 31.172.31.251 xt_multiport(E) Nov 14 09:19:52 31.172.31.251 twofish_generic(E) Nov 14 09:19:52 31.172.31.251 twofish_avx_x86_64(E) Nov 14 09:19:52 31.172.31.251 twofish_x86_64_3way(E) Nov 14 09:19:52 31.172.31.251 twofish_x86_64(E) Nov 14 09:19:52 31.172.31.251 twofish_common(E) Nov 14 09:19:52 31.172.31.251 serpent_avx_x86_64(E) Nov 14 09:19:52 31.172.31.251 serpent_sse2_x86_64(E) Nov 14 09:19:52 31.172.31.251 serpent_generic(E) Nov 14 09:19:52 31.172.31.251 blowfish_generic(E) Nov 14 09:19:52 31.172.31.251 blowfish_x86_64(E) Nov 14 09:19:52 31.172.31.251 blowfish_common(E) Nov 14 09:19:52 31.172.31.251 cast5_avx_x86_64(E) Nov 14 09:19:52 31.172.31.251 cast5_generic(E) Nov 14 09:19:52 31.172.31.251 cast_common(E) Nov 14 09:19:52 31.172.31.251 ctr(E) Nov 14 09:19:52 31.172.31.251 des_generic(E) Nov 14 09:19:52 31.172.31.251 cbc(E) Nov 14 09:19:52 31.172.31.251 algif_skcipher(E) Nov 14 09:19:52 31.172.31.251 camellia_generic(E) Nov 14 09:19:52 31.172.31.251 camellia_aesni_avx_x86_64(E) Nov 14 09:19:52 31.172.31.251 camellia_x86_64(E) Nov 14 09:19:52 31.172.31.251 xts(E) Nov 14 09:19:52 31.172.31.251 xcbc(E) Nov 14 09:19:52 31.172.31.251 sha512_ssse3(E) Nov 14 09:19:52 31.172.31.251 sha512_generic(E) Nov 14 09:19:52 31.172.31.251 md4(E) Nov 14 09:19:52 31.172.31.251 algif_hash(E) Nov 14 09:19:52 31.172.31.251 af_alg(E) Nov 14 09:19:52 31.172.31.251 xfrm_user(E) Nov 14 09:19:52 31.172.31.251 xfrm4_tunnel(E) Nov 14 09:19:52 31.172.31.251 tunnel4(E) Nov 14 09:19:52 31.172.31.251 ipcomp(E) Nov 14 09:19:52 31.172.31.251 xfrm_ipcomp(E) Nov 14 09:19:52 31.172.31.251 esp4(E) Nov 14 09:19:52 31.172.31.251 xen_gntdev(E) Nov 14 09:19:52 31.172.31.251 xen_evtchn(E) Nov 14 09:19:52 31.172.31.251 ah4(E) Nov 14 09:19:52 31.172.31.251 xenfs(E) Nov 14 09:19:52 31.172.31.251 af_key(E) Nov 14 09:19:52 31.172.31.251 xfrm_algo(E) Nov 14 09:19:52 31.172.31.251 xen_privcmd(E) Nov 14 09:19:52 31.172.31.251 ipmi_poweroff(E) Nov 14 09:19:52 31.172.31.251 video(E) Nov 14 09:19:52 31.172.31.251 thermal(E) Nov 14 09:19:52 31.172.31.251 fan(E) Nov 14 09:19:52 31.172.31.251 ac(E) Nov 14 09:19:52 31.172.31.251 battery(E) Nov 14 09:19:52 31.172.31.251 ip6t_REJECT(E) Nov 14 09:19:52 31.172.31.251 nf_reject_ipv6(E) Nov 14 09:19:52 31.172.31.251 ip6table_filter(E) Nov 14 09:19:52 31.172.31.251 ip6_tables(E) Nov 14 09:19:52 31.172.31.251 ipt_REJECT(E) Nov 14 09:19:52 31.172.31.251 nf_reject_ipv4(E) Nov 14 09:19:52 31.172.31.251 xt_tcpudp(E) Nov 14 09:19:52 31.172.31.251 iptable_filter(E) Nov 14 09:19:52 31.172.31.251 ip_tables(E) Nov 14 09:19:52 31.172.31.251 x_tables(E) Nov 14 09:19:52 31.172.31.251 bridge(E) Nov 14 09:19:52 31.172.31.251 stp(E) Nov 14 09:19:52 31.172.31.251 llc(E) Nov 14 09:19:52 31.172.31.251 ext4(E) Nov 14 09:19:52 31.172.31.251 ecb(E) Nov 14 09:19:52 31.172.31.251 crc16(E) Nov 14 09:19:52 31.172.31.251 jbd2(E) Nov 14 09:19:52 31.172.31.251 crc32c_generic(E) Nov 14 09:19:52 31.172.31.251 mbcache(E) Nov 14 09:19:52 31.172.31.251 fuse(E) Nov 14 09:19:52 31.172.31.251 ipmi_devintf(E) Nov 14 09:19:52 31.172.31.251 ipmi_watchdog(E) Nov 14 09:19:52 31.172.31.251 w83627ehf(E) Nov 14 09:19:52 31.172.31.251 hwmon_vid(E) Nov 14 09:19:52 31.172.31.251 nf_conntrack_ipv4(E) Nov 14 09:19:52 31.172.31.251 nf_defrag_ipv4(E) Nov 14 09:19:52 31.172.31.251 nf_conntrack(E) Nov 14 09:19:52 31.172.31.251 loop(E) Nov 14 09:19:52 31.172.31.251 x86_pkg_temp_thermal(E) Nov 14 09:19:52 31.172.31.251 intel_powerclamp(E) Nov 14 09:19:52 31.172.31.251 coretemp(E) Nov 14 09:19:52 31.172.31.251 crct10dif_pclmul(E) Nov 14 09:19:52 31.172.31.251 crc32_pclmul(E) Nov 14 09:19:52 31.172.31.251 ghash_clmulni_intel(E) Nov 14 09:19:52 31.172.31.251 hmac(E) Nov 14 09:19:52 31.172.31.251 drbg(E) Nov 14 09:19:52 31.172.31.251 ansi_cprng(E) Nov 14 09:19:52 31.172.31.251 aesni_intel(E) Nov 14 09:19:52 31.172.31.251 aes_x86_64(E) Nov 14 09:19:52 31
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Mon, Nov 14, 2016 at 04:55:40PM +0100, Andreas Ziegler wrote: > Hi, > > few months later Ingo decided again to give it a try as he really > doesn't want to keep ipv6 disabled in 2016. > He tried Xen 4.8 - which didn't help, the crash reappeared. > He then managed to build Xen with debug=y and soon it crashed with the > following output, which looks a little bit longer than without debug: > > http://paste.debian.net/895464/ > Entry not found -- maybe it expired... Sorry. > If this still doesn't help, we would really appreciate more information > on how to do proper debugging, the information we found online is either > very old, confusing - or it's hidden very good? > What sort of information did you find ? What do you need ? Wei.
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Hi, few months later Ingo decided again to give it a try as he really doesn't want to keep ipv6 disabled in 2016. He tried Xen 4.8 - which didn't help, the crash reappeared. He then managed to build Xen with debug=y and soon it crashed with the following output, which looks a little bit longer than without debug: http://paste.debian.net/895464/ If this still doesn't help, we would really appreciate more information on how to do proper debugging, the information we found online is either very old, confusing - or it's hidden very good? Andreas. Original-Nachricht Betreff: Re: [Xen-devel] Kernel panic on Xen virtualisation in Debian Von: Wei Liu An: Ingo Jürgensmann Datum: 2.8.2016, 14:37:58 > On Tue, Aug 02, 2016 at 12:30:30PM +0200, Ingo Jürgensmann wrote: >> On 02.08.2016 11:20, Wei Liu wrote: >>> On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote: >>> What is also interesting is that you seem to be running some sort of >>> ip accounting software (pmacctd) which also segfault'ed. >> >> Yeah, it is segfaulting, because the database (in a domU VM) where it is >> storing the accounting is not yet available after the crash. When database >> is up&running, those segfaults go away. >> > > At least we can now rule out that it is not related to the issue you > reported. > >>> Still not sure what to make of that though. >> >> Me neither. ;-) >> >> I already tried to get a core dump by setting ulimit -c unlimited, but that >> didn't work as well, which makes me believe that the crash happens in >> hypervisor not in dom0 kernel. When it's dom0 kernel I would expect dumping >> a core file should work. >> > > We can't draw the conclusion that the crash is in hypervisor yet. If > your dom0 crash, hypervisor would normally decide to reboot the machine. > > Wei. >
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Tue, Aug 02, 2016 at 12:30:30PM +0200, Ingo Jürgensmann wrote: > On 02.08.2016 11:20, Wei Liu wrote: > >On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote: > >What is also interesting is that you seem to be running some sort of > >ip accounting software (pmacctd) which also segfault'ed. > > Yeah, it is segfaulting, because the database (in a domU VM) where it is > storing the accounting is not yet available after the crash. When database > is up&running, those segfaults go away. > At least we can now rule out that it is not related to the issue you reported. > >Still not sure what to make of that though. > > Me neither. ;-) > > I already tried to get a core dump by setting ulimit -c unlimited, but that > didn't work as well, which makes me believe that the crash happens in > hypervisor not in dom0 kernel. When it's dom0 kernel I would expect dumping > a core file should work. > We can't draw the conclusion that the crash is in hypervisor yet. If your dom0 crash, hypervisor would normally decide to reboot the machine. Wei. > -- > Ciao... //http://blog.windfluechter.net > Ingo \X/ XMPP: i...@jabber.windfluechter.net > > > gpg pubkey: http://www.juergensmann.de/ij_public_key.asc
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On 02.08.2016 11:20, Wei Liu wrote: On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote: What is also interesting is that you seem to be running some sort of ip accounting software (pmacctd) which also segfault'ed. Yeah, it is segfaulting, because the database (in a domU VM) where it is storing the accounting is not yet available after the crash. When database is up&running, those segfaults go away. Still not sure what to make of that though. Me neither. ;-) I already tried to get a core dump by setting ulimit -c unlimited, but that didn't work as well, which makes me believe that the crash happens in hypervisor not in dom0 kernel. When it's dom0 kernel I would expect dumping a core file should work. -- Ciao... //http://blog.windfluechter.net Ingo \X/ XMPP: i...@jabber.windfluechter.net gpg pubkey: http://www.juergensmann.de/ij_public_key.asc
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote: > Am 29.07.2016 um 13:45 schrieb Andreas Ziegler : > > > we tried with kernel 4.6 now, the crashed happened again, though. > > Please find attached the netconsole log of 3 from 4 crashes so far today… What is also interesting is that you seem to be running some sort of ip accounting software (pmacctd) which also segfault'ed. Still not sure what to make of that though. Wei. > > -- > Ciao... //http://blog.windfluechter.net > Ingo \X/ XMPP: i...@jabber.windfluechter.net > > gpg pubkey: http://www.juergensmann.de/ij_public_key.asc > > >
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Fri, Jul 29, 2016 at 01:45:55PM +0200, Andreas Ziegler wrote: > Hello Wei, > > we tried with kernel 4.6 now, the crashed happened again, though. > > next we want to try the Xen debug build, but we couldn't find any > information on how to enable debug for the build, perhaps you could give > us a hint. > First please read: http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source for general information. Then in the cloned xen.git tree -- assuming you use Xen 4.7 (maybe 4.6 as well) make -C xen menuconfig to get a Linux style menuconfig experience If you're using earlier version of Xen, you need to modify xen.git/Config.mk to set the debug variable. Feel free to ask if you have more questions. Wei.
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Hello Wei, we tried with kernel 4.6 now, the crashed happened again, though. next we want to try the Xen debug build, but we couldn't find any information on how to enable debug for the build, perhaps you could give us a hint. - Andreas Original-Nachricht Betreff: Re: [Xen-devel] Kernel panic on Xen virtualisation in Debian Von: Wei Liu An: Ingo Jürgensmann Datum: 25.7.2016, 15:13:06 > On Mon, Jul 25, 2016 at 01:41:41PM +0200, Ingo Jürgensmann wrote: >> On 25.07.2016 12:23, Wei Liu wrote: >> >> First, thank you for replying! Very much appreciated! :) >> >>> I did skim your emails. But the oops was happening in memcpy+0x6 which >>> indicated it came back to the origin question why would it got an >>> exception there. >>> >>> Just by staring at the code doesn't get me anywhere. Without a concrete >>> reproduction of the issue, I'm afraid I can't provide more input here. >> >> Well, from my point of view, it happens quite often when accessing the >> server via SSH. For example today it crashed when I wanted to add something >> and after I clicked into putty and typed the first char. In another putty, >> where I have my netconsole log open, I instantly saw the oops. >> >> But what exactly causing these kinds of reboots, I'm clueless as you too. >> Only that I do experience far more frequent crashes when accessing the >> server from workplace via putty on Windows than via SSH on OSX or Debian >> Linux. >> >>> There are several moving parts: >>> 0. Hardware >>> 1. Xen hypervisor >>> 2. Dom0 kernel >>> Your report and the debian report suggested that Dom0 kernel is less >>> likely to be the culprit because you've tried different Dom0 kernels. >> >> As just written in the other mail, I already tried kernel 4.5 from >> backports. Still crashing. >> >>> As for Xen, not sure if you would be up for trying a debug build from >>> source tree. That would help provide information on whether this is a >>> bug in Xen or not. >> >> Will try to build from Debian source, but how to enable debug build? >> > > I was thinking about building directly from xen.git. > > http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source > > Probably try the Xen 4.7 release. > > Wei. > >> -- >> Ciao... //http://blog.windfluechter.net >> Ingo \X/ XMPP: i...@jabber.windfluechter.net >> >> >> gpg pubkey: http://www.juergensmann.de/ij_public_key.asc
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On Mon, Jul 25, 2016 at 01:41:41PM +0200, Ingo Jürgensmann wrote: > On 25.07.2016 12:23, Wei Liu wrote: > > First, thank you for replying! Very much appreciated! :) > > >I did skim your emails. But the oops was happening in memcpy+0x6 which > >indicated it came back to the origin question why would it got an > >exception there. > > > >Just by staring at the code doesn't get me anywhere. Without a concrete > >reproduction of the issue, I'm afraid I can't provide more input here. > > Well, from my point of view, it happens quite often when accessing the > server via SSH. For example today it crashed when I wanted to add something > and after I clicked into putty and typed the first char. In another putty, > where I have my netconsole log open, I instantly saw the oops. > > But what exactly causing these kinds of reboots, I'm clueless as you too. > Only that I do experience far more frequent crashes when accessing the > server from workplace via putty on Windows than via SSH on OSX or Debian > Linux. > > >There are several moving parts: > >0. Hardware > >1. Xen hypervisor > >2. Dom0 kernel > >Your report and the debian report suggested that Dom0 kernel is less > >likely to be the culprit because you've tried different Dom0 kernels. > > As just written in the other mail, I already tried kernel 4.5 from > backports. Still crashing. > > >As for Xen, not sure if you would be up for trying a debug build from > >source tree. That would help provide information on whether this is a > >bug in Xen or not. > > Will try to build from Debian source, but how to enable debug build? > I was thinking about building directly from xen.git. http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source Probably try the Xen 4.7 release. Wei. > -- > Ciao... //http://blog.windfluechter.net > Ingo \X/ XMPP: i...@jabber.windfluechter.net > > > gpg pubkey: http://www.juergensmann.de/ij_public_key.asc
Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
On 25.07.2016 12:23, Wei Liu wrote: First, thank you for replying! Very much appreciated! :) I did skim your emails. But the oops was happening in memcpy+0x6 which indicated it came back to the origin question why would it got an exception there. Just by staring at the code doesn't get me anywhere. Without a concrete reproduction of the issue, I'm afraid I can't provide more input here. Well, from my point of view, it happens quite often when accessing the server via SSH. For example today it crashed when I wanted to add something and after I clicked into putty and typed the first char. In another putty, where I have my netconsole log open, I instantly saw the oops. But what exactly causing these kinds of reboots, I'm clueless as you too. Only that I do experience far more frequent crashes when accessing the server from workplace via putty on Windows than via SSH on OSX or Debian Linux. There are several moving parts: 0. Hardware 1. Xen hypervisor 2. Dom0 kernel Your report and the debian report suggested that Dom0 kernel is less likely to be the culprit because you've tried different Dom0 kernels. As just written in the other mail, I already tried kernel 4.5 from backports. Still crashing. As for Xen, not sure if you would be up for trying a debug build from source tree. That would help provide information on whether this is a bug in Xen or not. Will try to build from Debian source, but how to enable debug build? -- Ciao... //http://blog.windfluechter.net Ingo \X/ XMPP: i...@jabber.windfluechter.net gpg pubkey: http://www.juergensmann.de/ij_public_key.asc