Gabor Gombas wrote: > On Thu, Sep 11, 2008 at 09:13:00AM -0700, [EMAIL PROTECTED] wrote: >> On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly: >>> Which kernel version? If it's not recent, you should try a recent one. >>> We for example are stuck with 2.6.18 due to Xen and it also has problems >>> with jumbo frames; 2.6.2[456] works just fine. >> What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and >> AoE quite heavily and not had any problems. > > It does not use jumbo frames no matter what I've tried. I've stared at > the kernel source for some time and IMHO it _should_ work, but it does > not. If I boot a recent kernel (obviously without Xen) then jumbo frames > work perfectly.
This is an artifact of how the Xen networking works in the version of Xen with CentOS 5. When you create your domU, a vif will be created on the host dom0. The problem is, MTU changes in the domU don't trickle down to the dom0. You'll have to manually set the MTU on any created vif devices *and* the bridge to which they're connected. Consider the following situation: A dom0 machine exists with two ethernet adapters, one (eth0) for front-end traffic and the other (eth1) for AoE traffic. At boot, the bridges br0 and stbr0 are created, and eth0 and eth1 are added to them (respectively). eth1 is set to 9000 MTU, and stbr0 is also set to 9000 MTU. Each domU is created with two virtual ethernet adapters, one connected to br0 (for front-end traffic) and another to stbr0 (for AoE traffic). Upon instantiating the first domU, you would do something similar to: box# xm create some-domu box# ifconfig vif1.1 mtu 9000 (repeat until this command succeeds) box# ifconfig stbr0 mtu 9000 Such a wonderful hack! I have to do something very similar to this each time I start a domU. It's an endless source of trouble. This has some beautiful side-effects. Because the vif is initially created with a 1500 MTU, the system adds it to the bridge with a 1500 MTU. This causes the bridge device to throttle back to 1500 MTU. Any frames received with >1500 MTU during the time it takes you to successfully set the vif and bridge back to 9000 MTU are *dropped*, effectively causing a momentary loss of storage connectivity. Sometimes, the other domUs on the machine will throttle back to smaller frame sizes in the meantime (forcing you to investigate all domUs after starting a new one and manually resetting the frame size back to something reasonable). The fault of this lies in the network layer of Xen. It's got issues. It's also impossible to force a vif creation with a higher MTU. I've come to the conclusion that storage over ethernet is far from ideal for this kind of set-up, given the shortcomings of Xen. One thing I might want to investigate further is having the dom0 do all the storage processing (e.g. exporting all the AoE devices to the dom0 and passing Xen virtual block devices up to the domUs). Both methods have their shortcomings and advantages. Most of the disadvantages/shortcomings are due to using Ethernet as a storage fabric, however, and aren't specific to AoE. > Fortunately none of those Xen hosts are performance critical (yet) so I > didn't do any serious kernel debugging. I've also tried compiling a new > AoE module but the (then) current one from Coraid did not build with the > Xen-patched 2.6.18 and I had no time to try older versions. The performance will be absolutely abysmal without jumbo frame support. Try doing this and report your results! Good luck. -kelsey ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
