Hi! > >I should have mentioned that the AoE device is backed by a RAID setup that > >is > >able to write well above 120 MB/s. > >If I mount the same filesystem locally, on the server, bonnie tells me > >it's able to do > >sequential writes at ~370 MB/s. > > > >If I write straight to the AoE device, I can get the expected > >line-speed of the network, around ~110 MB/s. > >dd if=/dev/zero of=/dev/etherd/e1.1 bs=1M > > > >However, when mounting a filesystem, and copying a file onto the AoE > >device, I only see about ~70 MB/s. > > > >This leads me to thinking that the performance degradation I'm seeing > >is related to > >the filesystem or the network. > >Of course, I wouldn't expect a filesystem to give the same performance as > >the > >raw device, but I didn't expect to see a ~25% hit in performance, > >especially > >when doing a sequential write. > > > What filesystem do you use? XFS is known to be the recommended > filesystem for AoE. Actually I think this could be due to RAID block sizes: most AoE implementations assume a block size of 512Byte. If you're using a linux software RAID5 with a default chunk size of 512K and you're using 4 disks, a single "block" has 3*512K block size. This is what has to be written when changing data in a file for example. mkfs.ext4 or mkfs.xfs respects those block sizes, stride sizes, stripe width and so on (see man pages) when the information is available (which is not the case when creating a file system on an AoE device.
To check if you're hit by this is quite simple: install dstat or iostat on the server exporting the volume. Run your benchmark and watch the output of dstat/iostat: if you experience massive reads while writing, congrats, you found the root cause. To improve things a little, create the file system on the server that is exporting the AoE targets. To improve them even more -- especially with RAID5 and RAID6 -- choose a smaller chunk size. I'd be glad if you could post back some numbers... :-) On a side note: linear performance isn't what is counting when using network storage. You better measure iops (input/output operations per second). I use fio for benchmarks which lets you define your I/O patterns to (kind of) fit real world usage. -- Adi ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
