Hi Ben Thank you. I just ran some tests with the same data on EC and R3 volumes (same hardware). R3 is a lot faster....
EC untar 48.879s find 2.993s rm 11.244s R3 untar 10.938s find 0.722s rm 4.144s On Wed, Sep 27, 2017 at 3:12 AM, Ben Turner <[email protected]> wrote: > Have you done any testing with replica 2/3? IIRC my replica 2/3 tests out > performed EC on smallfile workloads, it may be worth looking into if you > can't get EC up to where you need it to be. > > -b > > ----- Original Message ----- > > From: "Dmitri Chebotarov" <[email protected]> > > Cc: "gluster-users" <[email protected]> > > Sent: Tuesday, September 26, 2017 9:57:55 AM > > Subject: Re: [Gluster-users] sparse files on EC volume > > > > Hi Xavi > > > > At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM > cached > > bricks. > > There is no RAID for data bricks, but I'll be using hardware RAID10 for > SSD > > cache disks (I can use 'writeback' cache in this case). > > > > 'small file performance' is the main reason I'm looking at different > options, > > i.e. using formated sparse files. > > I spent considerable amount of time tuning 10GB/kernel/gluster to reduce > > latency - the small file performance improved ~50% but it's still no good > > enough, especially when I need to use Gluster for /home folders. > > > > I understand limitations and single point of failure in case with sparse > > files. I'm considering different options to provide HA > (pacemaker/corosync, > > keepalived or using VMs - RHEV - to deliver storage). > > > > Thank you for your reply. > > > > > > On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez < [email protected] > > > wrote: > > > > > > Hi Dmitri, > > > > On 22/09/17 17:07, Dmitri Chebotarov wrote: > > > > > > > > Hello > > > > I'm running some tests to compare performance between Gluster FUSE mount > and > > formated sparse files (located on the same Gluster FUSE mount). > > > > The Gluster volume is EC (same for both tests). > > > > I'm seeing HUGE difference and trying to figure out why. > > > > Could you explain what hardware configuration are you using ? > > > > Do you have a plain disk for each brick formatted in XFS, or do you have > some > > RAID configuration ? > > > > > > > > > > Here is an example: > > > > GlusterFUSE mount: > > > > # cd /mnt/glusterfs > > # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1 > > 1+0 records in > > 1+0 records out > > 1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s* > > > > Sparse file (located on GlusterFUSE mount): > > > > # truncate -l 100GB /mnt/glusterfs/xfs-100G.img > > # mkfs.xfs /mnt/glusterfs/xfs-100G.img > > # mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G > > # cd /mnt/xfs-100G > > # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1 > > 1+0 records in > > 1+0 records out > > 1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s* > > > > The same goes for working with small files (i.e. code file, make, etc) > with > > the same data located on FUSE mount vs formated sparse file on the same > FUSE > > mount. > > > > What would explain such difference? > > > > First of all, doing tests with relatively small files tends to be > misleading > > because of caching capacity of the operating system (to minimize that, > you > > can add 'conv=fsync' option to dd). You should do tests with file sizes > > bigger than the amount of physical memory on servers. This way you > minimize > > cache effects and see the real sustained performance. > > > > A second important point to note is that gluster is a distributed file > system > > that can be accessed simultaneously by more than one client. This means > that > > consistency must be assured in all cases, which makes things go to bricks > > sooner than local filesystems normally do. > > > > In your case, all data saved to the fuse volume will most probably be > present > > on bricks once the dd command completes. On the other side, the test > through > > the formatted sparse file, most probably, is keeping most of the data in > the > > cache of the client machine. > > > > Note that using the formatted sparse file makes it possible a better use > of > > local cache, improving (relatively) small file access, but on the other > > side, this filesystem can only be used from a single client (single > mount). > > If this client fails for some reason, you will loose access to your data. > > > > > > > > > > How does Gluster work with sparse files in general? I may move some of > the > > data on gluster volumes to formated sparse files.. > > > > Gluster works fine with sparse files. However you should consider the > > previous points before choosing the formatted sparse files option. I > guess > > that the sustained throughput will be very similar for bigger files. > > > > Regards, > > > > Xavi > > > > > > > > > > Thank you. > > > > > > _______________________________________________ > > Gluster-users mailing list > > [email protected] > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > [email protected] > > http://lists.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
