Where did you read 2k IOPS? Each disk is able to do about 75iops as I'm using SATA disk, getting even closer to 2000 it's impossible
Il 13 ott 2017 9:42 AM, "Szymon Miotk" <[email protected]> ha scritto: > Depends what you need. > 2K iops for small file writes is not a bad result. > In my case I had a system that was just poorly written and it was > using 300-1000 iops for constant operations and was choking on > cleanup. > > > On Thu, Oct 12, 2017 at 6:23 PM, Gandalf Corvotempesta > <[email protected]> wrote: > > So, even with latest version, gluster is still unusable with small files > ? > > > > 2017-10-12 10:51 GMT+02:00 Szymon Miotk <[email protected]>: > >> I've analyzed small files performance few months ago, because I had > >> huge performance problems with small files writes on Gluster. > >> The read performance has been improved in many ways in recent releases > >> (md-cache, parallel-readdir, hot-tier). > >> But write performance is more or less the same and you cannot go above > >> 10K smallfiles create - even with SSD or Optane drives. > >> Even ramdisk is not helping much here, because the bottleneck is not > >> in the storage performance. > >> Key problems I've noticed: > >> - LOOKUPs are expensive, because there is separate query for every > >> depth level of destination directory (md-cache helps here a bit, > >> unless you are creating lot of directories). So the deeper the > >> directory structure, the worse. > >> - for every file created, Gluster creates another file in .glusterfs > >> directory, doubling the required IO and network latency. What's worse, > >> XFS, the recommended filesystem, doesn't like flat directory sturcture > >> with thousands files in each directory. But that's exactly how Gluster > >> stores its metadata in .glusterfs, so the performance decreases by > >> 40-50% after 10M files. > >> - complete directory structure is created on each of the bricks. So > >> every mkdir results in io on every brick you have in the volume. > >> - hot-tier may be great for improving reads, but for small files > >> writes it actually kills performance even more. > >> - FUSE driver requires context switch between userspace and kernel > >> each time you create a file, so with small files the context switches > >> are also taking their toll > >> > >> The best results I got were: > >> - create big file on Gluster, mount it as XFS over loopback interface > >> - 13.5K smallfile writes. Drawback - you can use it only on one > >> server, as XFS will crash when two servers will write to it. > >> - use libgfapi - 20K smallfile writes performance. Drawback - no nice > >> POSIX filesystem, huge CPU usage on Gluster server. > >> > >> I was testing with 1KB files, so really small. > >> > >> Best regards, > >> Szymon Miotk > >> > >> On Fri, Oct 6, 2017 at 4:43 PM, Gandalf Corvotempesta > >> <[email protected]> wrote: > >>> Any update about this? > >>> I've seen some works about optimizing performance for small files, is > >>> now gluster "usable" for storing, in example, Maildirs or git sources > >>> ? > >>> > >>> at least in 3.7 (or 3.8, I don't remember exactly), extracting kernel > >>> sources took about 4-5 minutes. > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> [email protected] > >>> http://lists.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
