Gluster will never perform well for small files.
I believe there is  nothing you can do with this.

From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Andreas Ericsson
Sent: Monday, March 12, 2018 1:47 PM
To: Gluster-users@gluster.org
Subject: [Gluster-users] Expected performance for WORM scenario

Heya fellas.

I've been struggling quite a lot to get glusterfs to perform even halfdecently 
with a write-intensive workload. Testnumbers are from gluster 3.10.7.

We store a bunch of small files in a doubly-tiered sha1 hash fanout directory 
structure. The directories themselves aren't overly full. Most of the data we 
write to gluster is "write once, read probably never", so 99% of all operations 
are of the write variety.

The network between servers is sound. 10gb network cards run over a 10gb (doh) 
switch. iperf reports 9.86Gbit/sec. ping reports a latency of 0.1 - 0.2 ms. 
There is no firewall, no packet inspection and no nothing between the servers, 
and the 10gb switch is the only path between the two machines, so traffic isn't 
going over some 2mbit wifi by accident.

Our main storage has always been really slow (write speed of roughly 1.5MiB/s), 
but I had long attributed that to the extremely slow disks we use to back it, 
so now that we're expanding I set up a new gluster cluster with state of the 
art NVMe SSD drives to boost performance. However, performance only hopped up 
to around 2.1MiB/s. Perplexed, I tried it first with a 3-node cluster using 2GB 
ramdrives, which got me up to 2.4MiB/s. My last resort was to use a single node 
running on ramdisk, just to 100% exclude any network shenanigans, but the write 
performance stayed at an absolutely abysmal 3MiB/s.

Writing straight to (the same) ramdisk gives me "normal" ramdisk speed (I don't 
actually remember the numbers, but my test that took 2 minutes with gluster 
completed before I had time to blink). Writing straight to the backing SSD 
drives gives me a throughput of 96MiB/sec.

The test itself writes 8494 files that I simply took randomly from our 
production environment, comprising a total of 63.4MiB (so average file size is 
just under 8k. Most are actually close to 4k though, with the occasional 
2-or-so MB file in there.

I have googled and read a *lot* of performance-tuning guides, but the 3MiB/sec 
on single-node ramdisk seems to be far beyond the crippling one can cause by 
misconfiguration of a single system.

With this in mind; What sort of write performance can one reasonably hope to 
get with gluster? Assume a 3-node cluster running on top of (small) ramdisks on 
a fast and stable network. Is it just a bad fit for our workload?



The information contained in this e-mail and in any attachments is confidential 
and is designated solely for the attention of the intended recipient(s). If you 
are not an intended recipient, you must not use, disclose, copy, distribute or 
retain this e-mail or any part thereof. If you have received this e-mail in 
error, please notify the sender by return e-mail and delete all copies of this 
e-mail from your computer system(s). Please direct any additional queries to: 
communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 
Group). Registered in Ireland no. 378073. Registered Office: South County 
Business Park, Leopardstown, Dublin 18.
Gluster-users mailing list

Reply via email to