On Dec 13 2016, Randy Rue <[email protected]> wrote: > Hi Nikolaus, > > >> > * I've mounted the volume with mount.s3ql calling for 24 uplink threads, >> is >> > this correct? >> There is no "correct" value. It means S3QL will try to do up to 24 >> uploads in parallel. > > > Forgive me for asking the question imprecisely. Perhaps a better question > is "Is this consistent with best practice" or "Do you have a > recommendation?"
>From the output of benchmark.py you can tell that when using zlib compression, increasing the number of threads above 2 does not give any benefits. So if you use zlib compression (as in your example below), using more than 2 threads is more likely to have negative effects than positive ones. >> * I've tried several different combinations of arguments attempting to >> set >> > the cache size and maximum cache entries but in every case, cache >> entries >> > top out at about ~4K and cache size seems to float around 30-35GB, >> > presumably shifting as the size of those 4K entries changes? >> >> You need to be more specific. Please post a concrete set of options that >> you used, the results that you got, and the results that you'd rather >> have. > > > My most recent attempt mounted using: > /usr/bin/mount.s3ql --nfs --compress zlib-6 --authfile /etc/s3ql.authinfo > --log syslog --threads 24 --cachedir /cache --cachesize 346030080 > --allow-other swift://tin/fast_dr/ /fast_dr This should result in a message of the form "Detected xxxx available file descriptors". What's xxx in your case? > In both cases, while I have multiple rsyncs running from a mounted NFS > client, "watch s3qlstat" shows the cache grow to ~4,000 entries and no > more. Cache size grows to 38-40GB at the most, and dirty cache eventually > grows to take up all of the cache. At that point whatever throughput I was > seeing starts to drop. > > At that point I've also tried changing the cache size on the fly using > "s3qlctrl cachesize /fast_dr/ 781250000" (that's 100GB) and saw the cache > size float upward maybe 5GB and then return to ~30GB. Cache entries didn't > change. Looks like you're limited by the number of cache entries. > I've also tried it with the argument "--max-cache-entries 40960" > included. Did that increase the number of cache entries to more than ~4000? > I would like to see more of the 300GB cache volume used to see if this will > help overall performance or at least postpone the point where all of the > cache is dirty and write-throughs start. > > Related: > >> What's limiting my cache entries and size? >> The number of available file descriptors and the --max-cache-entries >> argument. > > > Am I missing something about using the max-cache-entries argument? It > doesn't seem to make a difference. > > Forgive me if this isn't a s3ql question, but what determines the number of > available file descriptors? The first is a resource limit, you can check (and modify) it with "ulimit". In addition to this the kernel imposes a limit, cf. http://unix.stackexchange.com/questions/84227/limits-on-the-number-of-file-descriptors >> * In every case (depending on the number of rsync clients and their >> network >> > connections) I get up to about 200MB/s and no more despite the server >> > having bonded 10Gb connections and the back end swift cluster having >> > multiple 10Gb connections. Where's my bottleneck? >> I assume you mean what is limiting the upload speed to the server to 24 >> MB/s? That's not something that benchmark.py can determine. Do you get >> more than 24 MB/s when you use a different swift client? If not, then >> the server is to blame. > > > I mean "How can I find out why my total write speed is only as high as > 200MB/s when the NFS client, the s3ql server and the swift cluster all have > multiple 10Gb connections?" Well, benchmark.py already told you why. Even when you use 128k blocks, S3QL cannot process more than 169001 KiB/sec. That limitation is a combination of your CPU/disk/mainboard, the kernel, and Python. In addition to that, S3QL is not able to send data to the server any quicker than 24005 KB/s - even if it does not do any processing at all. So I would investigate that first. > I'll test some rsyncs from the client to a non-s3ql target on the s3ql > system as a data point for incoming speed, and some swift calls from the > s3ql server to the swift cluster. The latter would be more important. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
