On Apr 28 2015, Maxim Kostrikin <[email protected]> wrote:
> Hello,
> I am faced with performance issue. My s3ql mount attached to s3 backend
> with options: --cachesize 20000000 --max-cache-entries 1000000 --threads 16
> --allow-other --backend-options no-ssl,tcp-timeout=30 --nfs
> I copy into s3ql mount many small files 100 - 1000 bytes. With time copy
> speed slowdown almost stall.
Please be more precise. How did you measure the speed? How fast was it
initially, how fast at the end?
> I see almost 16 connections with s3 servers, but tcpdump shows small rate
> of uploading PUTs.
What do you mean with small? What do you expect it to be, and what did
you find instead?
Generally every file needs one network request, which means tens to
hundreds of milliseconds of network latency per file.
> Ok, I remounted with --threads 1024 ( I bet it was a bad idea), but
> connections with s3 was 24 max and lots of cross locks between
> threads.
How did you determine that?
> 1. How to increase parallels request to s3? Or increase upload
> performance?
In theory more threads should give you more simulateous connection -
unless you're limited by something else.
> 2. What limits could be reasonable for --threads parameter? Why? SQLite
> bottleneck?
Difficult to say without further information.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
