On Jun 20 2016, Peter Auyeung <[email protected]> wrote:
> I am trying to rsync between s3ql on a local disk to another s3ql on google
> storage
> The rsync is SUPER slow with pushing 200GB over a week or so.....
>
> s3qlstat on google storage:
> Preparing test data...
> Measuring throughput to cache...
> Cache throughput with 4 KiB blocks: 12198 KiB/sec
> Cache throughput with 8 KiB blocks: 24942 KiB/sec
> Cache throughput with 16 KiB blocks: 49939 KiB/sec
> Cache throughput with 32 KiB blocks: 85977 KiB/sec
> Cache throughput with 64 KiB blocks: 132026 KiB/sec
> Cache throughput with 128 KiB blocks: 187159 KiB/sec
> Measuring raw backend throughput..
> Backend throughput: 41466 KiB/sec
> Test file size: 0.00 MiB
Maybe try with a bigger test file?
If the data you are storing is distributed over similarly small files,
then it is possible that the bad performance is due to latency (as
opposed to bandwidth). For every file, S3QL needs to make at least one
PUT request. So if you have millions of tiny files, it will spend a lot
of time just waiting for the server to reply without ever being able to
fully utilize the bandwidth.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
