Yes it is exactly the case..... Wonder if there is a better way to tackle this
Thanks Peter On Monday, June 20, 2016 at 12:08:28 PM UTC-7, Nikolaus Rath wrote: > > On Jun 20 2016, Peter Auyeung <[email protected] <javascript:>> > wrote: > > I am trying to rsync between s3ql on a local disk to another s3ql on > google > > storage > > The rsync is SUPER slow with pushing 200GB over a week or so..... > > > > s3qlstat on google storage: > > Preparing test data... > > Measuring throughput to cache... > > Cache throughput with 4 KiB blocks: 12198 KiB/sec > > Cache throughput with 8 KiB blocks: 24942 KiB/sec > > Cache throughput with 16 KiB blocks: 49939 KiB/sec > > Cache throughput with 32 KiB blocks: 85977 KiB/sec > > Cache throughput with 64 KiB blocks: 132026 KiB/sec > > Cache throughput with 128 KiB blocks: 187159 KiB/sec > > Measuring raw backend throughput.. > > Backend throughput: 41466 KiB/sec > > Test file size: 0.00 MiB > > Maybe try with a bigger test file? > > If the data you are storing is distributed over similarly small files, > then it is possible that the bad performance is due to latency (as > opposed to bandwidth). For every file, S3QL needs to make at least one > PUT request. So if you have millions of tiny files, it will spend a lot > of time just waiting for the server to reply without ever being able to > fully utilize the bandwidth. > > > Best, > -Nikolaus > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
