[s3ql] Re: Clustering and Benchmarks

2014-10-21 Thread giles
Thanks Ed. Yes you're right, we've been looking at many possibilities for our app. Unfortunately I think S3QL won't scale with us but I wanted to check as it had a large number of the features we were looking for ie caching, dedup, ecryption and compression. The decision we're taking now is

Re: [s3ql] benchmark.py AttributeError: 'ArgumentParser' object has no attribute 'add_ssl'

2014-10-21 Thread Maxim Kostrikin
It helps. But cant figure out, how menchmark.py should work. It there are any how-to/manual? вторник, 21 октября 2014 г., 7:24:03 UTC+7 пользователь Nikolaus Rath написал: good...@gmail.com javascript: writes: Debian Wheezy. Python 3.3.0 running in pyvenv. (virtualenv-3.3.0)

Re: [s3ql] S3QL on local storage backend provided by NFS - suspected incompatibility?

2014-10-21 Thread Super Strobi
Update: I investigated bit more on the backgrounds of compression number of threads, and came to the following conclusion: Benchmark with *sshfs backend:* Preparing test data... Measuring throughput to cache... Cache throughput with 4 KiB blocks: 12446 KiB/sec Cache throughput with 8 KiB

[s3ql] mkfs.s3ql waiting forever

2014-10-21 Thread Andy Cress
Using s3ql-1.17, I encountered an issue that I haven't seen before, where mkfs.s3ql seems to be waiting forever (more than a few hours). An strace shows it waiting forever at recvfrom(4, I have a variety of other configurations that work fine, but I don't know what is going wrong with this one.

[s3ql] Re: mkfs.s3ql waiting forever

2014-10-21 Thread Andy Cress
I re-ran it with --debug all and got this output: [...] Opening connection to /backups/cloudcache/s3:=2F=2Feastporterunitrends=2Farchive=2F.db dump_table(contents): writing 1 rows ..ext_attributes.. Opening connection to /backups/cloudcache/s3:=2F=2Feastporterunitrends=2Farchive=2F.db

[s3ql] Re: mkfs.s3ql waiting forever

2014-10-21 Thread Andy Cress
After investigating on several systems, I can do mkfs.s3ql from another system to the same bucket, and then mount it ok from the original system, but any writes to the s3ql filesystem also hang. My current theory is that there is a firewall or proxy issue on the network where the original system

[s3ql] Benchmarks and Clustering.

2014-10-21 Thread giles
Hi Guys I've been looking through the wiki and I've noticed that this file system doesn't support mounting on multiple machines, I wanted to check whether there has been any benchmarks with this software. How many do we know how many uploads this can take before it maxes out. We're currently a

Re: [s3ql] Benchmarks and Clustering.

2014-10-21 Thread Nikolaus Rath
gi...@liquidhealthcare.com writes: Hi Guys I've been looking through the wiki and I've noticed that this file system doesn't support mounting on multiple machines, I wanted to check whether there has been any benchmarks with this software. I don't think anything has been published. You can