On Tuesday, 15 November 2016 14:30:21 UTC, Randy Rue wrote: > > * What scale are folks running this at? Is a PB typical, unusual, or > unheard of? >
I'm running a couple of buckets. One has 52TB highly duplicated source data compressed down to 4TB. The database contains ~41,000,000 entries and is a little over 6GB. The other is 6TB of previously deduplicated data compressed down to around 3.5TB. The database for this one is less than 100MB. Personally, I think I'm starting to stretch the practical limits for sqlite, and I'm wondering about splitting my larger bucket into several smaller ones. I'd lose quite heavily on deduplication but I'd be a little less worried about having database corruption destroy all my data. (I've had a few unexplained crashes but they are rare enough that I can't identify the problem to send as a meaningful bug report. Nikolaus, if you'd like to accept a vague report that I attempted to clarify over time, that would be acceptable to me.) > * What back ends are you running it on? Standard S3 only or is IAS > possible? > * What kind of GET/PUT traffic would a PB incur? > I'm using OVH Cloud which requires the swiftks backend. I've not really measured IOs in detail because OVH has a very simple charging model (storage and download by GB) and financially it's not been necessary for me to do so. Chris -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
