On Dec 05 2016, Chris Davies <[email protected]> wrote:
> On Tuesday, 15 November 2016 14:30:21 UTC, Randy Rue wrote:
>>
>> * What scale are folks running this at? Is a PB typical, unusual, or
>> unheard of?
>>
>
> I'm running a couple of buckets. One has 52TB highly duplicated source data
> compressed down to 4TB. The database contains ~41,000,000 entries and is a
> little over 6GB. The other is 6TB of previously deduplicated data
> compressed down to around 3.5TB. The database for this one is less than
> 100MB.
>
> Personally, I think I'm starting to stretch the practical limits for
> sqlite, and I'm wondering about splitting my larger bucket into several
> smaller ones. I'd lose quite heavily on deduplication but I'd be a little
> less worried about having database corruption destroy all my data. (I've
> had a few unexplained crashes but they are rare enough that I can't
> identify the problem to send as a meaningful bug report. Nikolaus, if you'd
> like to accept a vague report that I attempted to clarify over time, that
> would be acceptable to me.)
Yes, please provide a report. There's might be nothing I can do about it,
but it would be good to keep track of how often it is happening and if
anyone else is affected.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
