On Nov 15 2016, Nikolaus Rath <[email protected]> wrote:
> On Nov 15 2016, Randy Rue <[email protected]> wrote:
>> * What's the DB look like for a PB of data with several billion files?
>> Size, performance?
>
> Before is O(log n), so that should be good. Size is linear in stored
> data and files. I would create a filesystem with a few GBs and thousands
> of files, and extrapolate from there.
Actually, there is one important thing I forgot to mention:
While the performance is O(log n), you will experience an extreme drop
of performance if important parts of the metadata DB no longer fit in
your RAM. The amount of work that S3QL has to do does not change, but it
will suddenly have to wait ages to read something from the DB.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
