On Dec 10 2015, Daniel Jagszent <[email protected]> wrote:
>>> > [...] At the end the filesystem had 16 million directory
>>> > entries and 1.5 million inodes (Burp uses hard links excessively) and
>>> > the sqlite database that S3QL uses to store the filesystem structure was
>>> > 1.2 GB uncompressed.
>>
>> This is not unreasonable though. Note that ext4 would require at least 5
>> GB of metadata as well - just to store the inodes (assuming 4096 bytes
>> inode size). That's not yet counting directory entry *names*.
>
> Sure. The size of the sqlite database is reasonable for so many
> inodes/directory entries. But I suspect that ext4 will scale better in
> terms of execution time for normal operations like e.g. file system
> stats (|df|). S3QL needs to do several full table scans
> <https://bitbucket.org/nikratio/s3ql/src/default/src/s3ql/fs.py?fileviewer=file-view-default#fs.py-916:918>
> for that and this will take its time for tables that big (In my case
> approx. 10 seconds).
True. But this could be fixed easily by keeping the relevant numbers in
memory and updating them on write/delete. This would be rather simple,
it's just that no one ever complained about the time required for df or
s3qlstat before. Personally don't use it more than once every few weeks,
so the performance never bothered me.
>> However, I have plans in the drawer to fix this at some point. The idea
>> is to handle reads/writes for blocks that are already cached entirely at
>> the C level. This will allow concurrenty *and* at the same time boost
>> single-threaded performance as well. Just need to find the time...
>
> That sounds great! (More performance always does :) )
> Am I right in assuming that this will speed up read/write syscalls but
> not stuff that solely works on the database?
Yes
> (like opendir or the attr
> and xattr calls)
xattr requires database access and would stay slow.
opendir and getattr don't necessarily act on the database (getattr
results are cached), but they still wouldn't be speed up because S3QL
itself is mostly single threaded.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
