On Sep 08 2016, Roger Gammans <rgamm...@gammascience.co.uk> wrote:
> On 7 September 2016 at 22:37, Nikolaus Rath <nikol...@rath.org> wrote:
>
>> On Sep 07 2016, Roger Gammans <rgamm...@gammascience.co.uk> wrote:
>> > apsw.CorruptError: CorruptError: database disk image is malformed
>>
>> That means you'll have to discard the locally cached metadata. The next
>> fsck.s3ql will recover whatever was most recently stored in S3.
>>
>> > Traceback (most recent call last):
>> > File "/usr/lib/s3ql/s3ql/block_cache.py", line 535, in upload
>> > obj_id = self.db.rowid('INSERT INTO objects (refcount, size)
>> VALUES(1,
>> > -1)')
>> > File "/usr/lib/s3ql/s3ql/database.py", line 104, in rowid
>> > self.conn.cursor().execute(*a, **kw)
>> > File "src/cursor.c", line 231, in resetcursor
>> > apsw.FullError: FullError: database or disk is full
>>
>>
> The odd thing here is there is now 6.8G free in that partition and we
> monitor the
> filesystem size every 60s and the monitoring doesn't show a spike. So
> either something used a lot of disk space
> quickly or we reached a maximum cache Db size? ( I thought it was sqlite ,
> which apparently has 2^64 as a max rows
> entry )
Yes, you should not be reaching a maximum size anytime soon.
>> > Can you shed any light on it ; or give be guidance on metadata sizing
>> > ?
>>
>> Required metadata grows linearly with stored data. The proportionaly
>> factor depends on how big the stored files are, and what block size you
>> chose.
>>
>
> Is that linear with size before or after de-duplication? Given we have
> mulitple backup snapshots create with s3qlcp it makes a big
> difference.
Both. De-duplicated data takes a little less metadata than unique data,
but still scales linearly.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
