On 11/17/2015 12:11 AM, Deon Brewis wrote: > We found that using SQLITE memory mapped I/O (mmap_size > 0) in OSX isn't > usable in any way shape or form. It will inevitably lead to catalog > corruption if you hard-reboot OSX, even without the database or application > open. >
I tried to turn on mmap I/O in SQLite 3.9.1 in our application on CentOS 7 and Debian Jessie, but after 24-48h of continous writes to the DB I always get a corruption: SQLite result 0xb: database corruption at line 76915 of [767c1727fe] SQLite result 0xb: statement aborts at 10: [DELETE FROM revision_blocks WHERE revision_id=:revision_id] Query "DELETE FROM revision_blocks WHERE revision_id=:revision_id" failed: (code 0xb: database disk image is malformed) database disk image is malformed After this 'pragma integrity check' says: row 90814 missing from index sqlite_autoindex_revision_ops_1 wrong # of entries in index sqlite_autoindex_revision_ops_1 row 1046646 missing from index idx_revmap row 1046646 missing from index sqlite_autoindex_revision_blocks_1 wrong # of entries in index idx_revmap wrong # of entries in index sqlite_autoindex_revision_blocks_1 There are not reboots involved, just multiple processes accessing a WAL DB. Without mmap I/O I've never seen corrupted DBs in our application. The corruption doesn't lead to any data loss, apparently only the indexes were corrupted beacuse I was able to recover the DBs with .clone+setting journal mode again. However I haven't had the time yet to write a standalone testcase and attempt to reproduce this with SQLite alone, so I can't rule out that this isn't a memory corruption bug in our application, or one of the other libraries that we link with. I notice that SQLite doesn't use msync, and msync(2) says 'Without use of this call there is no guarantee that changes are written back before munmap(2) is called'. Is f(data)sync enough to ensure changed mmap pages are written to the disk? -- Edwin T?r?k | Co-founder and Lead Developer Skylable open-source object storage: reliable, fast, secure http://www.skylable.com
