Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-19 Thread Brendan Duddridge
I'm using SQLCipher in my main app and it's using SQLite 3.15.2. However, my little sample app where I could also easily demonstrate the problem, is using whatever the built-in SQLite version is in macOS Sierra 10.12.4. I couldn't find SQLITE_MMAP_READWRITE in the Couchbase Lite source anywhere,

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-17 Thread Brendan Duddridge
Thanks Richard for your reply. Sorry about the COLLATE problem. That's a Couchbase Lite thing. I find it weird that a different WAL file is getting in there somehow when a power failure occurs. I'm a bit stumped at how it can even write to the file system the moment the power shuts down so I

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-16 Thread Brendan Duddridge
caused the Mac's power to be killed. When the database is opened, it's using WAL journal_mode and also has pragma fullfsync=1 enabled. Any ideas of settings I could try to resolve this problem would be greatly appreciated by me and my customers. Thanks, Brendan Duddridge

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
> I’m the architect of Couchbase Lite and the lead developer for iOS and > macOS. I enabled SQLite’s memory-mapped I/O at least two years ago. I never > considered it could be problematic since (a) Brendan is IIRC the only > iOS/Mac developer who’s reported database corruption, and (b) I assumed >

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
cy to copy > files around for eg. auto-save purposes. > > I'd be very *very* surprised if there's an sqlite bug here. I'd take the > advice given to you on bountysource and watch your application's file > system operations to begin to understand what is going on behind the > scenes.

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
y > files around for eg. auto-save purposes. > > I'd be very *very* surprised if there's an sqlite bug here. I'd take the > advice given to you on bountysource and watch your application's file > system operations to begin to understand what is going on behind the > scenes. > > -Ro

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
before I cut the power. On Tue, Apr 18, 2017 at 2:23 AM, Richard Hipp <d...@sqlite.org> wrote: > On 4/18/17, Brendan Duddridge <brend...@gmail.com> wrote: > > In both cases though I can cause SQLite file corruption by cutting the > > power on my MacBook Pro. > >

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
://github.com/couchbase/couchbase-lite-ios/issues/1482 (newer posts are at the bottom) On Tue, Apr 18, 2017 at 2:52 AM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 18 Apr 2017, at 6:01am, Brendan Duddridge <brend...@gmail.com> wrote: > > > Perhaps the only think I can think

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
Hi Jens, It would be a good test if you could independently verify my findings using the sample app I wrote to see if you are able to reproduce the corruption with memory mapped I/O turned ON and the fact that there's no corruption when it's turned OFF. I know it seem strange given the

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
ive may also re-order the data so that later writes > may be present, while earlier writes are not. > > This is not a theoretical edge case. This scenario is easily reproduced > with real world workloads and drive power failures.” > > - Deon > > -Original

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

2017-04-18 Thread Brendan Duddridge
Hi Deon, BINGO I commented out this line of code in the Couchbase lite CBL_SQliteStorage.m source file and no more corruption //int err = sqlite3_config(SQLITE_CONFIG_MMAP_SIZE, (SInt64)kSQLiteMMapSize, (SInt64)-1); It would be really awesome of this were added to the How to