Re: [sqlite] 'database disk image is malformed' only on the mac

On 6 Oct 2017, at 1:55pm, Fahad wrote: > I can now reliably corrupt my database - and this happens only when some > other process is writing to the database while I perform sqlite3_close_v2 on > the connections. Reliable corruption will help investigation tremendously. To

Re: [sqlite] 'database disk image is malformed' only on the mac

I can now reliably corrupt my database - and this happens only when some other process is writing to the database while I perform sqlite3_close_v2 on the connections. I'll explain what I'm doing. Since the last I wrote, I've disabled all the flags other than these: #define SQLITE_ENABLE_FTS3 1

Re: [sqlite] 'database disk image is malformed' only on the mac

27 AM To: sqlite-users@mailinglists.sqlite.org Subject: Re: [sqlite] 'database disk image is malformed' only on the mac I don't think so: https://sqlite.org/threadsafe.html "With -DSQLITE_THREADSAFE=2 the threading mode is multi-thread." Setting it to 0 disables all mutexes (assumes sin

Re: [sqlite] 'database disk image is malformed' only on the mac

I don't think so: https://sqlite.org/threadsafe.html "With -DSQLITE_THREADSAFE=2 the threading mode is multi-thread." Setting it to 0 disables all mutexes (assumes single threaded) So I've set it to be multi-threaded. Okay so I've wrapped @synchronized(..) around my database usage, stopped

Re: [sqlite] 'database disk image is malformed' only on the mac

>#define SQLITE_ENABLE_FTS3 1 >#define SQLITE_OMIT_DEPRECATED 1 >#define SQLITE_OMIT_SHARED_CACHE 1 >#define SQLITE_OMIT_AUTOMATIC_INDEX 1 >#define SQLITE_OMIT_DECLTYPE 1 > >#define SQLITE_DEFAULT_MMAP_SIZE 0 >#define SQLITE_DEFAULT_MEMSTATUS 0 >#define SQLITE_DEFAULT_SYNCHRONOUS 1 > >#define

Re: [sqlite] 'database disk image is malformed' only on the mac

On 09/21/2017 04:20 AM, Fahad wrote: Thanks Jens, yes I didn't take the warnings from the Thread Sanitizer lightly. Although I'm very confident with the actual implementation (using thread local dictionaries) and have verified time and again using unit tests that my code is otherwise

Re: [sqlite] 'database disk image is malformed' only on the mac

On Wed, 20 Sep 2017 06:41:05 -0700 (MST), Fahad wrote: [...] > 1) Thread A: Create a new connection, if one already does not exist for the > thread. Store it's 'reference count' in the thread storage (I close a > connection when the count becomes zero). > 2) Thread A: Get a

Re: [sqlite] 'database disk image is malformed' only on the mac

Thanks Jens, yes I didn't take the warnings from the Thread Sanitizer lightly. Although I'm very confident with the actual implementation (using thread local dictionaries) and have verified time and again using unit tests that my code is otherwise thread-safe, I really do think there are perhaps

Re: [sqlite] 'database disk image is malformed' only on the mac

No I'm not. -- Sent from: http://sqlite.1065341.n5.nabble.com/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 'database disk image is malformed' only on the mac

Fahad, Are you calling sqlite3_wal_checkpoint_v2? Brian Macy On Sep 20, 2017, 1:59 PM -0400, wrote: > > I've run the Thread Sanitizer with my own SQLite-based on macOS, and haven't > seen any warnings in sqlite3.c. So what you got could be a real warning sign.

Re: [sqlite] 'database disk image is malformed' only on the mac

> On Sep 19, 2017, at 8:20 PM, Fahad wrote: > > I recently switched on the Thread Sanitizer in Xcode only to find that it > was complaining of race conditions inside of the sqlite3.c code, that the > various readers and writers were trying to read / write to the same >

Re: [sqlite] 'database disk image is malformed' only on the mac

I've tried that as well. Since I'm using PRAGMA journal_mode=WAL on all the connections, I've had issues with MMAP (as acknowledged by the threads above) so have had to disable that. I also need FTS 3 to work. The rest of the flags to do with synchronisation and threading, I've enabled / disabled

Re: [sqlite] 'database disk image is malformed' only on the mac

I apologise for the many posts, but I'm writing in hope that one of you may point out something that I'm either doing wrong, or a concept I haven't fully grasped. I'm aware that prepared statements are tied to the database connection they were created for. In order to get more speed out of my

Re: [sqlite] 'database disk image is malformed' only on the mac

I can't be certain now but I think this thread is related: http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366.html It was this thread that I landed on earlier this year to presumably fix the issues I was experiencing personally. This did help, in

Re: [sqlite] 'database disk image is malformed' only on the mac

On 20 Sep 2017, at 4:20am, Fahad wrote: > These are the flags I've finally settled on: Revert all those settings. Allow SQLite to use its default settings. See if that makes your problem go away. This is purely for testing. Once you know whether it works or not you can

Re: [sqlite] 'database disk image is malformed' only on the mac

Hi Dan The plugin I'm referring to is a 'Share' plugin that one can embed inside of a mac app, which then shows up in the "Sharing" menu in Safari. When you click on it, it launches in its own process, allowing you to share the currently viewed website with your main app. Thus, the main app and

Re: [sqlite] 'database disk image is malformed' only on the mac

On 09/15/2017 10:38 AM, Fahad wrote: I tried it with sqlite3_shutdown(), didn't help. I was able to reproduce it again using the main app and the safari share plugin. You don't need to call sqlite3_shutdown(). All it does is release the resources allocated by sqlite3_initialize() - which are

Re: [sqlite] 'database disk image is malformed' only on the mac

On 15 Sep 2017, at 2:24pm, Fahad wrote: > #define SQLITE_ENABLE_STAT4 1 Should not affect your problem. > I also changed this: > > #define SQLITE_DEFAULT_WAL_AUTOCHECKPOINT 1000 > > to 250 That may affect your problem. Try extreme values, with the objective being to

Re: [sqlite] 'database disk image is malformed' only on the mac

Hi Simon Wanted to chime in to say I was able to in fact cause the database to go corrupt from other external processes trying to write to the database (in WAL mode) while the main app was also using the database. So I have a feeling it's something else. I've tried hard to create a mini-app that

Re: [sqlite] 'database disk image is malformed' only on the mac

On 15 Sep 2017, at 4:38am, Fahad wrote: > How can I prevent this from happening? Like I said, using a unit test I was > able to reproduce this 1 out of 20 tries but when using the real app and > plugin at the same time, I am able to reproduce every 5th try (it seems the > app

Re: [sqlite] 'database disk image is malformed' only on the mac

Hi Simon I tried it with sqlite3_shutdown(), didn't help. I was able to reproduce it again using the main app and the safari share plugin. Main App: Finished writing to db, just opened a new connection to read (after opening a connection I set PRAGMA query_only=1; PRAGMA read_uncommitted=1; and

Re: [sqlite] 'database disk image is malformed' only on the mac

On 18 Aug 2017, at 12:30am, Fahad G wrote: > I do not have a way to reproduce this just yet, but I've been seeing way too > many diagnostic logs from customers where their databases are being corrupt > primarily on the Mac (the exact same code is shared between a Mac app,