Re: [sqlite] SQLITE_OMIT_* vs amalgamation

I'm building for both Windows and Linux and it's a reproducible build, so I'd rather not depend on anything extra. Thanks, - Kim On Wed, Apr 19, 2017 at 2:10 AM, David Burgess wrote: > If you are running Linux, the build from source (i.e. making your own > amalgamation) is

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> On Apr 18, 2017, at 9:46 PM, Deon Brewis wrote: > > I did report it: > http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog-corruption-td85620.html Thanks for the pointer to the thread. There was a reply by Dan Kennedy shortly thereafter: >> As of

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> That is a hugely important piece of information, and while I thank you for > letting me know, I’m shocked that this hasn’t been conveyed to the SQLite > developers (or, apparently, to Apple.) Did you report it anywhere? I did report it:

[sqlite] gnu gcc address sanitizer finds stack-use-after-scope in sqlite3VdbeExec

Hi there, I just downloaded version 3.18 and after compiling it with the following command gcc shell.c sqlite3.c -g -fsanitize=address -DSQLITE_DEBUG -o sqlite -ldl -pthread running sqlite command .selftest --init I get a sanitizer error message. In my opinion this is probably due to erroneous

Re: [sqlite] SQLITE_OMIT_* vs amalgamation

If you are running Linux, the build from source (i.e. making your own amalgamation) is straightforward. You may have to install a few extras, but nothing onerous. On Wed, Apr 19, 2017 at 5:57 AM, Kim Gräsman wrote: > Hi all, > > The docs at

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> On Apr 18, 2017, at 2:35 PM, Bob Friesenhahn > wrote: > > If the filesystem implementation is not fully-coherent, then data written by > programmed file I/O may not be reflected in the memory mapped space, > resulting in programs using something other than

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On Tue, 18 Apr 2017, Jens Alfke wrote: On Apr 18, 2017, at 2:20 AM, Deon Brewis wrote: It's not like it was subtle - it's a dead on repro. I was able to repro this by doing a power cycle 2 hours after shutting the app down. OSX didn't seem to have any interest in

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Jens, It would be a good test if you could independently verify my findings using the sample app I wrote to see if you are able to reproduce the corruption with memory mapped I/O turned ON and the fact that there's no corruption when it's turned OFF. I know it seem strange given the

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> On Apr 18, 2017, at 12:55 PM, Jeffrey Mattox wrote: > > I'm an iOS and macOS developer. Mac app bundles are special in other ways > beside just having a bit set. Brendan is not storing his database in the app bundle. Both he and Simon have already said that. —Jens

[sqlite] SQLITE_OMIT_* vs amalgamation

Hi all, The docs at https://sqlite.org/compile.html are a little ominous when it comes to the OMIT flags with the amalgamation: > Important Note: The SQLITE_OMIT_* options may not work with the amalgamation. > SQLITE_OMIT_* compile-time options usually work correctly only when SQLite is > built

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

I'm an iOS and macOS developer. Mac app bundles are special in other ways beside just having a bit set. For one, there's a security check somewhere that verifies that the app bundle has not been changed, as those files are expected to be read-only. Apple says apps should put their data

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> On Apr 18, 2017, at 2:20 AM, Deon Brewis wrote: > > It's not like it was subtle - it's a dead on repro. I was able to repro this > by doing a power cycle 2 hours after shutting the app down. OSX didn't seem > to have any interest in flushing mmap files until you soft

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 7:37pm, Brendan Duddridge wrote: > Well, perhaps the way I've been bundling the db.sqlite file within a > package triggers this particular bug. Although a package is really just a > sub-folder, so I don't see how that would make a difference. I don’t

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> I’m the architect of Couchbase Lite and the lead developer for iOS and > macOS. I enabled SQLite’s memory-mapped I/O at least two years ago. I never > considered it could be problematic since (a) Brendan is IIRC the only > iOS/Mac developer who’s reported database corruption, and (b) I assumed >

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 6:46pm, Jens Alfke wrote: > This is a statement about hard disk controller firmware and is true for any > OS. (I used to work for Apple, and corresponded with filesystem architect > Dominic Giampaolo about this back in the day.) Some disk controllers

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

> On Apr 18, 2017, at 1:50 AM, Deon Brewis wrote: > > Are you by change using memory mapped IO (MMAP_SIZE something other than 0)? > > This does not work on OSX. Not even remotely. I tracked an issue down in > November 2015, and was able to trivially corrupt a database 100%

Re: [sqlite] performing an UPDATE on a table-valued function

On Tue, Apr 18, 2017 at 12:04 AM, Hick Gunter wrote: > A TVF is just a "calling convention" for abbreviating a specific SELECT on > a virtual table. It implies neither mutability of the returned values, nor > persistence beyond the scope of the statement. Indeed the carray()

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 11:52am, Richard Hipp wrote: > I was unaware that memory-mapped I/O was busted on MacOS. I will > investigate further, but probably the solution will be that we will > completely disable memory-mapped I/O on MacOS, just as we have had to > do for OpenBSD.

[sqlite] How difficult for bindings to support API 14 or below?

I notice that the Android bindings support API 16 by default, with a custom branch for API 15. Naively, it looks like API 15 support has a small number of changes relating to a missing/changed(?) exception. Can anyone offer insight into how difficult it would be to support API 14 or below? It

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 4/18/17, Brendan Duddridge wrote: > > I commented out this line of code in the Couchbase lite CBL_SQliteStorage.m > source file and no more corruption > > //int err = sqlite3_config(SQLITE_CONFIG_MMAP_SIZE, > (SInt64)kSQLiteMMapSize, (SInt64)-1); > > > It would be

[sqlite] Magic numbers on sqlite source code

Hello Richard ! Time to time I look at some commits that call my attention like this: [e47fdb49] Leaf: Refactor the fts3ColumnMethod() function so that all branches can be covered. (user: dan

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Deon, BINGO I commented out this line of code in the Couchbase lite CBL_SQliteStorage.m source file and no more corruption //int err = sqlite3_config(SQLITE_CONFIG_MMAP_SIZE, (SInt64)kSQLiteMMapSize, (SInt64)-1); It would be really awesome of this were added to the How to

Re: [sqlite] strange behaviour on sqlite shell output…

If only Hollerith had remembered his lingual heritage... -Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Rowan Worth Gesendet: Dienstag, 18. April 2017 10:41 An: SQLite mailing list

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Yip. Tried that. At some point I had like 6 or 7 debug switches in the app to try all manner of FULLSYNC/WAL/SYNCHRONOUS combinations. At the end it was the MMAP_SIZE that did it. It's not like it was subtle - it's a dead on repro. I was able to repro this by doing a power cycle 2 hours after

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 10:07am, Brendan Duddridge wrote: > >// Enable memory-mapped I/O if available Please try an alternative version of that code which definitely does not use memory mapping in any way. You may be able to do it by changing that IFDEF sequence. Or you

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Deon, That's interesting about the memory mapped IO. I just looked in the source for Couchbase Lite and they do something like that on setup: + (void) firstTimeSetup { // Test the version of the actual SQLite implementation at runtime. Necessary because // the app might be linked

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Simon, I'm using Couchbase Lite actually, not Couchbase server. Couchbase Lite 1.4 to be specific. Couchbase Lite 2.0 is still under development and it does in fact use SQLite under the hood as does Couchbase Lite 1.4. They had worked on using ForestDB, but I think they've abandoned that for

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 9:50am, Deon Brewis wrote: > “From the OSX documentation: > > Note that while fsync() will flush all data from the host to the drive (i.e. > the "permanent storage device"), Deon, I’m not sure this is related, but have you seen

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Richard, I just did another test with my PowerFailureTest app. I launched it and the count of rows in it was 4402. Without even running the function to import my data that writes to the SQLite file, I cut the power (holding down the power button on my MBP). When my Mac finally rebooted, I saw

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 18 Apr 2017, at 6:01am, Brendan Duddridge wrote: > Perhaps the only think I can think of that I'm doing that may be unorthodox > is by storing the SQLite files inside a macOS package. But at the unix > layer that's really just a directory, so I don't know how that could

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Are you by change using memory mapped IO (MMAP_SIZE something other than 0)? This does not work on OSX. Not even remotely. I tracked an issue down in November 2015, and was able to trivially corrupt a database 100% of the time using the repro steps below. This happens long after our app gets

Re: [sqlite] strange behaviour on sqlite shell output…

On 18 April 2017 at 16:28, Hick Gunter wrote: > Richard Hipp wrote: > >I think the OP is referring to a problem that comes up because the field > width and precision of a printf() format are measured in bytes, not > characters, and if the input is multi-byte UTF then it is

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hi Rowan, My apologies. My sample app does use NSDocument. But my production app doesn't. And they both exhibit the same problem. On Tue, Apr 18, 2017 at 12:29 AM, Rowan Worth wrote: > Hey Brendan, > > I'm no OSX expert, but from what I've read this afternoon about NSDocument >

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hey Rowan, In my Mac app I'm actually not using NSDocument. I used to and then I changed my document subclass to extend NSObject instead, just in case NSDocument was doing bad things behind my back. I had also disabled auto-save and versions anyway. For a while though I tried extending

Re: [sqlite] strange behaviour on sqlite shell output…

>On 4/18/17, Hick Gunter wrote: >> I don't see any calls to sqlite3() functions in your "isolated test case". >> Maybe you are having problems with character encoding outside of sqlite? > >I think the OP is referring to a problem that comes up because the field width >and

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

On 4/18/17, Brendan Duddridge wrote: > In both cases though I can cause SQLite file corruption by cutting the > power on my MacBook Pro. Corruption-by-power-loss problems tend to be very sensitive to timing. If you are able to reliably generate the problem by leisurely

Re: [sqlite] strange behaviour on sqlite shell output…

On 4/18/17, Hick Gunter wrote: > I don't see any calls to sqlite3() functions in your "isolated test case". > Maybe you are having problems with character encoding outside of sqlite? I think the OP is referring to a problem that comes up because the field width and precision of

Re: [sqlite] strange behaviour on sqlite shell output…

I don't see any calls to sqlite3() functions in your "isolated test case". Maybe you are having problems with character encoding outside of sqlite? -Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von aotto Gesendet: Montag,

Re: [sqlite] performing an UPDATE on a table-valued function

A TVF is just a "calling convention" for abbreviating a specific SELECT on a virtual table. It implies neither mutability of the returned values, nor persistence beyond the scope of the statement. Indeed the carray() example you give later is an eponymous ephemeral table, i.e. it does not

Re: [sqlite] Database corruption, and PRAGMA fullfsync on macOS

Hey Brendan, I'm no OSX expert, but from what I've read this afternoon about NSDocument and friends combined with what I know about sqlite I have to say you are completely mad to continue passing sqlite databases to NSDocument, *especially* as you don't define your own sub-class to do any of the