Re: [sqlite] Truncation of floating point numbers in SQLite?
Trying to retrieve a stored qNaN or sNaN returns a column type of NULL and a value of 0. On Thu Jan 29 2015 at 8:56:35 PM RSmithwrote: > > On 2015/01/29 05:05, James K. Lowden wrote: > > There's no reason to think, if the data are provided in binary form, > that they won't be returned in the identical form absent an > > explicit conversion. If that's not so, I'd sure like to know why. I'm > faintly surprised NaNs can't be stored, too. Why should > > SQLlite interpret them if they're bound to a double? > > Indeed, which is what all the posts have been saying more or less in terms > of round-tripping all but NaNs. > > In the case of NaN though, there are two defined NaNs, namely qNaN and > sNan which both means the same but the sNaN will cause an > exception even at hardware level by merely passing through any register, > which I believe is its intended purpose. This means that in > order for software to work correctly, it should never let an sNaN pass > through untouched, it should produce an error so all parties > are savvy to the wrongness that just occured (Even if the software in use > is not specifically checking for NaN, an sNaN should still > cause an exception from lower down). > > qNaN should pass through and round-trip same as any other float. I'm not > sure how SQLite handles either of these NaNs, but am now > quite interested to know. > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Truncation of floating point numbers in SQLite?
On 1/29/15, Donald Shepherdwrote: > I'm still not convinced whether it's the behaviour causing my problem, but > it does look like negative zero is another special case: > > SQLite version 3.8.7.2 2014-11-18 20:57:56 > Enter ".help" for usage hints. > sqlite> create table datatable2 (doublevalue real); > sqlite> insert into datatable2 values(-0.0); > sqlite> select * from datatable2; > 0.0 > I suppose so. SQLite converts integer floating point values to actual integers for storage (because that takes up less space on disk) and then converts back to double upon retrieval. That round-trip would change -0.0 into +0.0. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Truncation of floating point numbers in SQLite?
I'm still not convinced whether it's the behaviour causing my problem, but it does look like negative zero is another special case: SQLite version 3.8.7.2 2014-11-18 20:57:56 Enter ".help" for usage hints. sqlite> create table datatable2 (doublevalue real); sqlite> insert into datatable2 values(-0.0); sqlite> select * from datatable2; 0.0 When inserting it through the API I get the same results, i.e. the signed bit looks like it gets stripped so technically the value retrieved is not the bitwise equivalent of what's written in even though it is the logical equivalent (-0.0 == 0.0). On Thu Jan 29 2015 at 10:13:55 AM Donald Shepherdwrote: > Thanks for the reassurances. I have a case where differences in doubles > would explain what I'm seeing but I have no evidence that it is the case > (evidence compilation is still underway), hence my attempt to plumb the > depths of the list's knowledge to see if there was any known edge cases to > be aware of (other than the NaN one I'd already run into). :) > > On Thu Jan 29 2015 at 10:10:35 AM Simon Slavin > wrote: > >> >> On 28 Jan 2015, at 10:47pm, Donald Shepherd >> wrote: >> >> > This is a bit of a speculative question related to a problem I'm having >> - >> > are there legal values of a C++ double that would get truncated when >> > written into and read from an SQLite database? >> >> In theory there should be no problem here. >> >> C doubles have 15 to 16 digits of precision. >> >> In SQLite databases, numbers which can't be stored as integers are stored >> as IEEE 754-2008 64-bit floating point numbers, sometimes known as >> 'binary64'. These give 15 to 17 digits of precision. >> >> My understanding is that it is possible to store every distinct C double >> value as a distinct binary64 value. >> >> If it's the conversion that worries you, you can read the SQLite source >> code to find the programming used to encode and decode numbers into this >> format. I am not competent to read that source code and tell you >> definitely that it works for all C double values. >> >> Simon. >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Best Practice for read-only access
On 1/29/15, Duquette, William H (393K)wrote: > Howdy! > > I've got an object that encapsulates access to an SQLite database, i.e., all > writes to the database are done in terms of method calls to the object. > However, I want to give the application read-only access to the database for > queries. There are two obvious ways to do this: > > 1. I can define an "authorizer", and have it deny write access for queries > coming from outside the object. > > 2. I can open two database handles on the one file, one of them read-only, > and give the outside application access to the read-only database handle. > > At present the application is single-threaded, so simultaneous access isn't > an issue (and I'm using WAL mode anyway). > > I'm currently using #1; I enable the authorizer before queries from outside, > and remove it afterward, each time. > > I kind of like #2--it's simpler--but I'm worried that it would increase > memory usage considerably. > I vote for #2. Measure the memory usage if that is a concern. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Partial indexes not working for me
Filip, I don't suppose it would fit your needs to index on the column you're comparing rather than on "id" would it? That would cause the query planner to use your indexes, I believe. create index "i1" on "t" (uniqueID) where UniqueId ==55; Alternatively, you can create an "unnecessary" compound index such as create index "i1" on "t" (uniqueID, id) where UniqueId ==55; Even though the documentation seems to explicitly allow creating partial indexes on columns other than the indexed column: "The columns referenced in the WHERE clause of a partial index can be any of the columns in the table, not just columns that happen to be indexed. " (It may well be that I'm missing something myself.) I tried experimenting with 3.8.8, removing references to NULL and using "== 55" as a simple test condition; results follow. Your problem did not go away when I got rid of NULL checking, but creation of a partial index on column "id" which tests instead on column "uniqueID" DID cause the index to go unused. (Whether sqlite is working properly or not -- I'll leave that for others to say.) sqlite> sqlite> /* Test comparing UniqueID to 55 rather than to null */ sqlite> /* Note that index is on column "id" and not "uniqueID" */ sqlite> /* EXPLAIN Q. Plan shows partial index would not be invoked*/ sqlite> drop table if exists t; sqlite> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, ...> "uniqueId", "syncFolder" INTEGER); sqlite> ANALYZE; sqlite> create index "i1" on "t" (uniqueID, id) where UniqueId ==55; sqlite> ANALYZE; sqlite> explain query plan select * from "t" where UniqueID ==55; 0|0|0|SEARCH TABLE t USING INDEX i1 (uniqueId=?) sqlite> sqlite> sqlite> sqlite> /* Try to see if it fails only on INTEGER PRIMARY KEY.*/ sqlite> /*Don't use variable "id" at all, use "flags" instead.*/ sqlite> /* Query planner would still not use index */ sqlite> drop table if exists t; sqlite> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, ...> "uniqueId", "syncFolder" INTEGER); sqlite> sqlite> ANALYZE; sqlite> create index "i1" on "t" (flags) where UniqueId ==55; sqlite> ANALYZE; sqlite> explain query plan select * from "t" where UniqueID ==55; 0|0|0|SCAN TABLE t sqlite> sqlite> sqlite> sqlite> sqlite> /* Try making index compound, with "id" secondary*/ sqlite> /* This DOES provoke the query planner into using the index*/ sqlite> /* (at the expense of some efficiency, I suppose)*/ sqlite> drop table if exists t; sqlite> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, ...> "uniqueId", "syncFolder" INTEGER); sqlite> ANALYZE; sqlite> create index "i1" on "t" (uniqueID, id) where UniqueId ==55; sqlite> ANALYZE; sqlite> explain query plan select * from "t" where UniqueID ==55; 0|0|0|SEARCH TABLE t USING INDEX i1 (uniqueId=?) sqlite> sqlite> sqlite> /* Of course it also works if you don't index on column "id" */ sqlite> drop table if exists t; sqlite> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, ...> "uniqueId", "syncFolder" INTEGER); sqlite> sqlite> create index "i1" on "t" (uniqueId) where UniqueId ==55; sqlite> explain query plan select * from "t" where UniqueID ==55; 0|0|0|SEARCH TABLE t USING INDEX i1 (uniqueId=?) sqlite> sqlite> select sqlite_version(); 3.8.8 sqlite> pragma compile_options; ENABLE_FTS3 ENABLE_RTREE SYSTEM_MALLOC THREADSAFE=0 sqlite> *I used windoze pre-compiled exe withOUT STAT4, btw.*
Re: [sqlite] "database disk image is malformed" error occurs more
On 29 Jan 2015, at 7:04pm, Mario M. Westphalwrote: > The diagnosis log of my application reports the output of integrity_check() > already. > > I retrieved the log from the most recent error report. This is my application > has logged: > > '*** IN DATABASE MAIN *** > ON TREE PAGE 385120 CELL 24: INVALID PAGE NUMBER 151192068 > CORRUPTION DETECTED IN CELL 24 ON PAGE 385120 > CORRUPTION DETECTED IN CELL 25 ON PAGE 385120 > MULTIPLE USES FOR BYTE 1612 OF PAGE 385120 > FRAGMENTATION OF 30 BYTES REPORTED AS 0 ON PAGE 385120' Okay. First, stop doing VACUUM after this. You're not improving things and you may be making things worse. Second, a corrupt database may remain corrupt. So we try to distinguish between (A) and (B): A) Something corrupted my database but that was just once and it has never happened again B) Something is continually corrupting my database. So have you tried replacing that database with one which isn't corrupt and seeing whether the new 'clean' one somehow becomes corrupt ? Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] "database disk image is malformed" error occurs more
On Thu, Jan 29, 2015 at 2:07 PM, Mario M. Westphalwrote: > Most database damaged errors encountered over time could be pinned to > power failures, disk or *network problems*. > > Network problems? I might have missed a good chunk of this thread, but, this begs to be asked Are you running a client/server model in which the server is the ONLY machine accessing the database file, or, do you have multiple machines touching the file via a network share? If you're running multiple machines talking via a network interface directly to the database, you need to stop, ESPECIALLY with the up in frequency you seem to be running into this problem. I did note you did read the "How To Corrupt" page, but you may have missed the whole networking thing that shouldn't be done. If you're running client/server in that a client opens a custom network protocol to a server application, and the server application touches the database BY ITSELF, then you need to look at what the hardware is doing between the application and the storage device. I can't say for certain, and maybe Dr Hipp and others will need to get involved in looking at the low level SQLite code base, but if YOUR code base code is from 2008, and it is now 2015, and you've got applications talking with a single source (Meaning one customer = one source of their own data) with different versions of the SQLite code, *MAYBE* you're looking at an older and bugged version of SQLite that is doing one thing to the raw data while a working version comes back and informs you with the "WTF?" errors. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Partial indexes not working for me
Actually running ANALYZE didn't seem to help. There are other partial indexes I tried and none of them were used: sqlite> create index "i2" on "t" ("id") where "flags" & 1; sqlite> explain query plan select * from "t" where "flags" & 1; 0|0|0|SCAN TABLE t sqlite> create index "i3" on "t" ("id") where "syncFolder" <> 0; sqlite> explain query plan select * from "t" where "syncFolder" <> 0; 0|0|0|SCAN TABLE t It is an oversimplification of my actual database, where all of these queries are used together in one condition ("flags" & 1) AND ("uniqueId" IS NULL OR "syncFolder" <> 0) that I was hoping to cover with a partial index. Currently I use a bunch of triggers to basically create the index myself, but I was hoping to replace it with the partial indexes and avoid having the complex triggers. The table "t" usually has thousands to millions of rows, while the index itself should cover only few rows in most cases, typically none. Best regards, Filip Navara On Thu, Jan 29, 2015 at 9:29 PM, Richard Hippwrote: > On 1/29/15, Filip Navara wrote: > > Hello, > > > > I tried really hard to get partial indexes working, but SQLite refuses to > > use them: > > > >> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, > > "uniqueId", "syncFolder" INTEGER); > >> create index "i1" on "t" ("id") where "uniqueId" IS NULL; > >> explain query plan select * from "t" where "uniqueId" IS NULL; > > 0|0|0|SCAN TABLE t > >> explain query plan select * from "t" indexed by "i1" where "uniqueId" IS > > NULL; > > Error: no query solution > > > > Any advice what am I doing wrong? > > > > I seem to recall adding a rule to the query planner that refuses to > use an IS NULL constraint with an index unless you have first run > ANALYZE. It might also require compiling with SQLITE_ENABLE_STAT4. > > The usual case with partial indexes is WHERE field IS NOT NULL --- > with a "NOT". You are taking partial indexes into an area for which > they were not optimized. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Partial indexes not working for me
On 1/29/15, Filip Navarawrote: > Hello, > > I tried really hard to get partial indexes working, but SQLite refuses to > use them: > >> create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, > "uniqueId", "syncFolder" INTEGER); >> create index "i1" on "t" ("id") where "uniqueId" IS NULL; >> explain query plan select * from "t" where "uniqueId" IS NULL; > 0|0|0|SCAN TABLE t >> explain query plan select * from "t" indexed by "i1" where "uniqueId" IS > NULL; > Error: no query solution > > Any advice what am I doing wrong? > I seem to recall adding a rule to the query planner that refuses to use an IS NULL constraint with an index unless you have first run ANALYZE. It might also require compiling with SQLITE_ENABLE_STAT4. The usual case with partial indexes is WHERE field IS NOT NULL --- with a "NOT". You are taking partial indexes into an area for which they were not optimized. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Partial indexes not working for me
Hello, I tried really hard to get partial indexes working, but SQLite refuses to use them: > create table "t" ("id" INTEGER NOT NULL PRIMARY KEY, "flags" INTEGER, "uniqueId", "syncFolder" INTEGER); > create index "i1" on "t" ("id") where "uniqueId" IS NULL; > explain query plan select * from "t" where "uniqueId" IS NULL; 0|0|0|SCAN TABLE t > explain query plan select * from "t" indexed by "i1" where "uniqueId" IS NULL; Error: no query solution Any advice what am I doing wrong? Thanks, Filip ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] "database disk image is malformed" error occurs more
The core code is in place since about 2008. I took advantage of changes in SQLite over time, from using the shared cache to switching to WAL mode for databases which are not opened in read-only mode. These changes were made between 12 and six months ago, and tested during beta tests and also in the wild. Most database damaged errors encountered over time could be pinned to power failures, disk or network problems. But a too high number of recent reports (couple of months) could not be linked to any hardware problem or power failure. My application uses multiple concurrent threads, but each thread works with its own instance of SQLite (on the same database). Transactions are used to improve performance and for control flow. Every error returned by SQLite is logged to the log file. If a SQLite function returns the dreaded “disk image malformed” error, my application immediately stores that error and remembers it – the database is marked as defective and the user is notified as soon as possible. My users run daily backups of all their important data, including the database so usually they can roll-back to the last known working backup and continue. I will implement Richard’s suggestions to gather more info to the log file. The next time a user reports the problem, we may get extra hints about why and when this happened. Thanks for the great support and advice. -- Mario ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Best Practice for read-only access
Howdy! I've got an object that encapsulates access to an SQLite database, i.e., all writes to the database are done in terms of method calls to the object. However, I want to give the application read-only access to the database for queries. There are two obvious ways to do this: 1. I can define an "authorizer", and have it deny write access for queries coming from outside the object. 2. I can open two database handles on the one file, one of them read-only, and give the outside application access to the read-only database handle. At present the application is single-threaded, so simultaneous access isn't an issue (and I'm using WAL mode anyway). I'm currently using #1; I enable the authorizer before queries from outside, and remove it afterward, each time. I kind of like #2--it's simpler--but I'm worried that it would increase memory usage considerably. Thoughts? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] "database disk image is malformed" error occurs more
The diagnosis log of my application reports the output of integrity_check() already. I retrieved the log from the most recent error report. This is my application has logged: '*** IN DATABASE MAIN *** ON TREE PAGE 385120 CELL 24: INVALID PAGE NUMBER 151192068 CORRUPTION DETECTED IN CELL 24 ON PAGE 385120 CORRUPTION DETECTED IN CELL 25 ON PAGE 385120 MULTIPLE USES FOR BYTE 1612 OF PAGE 385120 FRAGMENTATION OF 30 BYTES REPORTED AS 0 ON PAGE 385120' ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] "database disk image is malformed" error occurs more
My application does not phone home :-/ but I can add output of these functions to the log file my application maintains. My users know how to collect these log files and send them to me. I will also add the error logging callback to my wrapper class and route it to the log file. This should give additional information in case these errors repeat. I will do that right away and ship this with the next update. Will take a couple of weeks to saturate the user basis. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Questions about SQLITE_CONFIG_SCRATCH and SQLITE_CONFIG_PAGECACHE
Hi 1) Question about SQLITE_CONFIG_SCRATCH In SQLite documentation about SQLITE_CONFIG_SCRATCH, I read: === BEGIN QUOTE https://sqlite.org/c3ref/c_config_getmalloc.html === SQLite will never require a scratch buffer that is more than 6 times the database page size. If SQLite needs needs additional scratch memory beyond what is provided by this configuration option, then sqlite3_malloc() will be used to obtain the memory needed. === END QUOTE === I stumbled upon code where the scratch buffer size is configured to only 10 KB only, yet some DB have 16KB page sizes: const int KSize = 10*1024; const int KBufferCount = 8; static uint64_t sqliteScratchBuffer[KSize*KBuferSize/sizeof(uint64_t)]; status = sqlite3_config( SQLITE_CONFIG_SCRATCH, [0], KSize, KBufferCount); Is it safe to have only 10KB of scratch buffer when DB page size can be 16KB? Is it ideal? I don't find guidelines about configuring SQLITE_CONFIG_SCRATCH. What happens if the scratch buffer was bigger than 6*page size? Would memory just be wasted? (since doc says it never allocate more than 6*page size). 2) Question about SQLITE_CONFIG_PAGECACHE In order to reduce the number of malloc calls, I consider configuring SQLITE_CONFIG_PAGECACHE with a static buffer. However, the application opens multiple databases with various page sizes (1KB, 4MB, 8MB, 16MB). So what happens if if do for example: // Max DB page size is 16KB. SQLite doc says to add 40 bytes for page header. const int KPageSize = 16*1024 + 40; const int KPageCount= 512; static uint64_t sqlitePageCache[KPageSize*KPageCount/sizeof(uint64_t)]; status = sqlite3_config( SQLITE_CONFIG_PAGECACHE, [0], KPageSize, KPageCount); Will SQLite use 16KB (=KPageSize) in that buffer to store each page of DBs even for the DBs where page size is only 1KB or 4KB or 8KB? If so, it will waste memory for 1KB, 4KB or 8KB pages and SQLITE_CONFIG_PAGECACHE does not look like a good idea in such case. Regards Dominique ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Truncation of floating point numbers in SQLite?
>I wonder what happens if you put SQLite on a computer with no native IEEE >maths library. Same as compiling with SQLITE_OMIT_FLOATING_POINT on a computer/compiler that *does* have floating point I should imagine -- you end up with a version of SQLite with all floating point omitted. --- Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. Sometimes theory and practice are combined: nothing works and no one knows why. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Truncation of floating point numbers in SQLite?
On 2015/01/29 05:05, James K. Lowden wrote: There's no reason to think, if the data are provided in binary form, that they won't be returned in the identical form absent an explicit conversion. If that's not so, I'd sure like to know why. I'm faintly surprised NaNs can't be stored, too. Why should SQLlite interpret them if they're bound to a double? Indeed, which is what all the posts have been saying more or less in terms of round-tripping all but NaNs. In the case of NaN though, there are two defined NaNs, namely qNaN and sNan which both means the same but the sNaN will cause an exception even at hardware level by merely passing through any register, which I believe is its intended purpose. This means that in order for software to work correctly, it should never let an sNaN pass through untouched, it should produce an error so all parties are savvy to the wrongness that just occured (Even if the software in use is not specifically checking for NaN, an sNaN should still cause an exception from lower down). qNaN should pass through and round-trip same as any other float. I'm not sure how SQLite handles either of these NaNs, but am now quite interested to know. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users