Re: [sqlite] EXT : Unknown module FTS4
Hello, On Mon, Jul 30, 2012 at 6:43 PM, Black, Michael (IS)wrote: > You've got me totally confusedyou say "shared library" and > "dynamically linked" but then say it's embedded in the GUI. > > Which is it? > > Are you on Unix/Linux? > > Can you show us your Makefile or an example build line? > What are you compiling with? > > Have you duplicated your GUI build process on another program using all > the same settings and succeeded? > After some investigation, it looks like when building shared library using GCC, it exports all the SQLite functions too. And for some reason at runtime, functions gets resolved to the statically linked version in the GUI. I hope using visibility settings in GCC will fix this problem. I will try it and let you guys know. -- -n ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent)
On Wed, Aug 1, 2012 at 2:43 PM, Black, Michael (IS)wrote: > You may be interested in this article: > http://www.drdobbs.com/parallel/multithreaded-file-io/220300055?pgno=2 > > Mutli-threaded reading of multiple files (which is basically what you're > talking about by splitting a file in half) is only faster if you have > multiple disks (in this article that's a RAID-5 system). Or if you have a network in the way. Think of ql.io (http://ql.io). Think of GDA, a GNOME database API that uses SQLite3 as its client-side engine and virtual tables to access remote databases: since SQLite3 treats each table/index operation as synchronous GDA may not be able to issue remote operations as fast as possible. What would it take to have the core engine do asynchronous operations? First, the backend/virtual table interfaces would have to change to be async-oriented, with functions to start operations and completion notification, and some way to optionally share an event loop with other components. Second the VDBE code (and the supporting VM) would have to have a concept of co-routines, with each co-routine having a scratch memory register space (perhaps for a stack), and a yield-type operation that puts the current co-routine to "sleep" until some completion notification is received and which wakes one co-routine that's ready to run. Worthwhile? Not for disk/memory DBs, no. But between virtual tables (see GDA) and the potential for remote backends, I think the answer is yes. Also, if you look at ql.io, I think you'll really see the value in SQLite being able to do async! Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Documentation bug in result_blob.html
Jesse Weinstein wrote: > > These are not the actual names of the functions. The real names (as > listed at the top of that page) have _error_ in the middle, i.e. > sqlite3_result_error_toobig() and sqlite3_result_error_nomem(). > Thanks. This is now fixed: https://www.sqlite.org/src/info/bec97c9813 -- Joe Mistachkin ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent)
On 1 Aug 2012, at 8:25pm, Udi Karniwrote: > - only 2 processes > - only for simple full scans where the block range can be divided in two > - only when there is no ORDER/GROUP BY where sub results from the 2 threads > have to be combined Premature optimization ? SQLite, just by itself with default compilation and a set of PRAGMAs which suit your requirements, is extremely fast. Write your application first, and only if it turns out to be too slow worry about clever tricks like that. And if it really is too slow, and you want tricks like the above, it's probably better to switch to a different DBMS which will itself speed things like the above up because it does caching and other such tricks. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent)
You may be interested in this article: http://www.drdobbs.com/parallel/multithreaded-file-io/220300055?pgno=2 Mutli-threaded reading of multiple files (which is basically what you're talking about by splitting a file in half) is only faster if you have multiple disks (in this article that's a RAID-5 system). Random I/O gains a bit by threading due to the probability of intersecting common disk blocks. Michael D. Black Senior Scientist Advanced Analytics Directorate Advanced GEOINT Solutions Operating Unit Northrop Grumman Information Systems From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Udi Karni [uka...@gmail.com] Sent: Wednesday, August 01, 2012 2:25 PM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent) You are right. True Parallel Query can get very complicated. I was hoping for something very limited for starters - for example - - only 2 processes - only for simple full scans where the block range can be divided in two - only when there is no ORDER/GROUP BY where sub results from the 2 threads have to be combined Basically only for queries such as SELECT COUNT/MIN/MAX FROM TABLE WHERE Sounds very limited / what's-the-point kind of thing - but it would actually be very useful when working with large data where you find yourself doing a lot of QA and study of the data - "how many rows have this range of codes / are null", etc. Having 2 processes working simultaneously might cut run times in half - and save many minutes. Going higher than 2 might hit disk read limitations anyway - so 2 might be plenty for version 1. In other words - nothing grand - just a small optimization that will kick in on simple stuff. Pick some low hanging fruit. A "would be nice" if not too complicated. On Wed, Aug 1, 2012 at 5:57 PM, Christian Smith < csm...@thewrongchristian.org.uk> wrote: > On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote: > > > > On 14 Jul 2012, at 3:12pm, Udi Karniwrote: > > > > > I know > > > nothing about writing DB engines - so I don't know whether adding a 2nd > > > parallel process adds 10K or 10M to the code base. > > > > You've reached the limit of what I know about parallelization. I hope > someone else can chime in. > > > Using SQLite's VM architecture, I would guess that adding this sort of > parallelization would be non-trival. You need a parallel VM, significantly > different to the current sequential VM, at at least a way of managing > asynchronous IO, with perhaps a callback mechanism into the VM to handle IO > completion. > > While not certain, I guess other databases handle this by using tree based > execution plans, where any single execution node can easily be split into > branches to another thread/process/machine, then merged in the parent tree > node, with each branch handling a certain key range. > > This might make sense, for example, with a partitioned table, where each > partition is on it's own spindle, so a full table scan can be executed in > parallel on each spindle and merged as a final step. So, for a table scan > between k0 and k3, find intermediate keys to split the query between > spindles: > > (k0-k3) > /|\ > / | \ > / | \ >/ | \ > /|\ > (k0-k1] (k1-k2] (k2-k3) >| | | > disk1disk2disk3 > > I sat through an Oracle internals course once, and the instructor gave us > an example of a setup such as this where data was partitioned across 24 > disks, and the resulting full table scans were in fact quicker than index > based scans for the data set they were using. > > Of course, the above would be useless for SQLite anyway, being a single > file database. And even with the likes of Oracle, Stripe And Mirror > Everything (SAME) might also largely defeat parallel scans. > > All in all, the added bloat would be measured in MB, rather than KB. > > Christian > > disclaimer: Not a practical DB implementation expert. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent)
You are right. True Parallel Query can get very complicated. I was hoping for something very limited for starters - for example - - only 2 processes - only for simple full scans where the block range can be divided in two - only when there is no ORDER/GROUP BY where sub results from the 2 threads have to be combined Basically only for queries such as SELECT COUNT/MIN/MAX FROM TABLE WHERE Sounds very limited / what's-the-point kind of thing - but it would actually be very useful when working with large data where you find yourself doing a lot of QA and study of the data - "how many rows have this range of codes / are null", etc. Having 2 processes working simultaneously might cut run times in half - and save many minutes. Going higher than 2 might hit disk read limitations anyway - so 2 might be plenty for version 1. In other words - nothing grand - just a small optimization that will kick in on simple stuff. Pick some low hanging fruit. A "would be nice" if not too complicated. On Wed, Aug 1, 2012 at 5:57 PM, Christian Smith < csm...@thewrongchristian.org.uk> wrote: > On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote: > > > > On 14 Jul 2012, at 3:12pm, Udi Karniwrote: > > > > > I know > > > nothing about writing DB engines - so I don't know whether adding a 2nd > > > parallel process adds 10K or 10M to the code base. > > > > You've reached the limit of what I know about parallelization. I hope > someone else can chime in. > > > Using SQLite's VM architecture, I would guess that adding this sort of > parallelization would be non-trival. You need a parallel VM, significantly > different to the current sequential VM, at at least a way of managing > asynchronous IO, with perhaps a callback mechanism into the VM to handle IO > completion. > > While not certain, I guess other databases handle this by using tree based > execution plans, where any single execution node can easily be split into > branches to another thread/process/machine, then merged in the parent tree > node, with each branch handling a certain key range. > > This might make sense, for example, with a partitioned table, where each > partition is on it's own spindle, so a full table scan can be executed in > parallel on each spindle and merged as a final step. So, for a table scan > between k0 and k3, find intermediate keys to split the query between > spindles: > > (k0-k3) > /|\ > / | \ > / | \ >/ | \ > /|\ > (k0-k1] (k1-k2] (k2-k3) >| | | > disk1disk2disk3 > > I sat through an Oracle internals course once, and the instructor gave us > an example of a setup such as this where data was partitioned across 24 > disks, and the resulting full table scans were in fact quicker than index > based scans for the data set they were using. > > Of course, the above would be useless for SQLite anyway, being a single > file database. And even with the likes of Oracle, Stripe And Mirror > Everything (SAME) might also largely defeat parallel scans. > > All in all, the added bloat would be measured in MB, rather than KB. > > Christian > > disclaimer: Not a practical DB implementation expert. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_STATIC and temporary data
On Wed, Aug 01, 2012 at 04:48:48PM +, Rob Richardson scratched on the wall: > Is "acceptable" good enough? I admit I haven't played with this function > (actually, I never heard of it until today), but from what I read in the > documentation, the case described looked dangerous to me. It's somewhat undefined, and I suspect that's intentional. There are a lot of specifics that are undocumented because they change from version to version of SQLite. > SQLITE_STATIC > seemed to me to imply that the contents of the memory used by the sqlite > statement would never change over the life of the statement. It's more the lifetime of the binding. Which reminds me (and I see others on the list already pointed this out), freeing the memory after a call to _reset() would be a Bad Idea since the binding is still in effect. It would have to be _finalize() or _clear_bindings(). But yes... the key is that the memory remains valid for the lifetime of the binding, not the fact that is or isn't statically allocated. -j > -Original Message- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Jay A. Kreibich > Sent: Wednesday, August 01, 2012 12:16 PM > To: General Discussion of SQLite Database > Cc: c...@iki.fi > Subject: Re: [sqlite] SQLITE_STATIC and temporary data > > On Wed, Aug 01, 2012 at 08:49:19PM +1000, Yose Widjaja scratched on the wall: > > Dear Friends, > > > > So SQLITE_STATIC is meant to be used for data that is static. However, > > would it still be safe when it is used with data that expires after > > the > > sqlite3_step() function? > > > > For example: > > > > string hello = "hello world"; > > > > sqlite3_bind(statement, 1, hello.c_str(), hello.size(), > > SQLITE_STATIC); > > > > sqlite3_step(statement); > > > > hello = "moo"; > > > > Would there be anything that can potentially go wrong? I mean, since > > SQLITE_STATIC is meant to imply static stuff, would sqlite cache the > > value in such a way that subsequence SELECT_ statements actually use > > this static value that was passed in through the bind function? > > It would be better to keep the value valid until _reset() or > _finalize() is called, but, yes... this use of SQLITE_STATIC is > acceptable (and somewhat common). > >-j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_STATIC and temporary data
Jay A. Kreibich wrote: > On Wed, Aug 01, 2012 at 08:49:19PM +1000, Yose Widjaja scratched on the wall: >> Dear Friends, >> >> So SQLITE_STATIC is meant to be used for data that is static. However, >> would it still be safe when it is used with data that expires after the >> sqlite3_step() function? >> >> For example: >> >> string hello = "hello world"; >> >> sqlite3_bind(statement, 1, hello.c_str(), hello.size(), SQLITE_STATIC); >> >> sqlite3_step(statement); >> hello = "moo"; >> >> Would there be anything that can potentially go wrong? I mean, since >> SQLITE_STATIC is meant to imply static stuff, would sqlite cache the value >> in such a way that subsequence SELECT_ statements actually use this static >> value that was passed in through the bind function? > > It would be better to keep the value valid until _reset() or No, _reset is NOT enough, as it *won't* clear bindings. You *must* keep data till either sqlite3_clear_bindings or sqlite3_finalize is (successfully) called. > _finalize() is called, but, yes... this use of SQLITE_STATIC is > acceptable (and somewhat common). Above code is certainly incorrect. Maybe you won't be slapped with SIGSEGV immediately (depending on surrounding code and sqlite3 implementation details), but still this is incorrect and unsafe. You should insert sqlite3_clear_bindings(statement); or sqlite3_bind_null(statement, 1); before hello = "moo"; ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_STATIC and temporary data
Is "acceptable" good enough? I admit I haven't played with this function (actually, I never heard of it until today), but from what I read in the documentation, the case described looked dangerous to me. SQLITE_STATIC seemed to me to imply that the contents of the memory used by the sqlite statement would never change over the life of the statement. But please keep in mind that in this case (as in many other cases), my opinion likely to be worth exactly what you have paid for it. RobR -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Jay A. Kreibich Sent: Wednesday, August 01, 2012 12:16 PM To: General Discussion of SQLite Database Cc: c...@iki.fi Subject: Re: [sqlite] SQLITE_STATIC and temporary data On Wed, Aug 01, 2012 at 08:49:19PM +1000, Yose Widjaja scratched on the wall: > Dear Friends, > > So SQLITE_STATIC is meant to be used for data that is static. However, > would it still be safe when it is used with data that expires after > the > sqlite3_step() function? > > For example: > > string hello = "hello world"; > > sqlite3_bind(statement, 1, hello.c_str(), hello.size(), > SQLITE_STATIC); > > sqlite3_step(statement); > > hello = "moo"; > > Would there be anything that can potentially go wrong? I mean, since > SQLITE_STATIC is meant to imply static stuff, would sqlite cache the > value in such a way that subsequence SELECT_ statements actually use > this static value that was passed in through the bind function? It would be better to keep the value valid until _reset() or _finalize() is called, but, yes... this use of SQLITE_STATIC is acceptable (and somewhat common). -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLITE_STATIC and temporary data
On Wed, Aug 01, 2012 at 08:49:19PM +1000, Yose Widjaja scratched on the wall: > Dear Friends, > > So SQLITE_STATIC is meant to be used for data that is static. However, > would it still be safe when it is used with data that expires after the > sqlite3_step() function? > > For example: > > string hello = "hello world"; > > sqlite3_bind(statement, 1, hello.c_str(), hello.size(), SQLITE_STATIC); > > sqlite3_step(statement); > > hello = "moo"; > > Would there be anything that can potentially go wrong? I mean, since > SQLITE_STATIC is meant to imply static stuff, would sqlite cache the value > in such a way that subsequence SELECT_ statements actually use this static > value that was passed in through the bind function? It would be better to keep the value valid until _reset() or _finalize() is called, but, yes... this use of SQLITE_STATIC is acceptable (and somewhat common). -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Documentation bug in result_blob.html
In http://www.sqlite.org/c3ref/result_blob.html it says The sqlite3_result_toobig() interface causes SQLite to throw an error indicating that a string or BLOB is too long to represent. The sqlite3_result_nomem() interface causes SQLite to throw an error indicating that a memory allocation failed. These are not the actual names of the functions. The real names (as listed at the top of that page) have _error_ in the middle, i.e. sqlite3_result_error_toobig() and sqlite3_result_error_nomem(). Thanks for a very useful library (with generally excellent documentation)! Jesse Weinstein Software Engineer CliniComp, Intl. 9655 Towne Centre Drive San Diego, CA 92121 +1 [858] 546 8202 jesse.weinst...@clinicomp.com www.clinicomp.com This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you are not the intended recipient, please notify the sender immediately by reply email and delete this message. Any other use by you of this email or the information contained therein is strictly prohibited. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLITE_STATIC and temporary data
Dear Friends, So SQLITE_STATIC is meant to be used for data that is static. However, would it still be safe when it is used with data that expires after the sqlite3_step() function? For example: string hello = "hello world"; sqlite3_bind(statement, 1, hello.c_str(), hello.size(), SQLITE_STATIC); sqlite3_step(statement); hello = "moo"; Would there be anything that can potentially go wrong? I mean, since SQLITE_STATIC is meant to imply static stuff, would sqlite cache the value in such a way that subsequence SELECT_ statements actually use this static value that was passed in through the bind function? Cheers ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows I/O (was: Initial read speed greater than subsequent)
On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote: > > On 14 Jul 2012, at 3:12pm, Udi Karniwrote: > > > I know > > nothing about writing DB engines - so I don't know whether adding a 2nd > > parallel process adds 10K or 10M to the code base. > > You've reached the limit of what I know about parallelization. I hope > someone else can chime in. Using SQLite's VM architecture, I would guess that adding this sort of parallelization would be non-trival. You need a parallel VM, significantly different to the current sequential VM, at at least a way of managing asynchronous IO, with perhaps a callback mechanism into the VM to handle IO completion. While not certain, I guess other databases handle this by using tree based execution plans, where any single execution node can easily be split into branches to another thread/process/machine, then merged in the parent tree node, with each branch handling a certain key range. This might make sense, for example, with a partitioned table, where each partition is on it's own spindle, so a full table scan can be executed in parallel on each spindle and merged as a final step. So, for a table scan between k0 and k3, find intermediate keys to split the query between spindles: (k0-k3) /|\ / | \ / | \ / | \ /|\ (k0-k1] (k1-k2] (k2-k3) | | | disk1disk2disk3 I sat through an Oracle internals course once, and the instructor gave us an example of a setup such as this where data was partitioned across 24 disks, and the resulting full table scans were in fact quicker than index based scans for the data set they were using. Of course, the above would be useless for SQLite anyway, being a single file database. And even with the likes of Oracle, Stripe And Mirror Everything (SAME) might also largely defeat parallel scans. All in all, the added bloat would be measured in MB, rather than KB. Christian disclaimer: Not a practical DB implementation expert. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite3_stat problem
Return values are your friends. Use them. Store the error code from every sqlite function call, and if the error code is not SQLITE_OK (NOTE: Check that that is the correct name.), then display what the error code is. In particular, what is the return value of your sqlite3_bind_text() function call? If the documentation of a function states that a parameter requires a named special value, use that name. The fifth argument should be a destructor function pointer, SQLITE_STATIC or SQLITE_TRANSIENT. Read the documentation to understand what each one means. Learn to cringe every time you put a constant value into your code. People often call hard-coded constants "magic numbers" because they apparently showed up magically, since there's no other explanation of where they came from. Where did "140" come from? Why do you need it? The parameter requires the length of a string. So use strlen() (again, check the correctness. It's been so long since I've used these functions, the name could be wrong) to calculate the actual length of the string. The documentation states that if the fourth argument is not negative, it is the byte offset where the null terminator should be, and any null terminators before that are included in the bound string, and if any null terminators are included in the bound string, behavior is undefined. That's exactly the situation you have. And when you use strlen() or whatever you use to calculate the length of the string, be careful to take into account the difference between bytes and characters. The sqlite3_bind_text() call requires bytes, but your characters may be one or two bytes. RobR ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users