Re: [sqlite] Read only scaling optimization
On 11 Aug 2011, at 9:51pm, Drew Kozicki wrote: > Unfortunately reading from disk appears not to be the problem. Even at 32 > threads the IO appears to be very minimal. Our inability to scale appears to > be caused by a mutex in the caching. My CPU usage is at 30% and my disk is > near silent watching the performance monitor. This is on a 10 GB database > executing 32 different queries against 7 different tables. Purely as a method of research, having done some time trials and measured the results, as you obviously have done, a) Make a second copy of the data on the same disk, have half look at one copy, half look at the other copy, and see what that does to the times. b) If you have another shared physical volume available, do it again but this time put the second copy on the other hard disk. One tells you about contention for one file. The other tells you about contention for access to a single hard disk. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Read only scaling optimization
I'll try this. Unfortunately reading from disk appears not to be the problem. Even at 32 threads the IO appears to be very minimal. Our inability to scale appears to be caused by a mutex in the caching. My CPU usage is at 30% and my disk is near silent watching the performance monitor. This is on a 10 GB database executing 32 different queries against 7 different tables. > You > may try increasing page size - bigger block means less near-random > reads from the disc. It's good way. With page size 8k instead of default 1k selects performance may increasing ~3x. Note: PostgreSQL use 8k disk pages. Thank you all for your responses. On Wed, Aug 10, 2011 at 9:23 AM, Drew Kozickiwrote: > To Answer several questions at once. > > Simon, > Just checking: by 'queries' you mean 'SELECT', right ? You're not making > changes, just searching > > Yes to optimize we average about 5-6 indexes per table. > > D. Richard Hipp, > Open a separate database connection for each thread. Don't try to use the > same database connection on all threads because access to the database > connection is serialized. > > I'll look into this. Thank you > > Teg, > Why multiple threads? What kind of performance do you get if you only > use a single thread? > > Is it one thread per database perhaps? > > This program is ran on massive servers and the people that use it are > talking of running 100's of millions of records through and we're trying to > let them scale so that they can benefit from the new servers. We seem to > have peeked out single thread performance at aprox. 2-10 million > records/hour. > > Thank you once again in advance, > Drew > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Fri, Aug 12, 2011 at 12:28 AM, Richard Hippwrote: > On Thu, Aug 11, 2011 at 2:48 PM, Abhinav Upadhyay < > er.abhinav.upadh...@gmail.com> wrote: > >> On Fri, Aug 12, 2011 at 12:05 AM, Michael Stephenson >> wrote: >> > If you use INTEGER PRIMARY KEY, that column becomes your rowids; this >> does >> > not create a new, separate column in addition to the rowid column. >> Indeed, but the INTEGER PRIMARY KEY column would count as a user >> defined column and thus affect the FTS search :) The FTS table has all >> text data, so I really do need to create a separate column for the >> IDs. >> > > Every FTS table has a "docid" column that is not searched, that is a unique > integer key (like rowid), and which is not modified by VACUUM. Ah, that saves the day for me :-) . Using VACUUM brings down the size of my databse by 1/3rd, so I really wanted to use it. Thanks :) Abhinav ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Thu, Aug 11, 2011 at 2:48 PM, Abhinav Upadhyay < er.abhinav.upadh...@gmail.com> wrote: > On Fri, Aug 12, 2011 at 12:05 AM, Michael Stephenson >wrote: > > If you use INTEGER PRIMARY KEY, that column becomes your rowids; this > does > > not create a new, separate column in addition to the rowid column. > Indeed, but the INTEGER PRIMARY KEY column would count as a user > defined column and thus affect the FTS search :) The FTS table has all > text data, so I really do need to create a separate column for the > IDs. > Every FTS table has a "docid" column that is not searched, that is a unique integer key (like rowid), and which is not modified by VACUUM. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Fri, Aug 12, 2011 at 12:05 AM, Michael Stephensonwrote: > If you use INTEGER PRIMARY KEY, that column becomes your rowids; this does > not create a new, separate column in addition to the rowid column. Indeed, but the INTEGER PRIMARY KEY column would count as a user defined column and thus affect the FTS search :) The FTS table has all text data, so I really do need to create a separate column for the IDs. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
If you use INTEGER PRIMARY KEY, that column becomes your rowids; this does not create a new, separate column in addition to the rowid column. -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Abhinav Upadhyay Sent: Thursday, August 11, 2011 2:06 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Use of VACUUM On Thu, Aug 11, 2011 at 11:15 PM, Simon Slavinwrote: > > On 11 Aug 2011, at 6:35pm, Abhinav Upadhyay wrote: > >> The documentation page of the VACUUM command says that "The VACUUM >> command may change the ROWIDs of entries in any tables that do not >> have an explicit INTEGER PRIMARY KEY." So what are the possible cases >> in which the ROWIDs might change ? > > Not documented. So even if someone told you what they were in this version of SQLite, there might be other reasons in the next version. Theoretically it might renumber rows to close up gaps in the AUTOINCREMENT. That makes sense. > As it says, to stop it all you need to do is declare one of the columns as INTEGER PRIMARY KEY. Once you do that it assumes that you might be referring to those values elsewhere and won't change them. Indeed, I am using the ROWID as a reference in another table. Actually I have an FTS table and I don't really want to create an explicit column for storing the IDs, as I am afraid that matches from the ID column could affect the quality of search results. But if this is the only option, then I guess I need to give it a try. I might give this column a weight of 0.0 so that it doesn't create noise in the search results. Thanks Abhinav ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Thu, Aug 11, 2011 at 11:14 PM, Igor Tandetnikwrote: > On 8/11/2011 1:35 PM, Abhinav Upadhyay wrote: >> The documentation page of the VACUUM command says that "The VACUUM >> command may change the ROWIDs of entries in any tables that do not >> have an explicit INTEGER PRIMARY KEY." So what are the possible cases >> in which the ROWIDs might change ? > > ROWIDs might possibly change if the table doesn't have an explicit > INTEGER PRIMARY KEY column, and you run VACUUM commad on the database > containing this table. Which part of the statement you quoted do you > find unclear? I wanted to know, why would the ROWID change, but Simon's answer makes sense. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fwd: Case Sensitive FTS searches.
On Thu, Aug 11, 2011 at 8:57 AM, Dennis Suehrwrote: > After some digging through the sqlite3 source code, I came across the code > for the ICU tokenizer. After enabling that and then commenting out the one > line where u_foldCase() is called, i.e. icuOpen(), I retested and > case-sensitive searching now seems to work as expected for FTS. > > I then tried doing the same thing in icuLikeCompare() by commenting out > both u_foldCase() calls, i.e. for the string and the pattern and seem to > have implemented case-sensitive LIKE searching as well for non FTS tables. > > Can anyone see anything wrong with this approach? I still plan to implement > and register my own tokenizer, but will do it as high-lighted above. I think that's generally the right direction to take. > Finally, if this is a valid approach, then can I suggest that an additional > preprocessor macro be defined which would allow this behaviour to be enabled > for the general SQLite release code. This is probably a pretty low-volume use case, and once you're compiling anyhow you're probably better off just adding a new tokenizer. That way you won't have an issue where someone expects a particular tokenizer to be case-insensitive, but it's actually case sensitive. Better might be to parameterize the tokenizer. This could be something like allowing this: CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple(NOCASE)); CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple(BINARY)); CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple); -- defaults to NOCASE -scott ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Thu, Aug 11, 2011 at 11:15 PM, Simon Slavinwrote: > > On 11 Aug 2011, at 6:35pm, Abhinav Upadhyay wrote: > >> The documentation page of the VACUUM command says that "The VACUUM >> command may change the ROWIDs of entries in any tables that do not >> have an explicit INTEGER PRIMARY KEY." So what are the possible cases >> in which the ROWIDs might change ? > > Not documented. So even if someone told you what they were in this version > of SQLite, there might be other reasons in the next version. Theoretically > it might renumber rows to close up gaps in the AUTOINCREMENT. That makes sense. > As it says, to stop it all you need to do is declare one of the columns as > INTEGER PRIMARY KEY. Once you do that it assumes that you might be referring > to those values elsewhere and won't change them. Indeed, I am using the ROWID as a reference in another table. Actually I have an FTS table and I don't really want to create an explicit column for storing the IDs, as I am afraid that matches from the ID column could affect the quality of search results. But if this is the only option, then I guess I need to give it a try. I might give this column a weight of 0.0 so that it doesn't create noise in the search results. Thanks Abhinav ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On 8/11/2011 1:35 PM, Abhinav Upadhyay wrote: > The documentation page of the VACUUM command says that "The VACUUM > command may change the ROWIDs of entries in any tables that do not > have an explicit INTEGER PRIMARY KEY." So what are the possible cases > in which the ROWIDs might change ? ROWIDs might possibly change if the table doesn't have an explicit INTEGER PRIMARY KEY column, and you run VACUUM commad on the database containing this table. Which part of the statement you quoted do you find unclear? -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On Thu, Aug 11, 2011 at 1:35 PM, Abhinav Upadhyay < er.abhinav.upadh...@gmail.com> wrote: > Hi, > > The documentation page of the VACUUM command says that "The VACUUM > command may change the ROWIDs of entries in any tables that do not > have an explicit INTEGER PRIMARY KEY." So what are the possible cases > in which the ROWIDs might change ? > In the current implementation, I think the only way you can get the ROWIDs to change is to compile SQLite with the SQLITE_OMIT_XFER_OPT compile-time option. But, we reserve the right to change this in future releases. In other words, future releases of SQLite might be more aggressive about renumbering rowids. So do not depend on the current behavior. If you need to preserve rowids, then declare an INTEGER PRIMARY KEY column. > > Thanks > Abhinav > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Use of VACUUM
On 11 Aug 2011, at 6:35pm, Abhinav Upadhyay wrote: > The documentation page of the VACUUM command says that "The VACUUM > command may change the ROWIDs of entries in any tables that do not > have an explicit INTEGER PRIMARY KEY." So what are the possible cases > in which the ROWIDs might change ? Not documented. So even if someone told you what they were in this version of SQLite, there might be other reasons in the next version. Theoretically it might renumber rows to close up gaps in the AUTOINCREMENT. As it says, to stop it all you need to do is declare one of the columns as INTEGER PRIMARY KEY. Once you do that it assumes that you might be referring to those values elsewhere and won't change them. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite3_wal_checkpoint wrror
On Thu, Aug 11, 2011 at 1:03 PM, Lisa Daveywrote: > Hi, reinstalling itunes didn't work. More info on the error, the label on > the error is "AppleSyncNotifier.exe-Entry Point Not Found" Perhaps try this: https://discussions.apple.com/message/15712311#15712311 > > > > On Wed, Aug 10, 2011 at 9:43 PM, Lisa Davey wrote: > >> I'll try that first. Thanks. >> Lisa >> >> >> On Wed, Aug 10, 2011 at 9:40 PM, Richard Hipp wrote: >> >>> >>> >>> On Wed, Aug 10, 2011 at 9:35 PM, Lisa Davey wrote: >>> In the past few weeks I installed uptates to Bonjour, itues, safari, apple application support. A program called HyperCam 3, The Garmin communicator plugin and usb drivers. >>> >>> And you don't know which of these was the last to be installed when the >>> problem first appeared? Bummer. I was hoping you could help us narrow down >>> which application installer is causing this bug. It's their problem, not >>> our, but we don't know who to report it to >>> >>> When I say "reinstall iTunes", I'm merely guessing that will fix it. >>> It's my standard answer to this problem. And nobody every writes back, so I >>> assume it always works. Let us know if you find out differently. >>> >>> On Wed, Aug 10, 2011 at 9:30 PM, Richard Hipp wrote: > > > On Wed, Aug 10, 2011 at 9:15 PM, Lisa Davey wrote: > >> Hi, I'm getting an error message whenever I reboot my computer stating >> "procedure entry point sqlite3_wal_checkpoint could not be located in >> the >> dynamic link library sqlite3.dll" >> How do I fix this? >> > > Reinstall iTunes. > > What software did you install recently that caused this to stop > working? > > > >> Thanks >> ___ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > > > > -- > D. Richard Hipp > d...@sqlite.org > >>> >>> >>> -- >>> D. Richard Hipp >>> d...@sqlite.org >>> >> >> > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Use of VACUUM
Hi, The documentation page of the VACUUM command says that "The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY." So what are the possible cases in which the ROWIDs might change ? Thanks Abhinav ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite with 10M record
Hello, You are Absolutely right on "if it ain't broke don't fix it". However said software is indeed broken on 32bit machine. When software try to load couple of XML together it won't load and goes out of memory. Even if PC has 700 MB free memory, However for some strange reason it works on 64bit machine just fine. So my idea is due to memory consumption in 32bit machine is limited not to give more than 2 GB on one object, this is causing the issue. With Sqlite I feel it won't consume that much RAM and I gain speed as side benefits, so looks good option to me, at least worth a try on couple of days efforts. With Regards, Sumit Gupta -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 18:58 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Although I think you'll pick up speed going to a database my philosophy is "if it ain't broke don't fix it". If you're speed is OK now why worry about the organization? Putting it in a database only gives you more chance for corruption which could hose your entire database. If I were you I'd just memory map the whole thing rather than using XML or a database since you have a fixed record size. That's also the most efficient storage. Then you just do a binary search on the time. You can use a C++ boost memory map class to do that. http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess/sharedmemorybetwe enprocesses.html This would be blazingly fast. Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:28 AM To: 'General Discussion of SQLite Database' Subject: EXT :Re: [sqlite] SQLite with 10M record Hello, I have Experience with SQL Server and MySql, but not with SQLite. Since this application will run on Desktop Machine and Data it store is kind a Temporary, so I think Locking is not needed at all [as this reduce processing time for application]. The data is actually Reading on 48 different parameter for every second of day for 30 days. So it is like 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> And we need to query on Time Only, i.e. when we create say PDF we find average or Sum of all entry [rows and column] and find values etc. That part is already working in great speed so I am not worried there. Just that I need to organize application data better. With Regards, Sumit Gupta -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 17:54 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on
[sqlite] Fwd: Case Sensitive FTS searches.
Hello again, After some digging through the sqlite3 source code, I came across the code for the ICU tokenizer. After enabling that and then commenting out the one line where u_foldCase() is called, i.e. icuOpen(), I retested and case-sensitive searching now seems to work as expected for FTS. I then tried doing the same thing in icuLikeCompare() by commenting out both u_foldCase() calls, i.e. for the string and the pattern and seem to have implemented case-sensitive LIKE searching as well for non FTS tables. Can anyone see anything wrong with this approach? I still plan to implement and register my own tokenizer, but will do it as high-lighted above. Finally, if this is a valid approach, then can I suggest that an additional preprocessor macro be defined which would allow this behaviour to be enabled for the general SQLite release code. Thanks again, Dennis -- Forwarded message -- From: Dennis SuehrDate: 11 August 2011 15:31 Subject: Case Sensitive FTS searches. To: sqlite-users@sqlite.org Hello, I need to implement case-SENSITIVE searching for an FTS(4) table. As near as I can determine, that would seem to require a new user-defined tokenizer. I am hopeful that one or more of you might respond with hints, insights, experience, etc. which might aid me toward that end. Specifically, if I do need to create a new tokenizer, which of the three existing ones, i.e. simple, porter-stemmer, or icu should I base it on? Thanks in advance. - Dennis ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
It`s so hard for me. I'll be very grateful :jumping: -- View this message in context: http://old.nabble.com/SQLite-%2B-unicode-tp32235242p32243061.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Case Sensitive FTS searches.
Hello, I need to implement case-SENSITIVE searching for an FTS(4) table. As near as I can determine, that would seem to require a new user-defined tokenizer. I am hopeful that one or more of you might respond with hints, insights, experience, etc. which might aid me toward that end. Specifically, if I do need to create a new tokenizer, which of the three existing ones, i.e. simple, porter-stemmer, or icu should I base it on? Thanks in advance. - Dennis ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Constraint failed infos
On Thu, Aug 11, 2011 at 10:09 AM, Pavel Ivanovwrote: > This is a known issue and it has an open ticket: > http://www.sqlite.org/src/tktview?name=23b2128201. > It is a known "feature request". It is not a bug. There is a lot of additional overhead involved in keeping track of names of constraints and formatting detailed error messages showing specifically which constraint failed. We could do so, but it would come at the cost of slower performance, increased code footprint, increased run-time memory usage, and increased code complexity. I remain unconvinced that the improved error messages are worth the cost. > > Pavel > > > On Wed, Aug 10, 2011 at 1:05 PM, Julien Laffaye wrote: > > Hello, > > > > Is it possible to get more info when I get a constraint failed error > > from the C API? > > I am interested in the column name, table name and type of constraint. > > > > Regards, > > Julien > > ___ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
please kill me %-| -- View this message in context: http://old.nabble.com/SQLite-%2B-unicode-tp32235242p32242440.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Constraint failed infos
This is a known issue and it has an open ticket: http://www.sqlite.org/src/tktview?name=23b2128201. Pavel On Wed, Aug 10, 2011 at 1:05 PM, Julien Laffayewrote: > Hello, > > Is it possible to get more info when I get a constraint failed error > from the C API? > I am interested in the column name, table name and type of constraint. > > Regards, > Julien > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
> Don't use sqlite3_exec. Use sqlite3_prepare16 (which accepts wchar_t*), > sqlite3_step, sqlite3_finalize. Read text from columns with > sqlite3_column_text16 (which returns wchar_t*). I'd say it's not exactly this way. AFAIK, wchar_t on Linux is 32-bit, but sqlite3_column_text16 will return 16-bit characters. And that will be UTF-16 string. For OP it will make any difference only if he wants to feed the received string to some function accepting real wchar_t*. Some conversion will be necessary. Pavel On Thu, Aug 11, 2011 at 8:04 AM, Igor Tandetnikwrote: > NOCaut wrote: >> I now how work with sqlite guys >> >> my problem: in const char * and i wont wchar_t*. becouse wchar_t* - >> unicode type understand >> >> int sqlite3_exec( >> sqlite3*, /* An open database */ >> const char *sql, /* SQL to be executed */ >> sqlite3_callback, /* Callback function */ >> void *, /* 1st argument to callback function */ >> char **errmsg /* Error msg written here */ >> ); > > Don't use sqlite3_exec. Use sqlite3_prepare16 (which accepts wchar_t*), > sqlite3_step, sqlite3_finalize. Read text from columns with > sqlite3_column_text16 (which returns wchar_t*). > -- > Igor Tandetnik > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite with 10M record
Although I think you'll pick up speed going to a database my philosophy is "if it ain't broke don't fix it". If you're speed is OK now why worry about the organization? Putting it in a database only gives you more chance for corruption which could hose your entire database. If I were you I'd just memory map the whole thing rather than using XML or a database since you have a fixed record size. That's also the most efficient storage. Then you just do a binary search on the time. You can use a C++ boost memory map class to do that. http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess/sharedmemorybetweenprocesses.html This would be blazingly fast. Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:28 AM To: 'General Discussion of SQLite Database' Subject: EXT :Re: [sqlite] SQLite with 10M record Hello, I have Experience with SQL Server and MySql, but not with SQLite. Since this application will run on Desktop Machine and Data it store is kind a Temporary, so I think Locking is not needed at all [as this reduce processing time for application]. The data is actually Reading on 48 different parameter for every second of day for 30 days. So it is like 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> And we need to query on Time Only, i.e. when we create say PDF we find average or Sum of all entry [rows and column] and find values etc. That part is already working in great speed so I am not worried there. Just that I need to organize application data better. With Regards, Sumit Gupta -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 17:54 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite with 10M record
2011/8/11 Sumit Gupta: > 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> Use unixtime (time in seconds) and index on this field and your queries will be fast. You can use single blob as integers array for more compact database. -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
NOCautwrote: > I want use but sqlity3.h NOT have this function. Does not have which function? The one you can download from http://sqlite.org/download.html certainly declares all the functions I've mentioned. -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
I want use but sqlity3.h NOT have this function. and i create this post for you help me use this function -- View this message in context: http://old.nabble.com/SQLite-%2B-unicode-tp32235242p32241783.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] how let readonly connection wait for locked database ?
On Wed, Aug 10, 2011 at 10:35 PM, Wenbo Zhaowrote: > Hi guys, > I want a readonly connection wait for locked database instead of error > return immediately. > The question is equivalence to how to let a reaonly connection have > busy_handler triggered > when the database is locked. > In my practice, the readonly connection will fail when db locked and > busy_handler does not work. > > If you set "PRAGMA journal_mode=WAL" then readers will never be blocked. > Thanks. > > -- > > Best Regards, > ZHAO, Wenbo > > === > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite with 10M record
Hello, I have Experience with SQL Server and MySql, but not with SQLite. Since this application will run on Desktop Machine and Data it store is kind a Temporary, so I think Locking is not needed at all [as this reduce processing time for application]. The data is actually Reading on 48 different parameter for every second of day for 30 days. So it is like 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> And we need to query on Time Only, i.e. when we create say PDF we find average or Sum of all entry [rows and column] and find values etc. That part is already working in great speed so I am not worried there. Just that I need to organize application data better. With Regards, Sumit Gupta -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 17:54 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite with 10M record
Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLite with 10M record
Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
NOCautwrote: > I now how work with sqlite guys > > my problem: in const char * and i wont wchar_t*. becouse wchar_t* - > unicode type understand > > int sqlite3_exec( > sqlite3*, /* An open database */ > const char *sql, /* SQL to be executed */ > sqlite3_callback, /* Callback function */ > void *, /* 1st argument to callback function */ > char **errmsg /* Error msg written here */ > ); Don't use sqlite3_exec. Use sqlite3_prepare16 (which accepts wchar_t*), sqlite3_step, sqlite3_finalize. Read text from columns with sqlite3_column_text16 (which returns wchar_t*). -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
You think i most 1 -convert to UTF-8 2 -read un the const char * 3 - convert to anscii i right understand you? -- View this message in context: http://old.nabble.com/SQLite-%2B-unicode-tp32235242p32241427.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Loading Options from the command line binary
I finally figured out how to load multiple "dot commands", pragma settings, or settings from the command line tool. (Some users only have the default binary to rely on, ya know.) Since I have never found this information on the web before, I thought I would post it here to share the information. To load options before doing the query, they have to be separated from the query by a hard return, in the quoted string itself. So: > sqlite3 -header -column tmp.db ".width 5 30; select * from data;" doesn't really change the column widths. But, doing this does: > echo ".width 5 30\n select * from data;" | sqlite3 -header -column tmp.db To load multiple options, you have to put them on different lines, with the query on the last line: > echo ".width 5 30\n.timeout 15000\n select * from data;" | sqlite3 -header -column tmp.db Some systems I have tried this on /required /only a single hard return after the last option, with a semicolon between the multiple settings. Others required the hard returns without any semicolons. Experiment on your own platform/binary version. I don't know which versions/flavors this works on, but it works for me finally. Note: Your version of echo has to support turning '\n' into a hard return. Not all do. If not, you can use a perl -e (or similar) to do it as well. All the examples on websites show applying the dot commands from a sqlite> prompt. Doing the hard returns within the string makes it look like a user typing the commands in, with the hard returns. If anyone could fix the parser in the sqlite3 source code, I'm sure many novices would greatly appreciate it. The above trick has eluded me for several months of working with sqlite3 from the command line, and is still cumbersome, although usable now. One other option is to put the dot commands in a .sqliterc file in your home directory, one dot command per line. This works, but is not really feasible in an environment where it will run on multiple hosts and you don't control other accounts that will run it. Also, you can't pick and choose which ones are loaded. They are global settings at that point. I hope this helps someone out there looking for this, as well as possibly gets the parser fixed to make it easier to use in the future. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
Hello NOCaut, I convert everything to UTF-8 for insert and then back to ascii or unicode when I pull the data from the DB. C Thursday, August 11, 2011, 4:20:36 AM, you wrote: N> I now how work with sqlite guys N> my problem: in const char * and i wont wchar_t*. becouse wchar_t* - N> unicode type understand N> int sqlite3_exec( N> sqlite3*, /* An open database */ N> const char *sql, /* SQL to be executed */ N> sqlite3_callback, /* Callback function */ N> void *, /* 1st argument to callback function */ N> char **errmsg /* Error msg written here */ N> ); N>sqlite3 *db; N> char *zErrMsg = 0; N> int rc; N> rc = sqlite3_open("c:\\test.db", ); N> N> rc = sqlite3_exec(db, "Select * from table ", callback, 0, ); N> this code return char I want use wchar_t* for read unicode. N> Thanks! -- Best regards, Tegmailto:t...@djii.com ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite + unicode
I now how work with sqlite guys my problem: in const char * and i wont wchar_t*. becouse wchar_t* - unicode type understand int sqlite3_exec( sqlite3*, /* An open database */ const char *sql, /* SQL to be executed */ sqlite3_callback, /* Callback function */ void *, /* 1st argument to callback function */ char **errmsg /* Error msg written here */ ); sqlite3 *db; char *zErrMsg = 0; int rc; rc = sqlite3_open("c:\\test.db", ); rc = sqlite3_exec(db, "Select * from table ", callback, 0, ); this code return char I want use wchar_t* for read unicode. Thanks! -- View this message in context: http://old.nabble.com/SQLite-%2B-unicode-tp32235242p32240215.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] sqlite-src-3070701.zip compilation issues
Also, what command are you using to compile? On Thu, Aug 11, 2011 at 6:19 AM,wrote: > > Pls list the compilation errors.. > > --Original Message-- > From: Raja Kondu > Sender: sqlite-users-boun...@sqlite.org > To: sqlite-users@sqlite.org > ReplyTo: General Discussion of SQLite Database > Subject: [sqlite] sqlite-src-3070701.zip compilation issues > Sent: Aug 11, 2011 4:36 AM > > Hi, > > I have downloaded the SQLite version 3.7.7.1 from the *Legacy Source Code > Distribution Formats . * > ** > Here I am just compiling the source code from the "src" folder and I am > seeing lot of compilation errors by saying the header files missing and > some > of the header files are really missing. > > Could some one please tell me is there any issue in this specific release ? > > Earlier I use to compile the 3.6.17 version and now I am trying with the > latest version 3.7.7.1. > > Please help me to get rid of these compilation errors. > > > -- > Thanks, > Raja Kondu. > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > Sent from BlackBerry® on Airtel > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] tcl incrblob interface
On 08/10/2011 11:24 PM, Victor Mayevski wrote: > Hello, > > I am trying to learn how to use the "incrblob" command in the Tcl > interface and I can't get it to work. I create a one column table "t", > insert one empty value into it, than do "db incrblob t a 1", which > works fine, I get a file pointer back "incrblob_1". > Then I do "puts incrblob_1 "hello world"", which also works. However, > "chan close incrblob_1" fails with "Error: invalid argument" and doing > "select * from t" shows no new data inserted. > Any ideas what I am doing wroing? Note that the incrblob interface cannot grow the size of a blob. Only modify it's contents. Is the initial "empty value" large enough to hold "hello world\n"? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users