Re: [sqlite] Speeding up a query

2016-11-29 Thread Simon Slavin
On 29 Nov 2016, at 9:41pm, Kevin Baggett wrote: > Here's the query: > SELECT a.observation_day, a.observation_hour, a.observation_time, > a.text_file_name, a.start_byte, a.message_length, a.wmo_header, a.wmo_prefix, > max(a.rmkcorr_flag),b.wmo_ID,b.latitude,b.longitude from main.file_list a,

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Keith Medcalf
The standard computer answer applies: The fastest way to perform I/O is not to do it (and contrary to recent press, this discovery was not made by Microsoft Research a couplke of weeks ago but was "discovered" in the 1950's when, on average, a good secretary could find a file in the filing cab

[sqlite] Speeding up a query

2016-11-29 Thread Kevin Baggett
Hi, I am trying to speed up a query on my SQLite database using SQLite version 3.7.17 2013-05-20 00:56:22. The daily database has 2 tables: file_list and station_list. Yesterday's database had a file_list table of over 1.7 million records. station_list is pretty much constant at 21549 records.

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread James K. Lowden
On Thu, 24 Nov 2016 22:59:32 +0100 Florian Weimer wrote: > Does LMDB perform lock-free optimistic reads and > retroactively verifies that the entire read operation was consistent? In LMDB there are readers and reader-writers. A reader never writes; a reader-writer may read, but that read coun

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Simon Slavin
On 29 Nov 2016, at 6:10pm, Mark Hamburg wrote: > In the single WAL scenario, what I probably really want is a way for the > checkpoint operation on the write connection to do its work then wait for > exclusive access — standard reader/writer lock pattern — to do the WAL reset. > This would pr

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Jens Alfke
> On Nov 29, 2016, at 9:09 AM, Simon Slavin wrote: > > You cannot design a system which (A) provides up-to-date data to readers (B) > allows writers to get rid of their data immediately without ever locking up > and (C) guarantees that earlier changes to the data are ’saved' before later > ch

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Scott Hess
On Tue, Nov 29, 2016 at 10:10 AM, Mark Hamburg wrote: > On Nov 29, 2016, at 9:09 AM, Simon Slavin wrote: >>> On 29 Nov 2016, at 4:18pm, Mark Hamburg wrote: >>> >>> Does this make sense? Does it seem useful? (It seems useful to me when I >>> see multi-megabyte WAL files.) >> >> Sorry, but I cann

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Mark Hamburg
On Nov 29, 2016, at 9:09 AM, Simon Slavin wrote: > > >> On 29 Nov 2016, at 4:18pm, Mark Hamburg wrote: >> >> Does this make sense? Does it seem useful? (It seems useful to me when I see >> multi-megabyte WAL files.) > > Sorry, but I cannot spare the time right now to analyze the system you l

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Simon Slavin
On 29 Nov 2016, at 4:18pm, Mark Hamburg wrote: > Does this make sense? Does it seem useful? (It seems useful to me when I see > multi-megabyte WAL files.) Sorry, but I cannot spare the time right now to analyze the system you laid out. It usually takes half an hour to diagram out the read an

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Mark Hamburg
One other question about WAL mode and simultaneous readers and writers: How are people setting their page caches? My read is that shared cache is probably not what's wanted. I was setting my reader caches to be bigger than my writer cache under the assumption that writers write and then move on

Re: [sqlite] Read-only access which does not block writers

2016-11-29 Thread Mark Hamburg
Once I figured out how to reliably get the reader and writer connections open on a database — key point, when creating the database let the writer get through all of the creation work before opening any readers — I've been getting great concurrency behavior for simultaneous reads and writes in W

Re: [sqlite] trimming with tab and other special characters

2016-11-29 Thread Max Vlasov
On Tue, Nov 29, 2016 at 4:19 PM, Igor Tandetnik wrote: That's the exact opposite of your interpretation. For backslash escapes, > you need to rely on "facilities of your programming language". If you > cannot, and must use raw SQL queries, there are still ways to represent > arbitrary characters,

Re: [sqlite] trimming with tab and other special characters

2016-11-29 Thread Igor Tandetnik
On 11/29/2016 4:00 AM, Max Vlasov wrote: I wonder why OP and other authors of the discussion https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg49355.html was so sure about backslash escaping support, even Igor Tandetnik :) I said, and I quote: """ If you do it in your progr

Re: [sqlite] trimming with tab and other special characters

2016-11-29 Thread R Smith
On 2016/11/29 11:00 AM, Max Vlasov wrote: Reasonable enough, I wonder why OP and other authors of the discussion https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg49355.html was so sure about backslash escaping support, even Igor Tandetnik :) None of them were sure about

Re: [sqlite] trimming with tab and other special characters

2016-11-29 Thread Max Vlasov
Reasonable enough, I wonder why OP and other authors of the discussion https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg49355.html was so sure about backslash escaping support, even Igor Tandetnik :) On Tue, Nov 29, 2016 at 11:39 AM, Clemens Ladisch wrote: > Max Vlasov wrot

Re: [sqlite] trimming with tab and other special characters

2016-11-29 Thread Clemens Ladisch
Max Vlasov wrote: > trim(col, char(9)) > works, while > trim(col,'\t') > does not. SELECT trim('ttthello\tt\\\', '\t'); hello Works as designed. SQL does not use backslash escaping. Use char(9) or an actual tab character (' '). Regards, Clemens _

[sqlite] trimming with tab and other special characters

2016-11-29 Thread Max Vlasov
Hi, the search in the mailing list about the trim function reveals possible escaping support for the second parameter of the function, but in my case (sqlite 3.15.1) trim(col, char(9)) works, while trim(col,'\t') does not. Can someone clarify on that? Thanks Max ___