[sqlite] Destroy all evidence of a database

2015-04-22 Thread Scott Hess
On Wed, Apr 22, 2015 at 12:07 PM, Simon Slavin wrote: > On 22 Apr 2015, at 7:18pm, Scott Hess wrote: >> The only way SQLite can get to the disk is using the vfs, so if the >> vfs encrypts things, all of the files (main db, temp db, journal, >> everything) will be encry

[sqlite] Destroy all evidence of a database

2015-04-22 Thread Scott Hess
The only way SQLite can get to the disk is using the vfs, so if the vfs encrypts things, all of the files (main db, temp db, journal, everything) will be encrypted. I think in your case you can probably even get away without the more elaborate encrypted systems, because it sounds like you don't wa

[sqlite] Best way to temporarily store data before processing

2015-04-15 Thread Scott Hess
On Tue, Apr 14, 2015 at 1:10 PM, David Cotter wrote: > does this awesome sounding temp table then merge instantly when you commit it > even if it?s huge? Since I have no idea what "merge" means in this context, it's hard to answer your question. -scott

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Scott Hess
On Tue, Apr 14, 2015 at 9:37 AM, Jim Callahan wrote: > My recollection is that SQLite has a "temp" or "tmp" namespace available > for intermediate tables -- it was on my todo list, but I never got around > to exploring that option. CREATE TEMP TABLE ... works very well for things like this. If y

[sqlite] fts5

2015-04-08 Thread Scott Hess
On Wed, Apr 8, 2015 at 12:32 PM, Dan Kennedy wrote: > On 04/08/2015 04:49 AM, Scott Hess wrote: >> Something that bugged me a lot was that I had used deletion markers to >> cancel out hits, but did not provide a way for deletion markers to >> cancel out. The main problem

[sqlite] fts5

2015-04-07 Thread Scott Hess
On Thu, Sep 11, 2014 at 8:58 AM, Dan Kennedy wrote: > Fts5 is still in the experimental stage at the moment. > > If anybody has any ideas for useful features, or knows of problems with FTS4 > that could be fixed in FTS5, don't keep them to yourself! Apologies for not noticing this thread earlier!

[sqlite] Different timings on min() and max()

2015-04-07 Thread Scott Hess
On Tue, Apr 7, 2015 at 1:42 PM, Paul Caskey wrote: > Any idea why there is such a performance hit when I ask for both min() and > max() at the same time? Shouldn't it be just as fast as querying them > individually? This has come up before. Either alone can be satisfied from an index lookup, whi

[sqlite] It might be nice to have sqlite3_attach() and sqlite3_detach().

2015-03-19 Thread Scott Hess
On Thu, Mar 19, 2015 at 10:30 AM, Richard Hipp wrote: > On 3/18/15, Scott Hess wrote: >> I'm thinking I could use something like: >> >> SQLITE_API int sqlite3_attach(sqlite3* db, const char* zPath, const >> char* dbname); >> SQLITE_API int sqlite3_

[sqlite] It might be nice to have sqlite3_attach() and sqlite3_detach().

2015-03-18 Thread Scott Hess
I'm thinking I could use something like: SQLITE_API int sqlite3_attach(sqlite3* db, const char* zPath, const char* dbname); SQLITE_API int sqlite3_detach(sqlite3* db, const char* dbname); Right now, I have a helper in Chromium which does "ATTACH DATABASE ? AS ?". This works, but AFAICT this page

[sqlite] Memory leak?

2015-03-13 Thread Scott Hess
On Fri, Mar 13, 2015 at 3:13 AM, Matthias Schmitt wrote: >> On 12 Mar 2015, at 17:35, Richard Hipp wrote: >> >> Let's start with the basics: How do you know that the memory was in >> fact leaked and is not instead simply being held for reuse? > > The Xcode development environment comes with a de

[sqlite] Corrupted database

2015-03-04 Thread Scott Hess
On Wed, Mar 4, 2015 at 1:11 AM, Alexandr N?mec wrote: > we have a product that uses SQLite. Because it was running very stable for > years, we are still using SQLite 3.7.17. Now we've seen on one of our > installations that the database has been corrupted, we saw that there was a > power failure a

[sqlite] Why bind indexes start from 1 and column indexes start from 0?

2015-03-03 Thread Scott Hess
On Tue, Mar 3, 2015 at 3:33 AM, Paolo Bolzoni wrote: >> I can't confirm that 100% off the top of my head but I'm uncoordinated >> enough to repeatedly confuse the bind and column value API calls and use >> 0-based indices for both and haven't noticed any really untoward behaviour >> (beyond my cod

Re: [sqlite] Opening an in-memory database in two connections, one read-write, one read-only.

2013-05-02 Thread Scott Hess
No comment? Do I need to recast the problem or something? Thanks, scott On Fri, Apr 26, 2013 at 9:54 PM, Scott Hess wrote: > Someone over here was trying to use a shared-cache in-memory database with > two connections, one a read-write connection for purposes of populating > things,

[sqlite] Opening an in-memory database in two connections, one read-write, one read-only.

2013-04-26 Thread Scott Hess
Someone over here was trying to use a shared-cache in-memory database with two connections, one a read-write connection for purposes of populating things, the other a read-only connection for purposes of letting a (trusted) user make queries. They were surprised to find out that they could run wri

Re: [sqlite] update record in contentless FTS4

2013-04-16 Thread Scott Hess
On Fri, Apr 12, 2013 at 9:28 AM, Lukas Gebauer wrote: > I have contentless FTS4 index for searching some my external data > paired by docid. > > What I can do, when my existing previously indexed document was > changed? I need to update existing FTS4 index too. But documentation > says: "UPDATE a

Re: [sqlite] secure_delete versus journal_mode persist.

2013-01-29 Thread Scott Hess
On Tue, Jan 29, 2013 at 7:52 AM, Yuriy Kaminskiy wrote: > User-space database encryption and all "safe data erase" tools only > provide you > with warm feeling of safety without any real security. I agree. I am merely pointing out that I think that this result may be surprising in the case whe

[sqlite] secure_delete versus journal_mode persist.

2013-01-29 Thread Scott Hess
Given a 3.7.15.2 OSX binary fresh from the website, the following scenario seems suspect: > ./sqlite3 trial.db sqlite> pragma secure_delete = 1; 1 sqlite> pragma journal_mode = persist; persist sqlite> pragma journal_size_limit = 4096; 4096 sqlite> pragma page_size; 1024 sqlite> create table x (x)

Re: [sqlite] How to know what terms were created using FTS

2012-08-27 Thread Scott Hess
On Sat, Aug 18, 2012 at 10:00 AM, Mohit Sindhwani wrote: > On 17/8/2012 7:14 PM, Dominique Pellé wrote: >> This gives the tokens: >> >> sqlite> CREATE VIRTUAL TABLE ft USING fts4(x); >> sqlite> INSERT INTO ft VALUES("hello world"); >> sqlite> INSERT INTO ft VALUES("hello there"); >> >> sqlite> CRE

Re: [sqlite] VACUUMing large DBs

2012-03-22 Thread Scott Hess
On Tue, Mar 20, 2012 at 8:25 PM, Jay A. Kreibich wrote: > On Tue, Mar 20, 2012 at 01:59:59PM -0700, Udi Karni scratched on the wall: >> Is there a way to go directory from "original" to "journal/final" - >> skipping the creation of the Temp version? > >  No, it requires all three copies. <...> >  

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

2012-03-08 Thread Scott Hess
On Thu, Mar 8, 2012 at 9:28 AM, Pavel Ivanov wrote: >> Question:  Does anybody know of a better way to get memory shared among >> processes other than to create a fake file and mmap() it?  Are there some >> magic options to mmap() (perhaps Linux-only options) that prevent it from >> actually writi

Re: [sqlite] Tuning a SQLite database for best compression

2012-03-05 Thread Scott Hess
Note that importing the .dump file does effectively strip the indices and re-create them. In fact, there's a good chance that if you drop the indices, VACUUM, and then compress the database file, you'll find that it comes close to what you get with the .dump-then-compress values. -scott On Mon,

Re: [sqlite] Prepare SQL for Read-Only Database with Journal File

2012-02-17 Thread Scott Hess
On Fri, Feb 17, 2012 at 1:02 PM, Richard Hipp wrote: > On Fri, Feb 17, 2012 at 3:52 PM, Marc L. Allen > wrote: >> My concept of always consistent is a.. for lack of a better term... >> virtual concept.  That is, anytime anything accesses the database, as a >> database, that database is always con

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 12:28 PM, Simon Slavin wrote: > On 13 Feb 2012, at 7:51pm, Steinar Midtskogen wrote: >> One should think that the ability to calculate a moving average would >> be a pretty common request.  But people do it in their application >> code instead? > > Actually, my expectation

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 11:51 AM, Steinar Midtskogen wrote: > [Scott Hess] >> Unfortunately, I can't offhand think of a reasonable solution for you, >> I think I'd just use the SELECT to generate the data, while >> calculating the moving average in my application

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 9:24 AM, Steinar Midtskogen wrote: > [Peter Aronson] >> (2) You can associate data with an argument to a regular user-defined >> function using sqlite3_set_auxdata() and sqlite3_get_auxdata() as long >> as the value of the argument is static.  If you don't normally have a >

Re: [sqlite] Segment merging in FTS and updates-deletes

2011-11-11 Thread Scott Hess
On Fri, Nov 11, 2011 at 9:58 AM, nobre wrote: > Hi, I'm studying the indexing mechanism of FTS3/4, I can pretty much > understand how doclists, terms, segments are created and stored, but one > thing I can't grasp is about updating and deleting docs and keeping up the > index up to date. From the

Re: [sqlite] Readonly error when copying to mem-based db

2011-11-07 Thread Scott Hess
On Mon, Nov 7, 2011 at 9:02 AM, Mills, Steve wrote: >                if(pageSize > 0) { >                        SqliteStatement  cmd("PRAGMA page_size = ?", *this); > >                        cmd.BindToInt(1, syz); >                        cmd.Step(); >                } Can't prepare and bind a

Re: [sqlite] cache size and insert-only

2011-10-28 Thread Scott Hess
On Fri, Oct 28, 2011 at 5:36 AM, Jay A. Kreibich wrote: > On Fri, Oct 28, 2011 at 01:33:31PM +0200, Gert Corthout scratched on the wall: >> we have a database that only performs insert statements on a table >> (database contains only this table), is it in this case a good idea >> to reduce the cac

Re: [sqlite] FTS vs INDEX

2011-10-19 Thread Scott Hess
On Wed, Oct 19, 2011 at 12:50 PM, Fabian wrote: > 2011/10/19 Scott Hess >> To be clear, how it works is that new insertions are batched into a >> new index tree, with index trees periodically aggregated to keep >> selection efficient and to keep the size contained.  So w

Re: [sqlite] FTS vs INDEX

2011-10-19 Thread Scott Hess
On Wed, Oct 19, 2011 at 7:56 AM, Fabian wrote: > 2011/10/19 Alexey Pechnikov >> FTS use index multi-tree and de-facto has _no_ insert speed degradation. > > Thanks, that's good to hear! It makes me wonder why SQLite doesn't use that > same multi-tree mechanism for regular indexes, but that's a wh

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011-09-27 Thread Scott Hess
On Tue, Sep 27, 2011 at 2:07 PM, Patrick Proniewski wrote: > On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: > > gawk has fflush() > > none of them is available out of the box on Mac OS X, or FreeBSD. gawk can > be installed, but I'd rather use my "while true" loop instead of installing > ga

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011-09-27 Thread Scott Hess
On Tue, Sep 27, 2011 at 11:18 AM, Gabor Grothendieck < ggrothendi...@gmail.com> wrote: > On Tue, Sep 27, 2011 at 2:14 PM, David Garfield > wrote: > > Any entry in a pipe could be buffering. In a quick test here, awk is > > buffering. To find the buffering, try using the pieces up to a given > >

Re: [sqlite] How to reindex an FTS3 table after changing the tokenizer

2011-08-22 Thread Scott Hess
On Fri, Aug 12, 2011 at 11:27 AM, john Papier wrote: > I have a FTS3 table that was created with the simple tokenizer. I want to > change the tokenizer and reindex the table. > > Is there a way to change the tokenizer in place and have it reindex with > minimal code? > > Else the other option I wa

Re: [sqlite] Fwd: Case Sensitive FTS searches.

2011-08-11 Thread Scott Hess
On Thu, Aug 11, 2011 at 8:57 AM, Dennis Suehr wrote: > After some digging through the sqlite3 source code, I came across the code > for the ICU tokenizer.  After enabling that and then commenting out the one > line where u_foldCase() is called, i.e. icuOpen(), I retested and > case-sensitive searc

Re: [sqlite] How to convert a database with FTS2 to FTS3/4?

2011-07-31 Thread Scott Hess
On Fri, Jul 29, 2011 at 10:33 PM, Ray Van Dolson wrote: > sqlite newbie here.  I have a sqlite3 database that was built using the > FTS2 module.  As such I get errors about the non-existent fts2 module > when attempting to query the data. > > Is there a way to convert/migrate the data to FTS3 or s

[sqlite] Bug with fts3 when using a restrictive authorizer.

2011-06-22 Thread Scott Hess
If the authorizer prevents PRAGMA, then fts3 fails with an SQLITE_AUTH error from fts3DatabasePageSize(). http://crbug.com/85522 is the Chromium bug where this originated (I upgraded the Chromium SQLite version a few weeks back). http://codereview.chromium.org/7230021 is the short-term patch I'm

Re: [sqlite] Handling of IEEE-754 nan and +/-inf in SQLite?

2011-06-06 Thread Scott Hess
On Mon, Jun 6, 2011 at 1:46 PM, Nico Williams wrote: > I can see two ways to add proper IEEE754 support in a > backwards-compatible way: a) via a pragma to enable raising exceptions > or returning +-inf/NaNs, b) by adding a callback via which to report > such exceptions, with exceptions raised onl

Re: [sqlite] Page Size with NAND FLASH

2011-05-04 Thread Scott Hess
On Tue, May 3, 2011 at 4:44 PM, Sugathan, Rupesh wrote: > I am planning to use sqlite on a Linux system with JFFS2 file system on > NAND flash. NAND device that I am using has page size of 2048 bytes and > a erase sector size of 128K. I would like to take advantage of sqlite > rollback for the saf

[sqlite] WAL checkpoints versus fsync.

2011-03-30 Thread Scott Hess
[From a thread about WAL checkpoints...] On Wed, Mar 30, 2011 at 6:28 AM, Richard Hipp wrote: > On Wed, Mar 30, 2011 at 8:46 AM, Mikael wrote: >> Therefore I now wish to check with you, is SQLite implemented to somehow >> make checkpoints not be able to corrupt the main database file ever? (Say

Re: [sqlite] Optimizing list retrieval with a FTS3 table

2011-03-16 Thread Scott Hess
On Wed, Mar 16, 2011 at 12:15 PM, Travis Orr wrote: > Can someone enlighten me about this. To me a lot of the details appear > to be hidden since my main SongTable is a FTS3 virtual table. You don't provide your schema, but based on your queries, I'll make unwarranted assumptions :-). In fts3, t

Re: [sqlite] Fwd: Fwd: fts virtual table questions

2011-03-01 Thread Scott Hess
On Tue, Mar 1, 2011 at 4:54 PM, Paul Shaffer wrote: > I had an API problem, and I now think that the virtual table does not need > to be created each time. I can't find documentation that covers this. I don't think there is documentation which covers this, because it's how all tables (except temp

Re: [sqlite] apostrophes in strings...

2011-02-23 Thread Scott Hess
On Wed, Feb 23, 2011 at 12:50 PM, BareFeetWare wrote: > On 21/02/2011, at 8:11 AM, Scott Hess wrote: >> You can also convert: >>  ATTACH DATABASE x AS y KEY z >> to: >>  SELECT sqlite_attach(x, y, z) >> where the parameters can be turned into bind arguments.  The

Re: [sqlite] apostrophes in strings...

2011-02-23 Thread Scott Hess
I believe the key is used for attaching to encrypted databases. -scott On Wed, Feb 23, 2011 at 7:15 AM, Sam Carleton wrote: > Kevin, > > Thank you, that is what I needed.  Now to statisfy my curiosity...  What > exactly is the KEY value? > > On Tue, Feb 22, 2011 at 10:39 PM, Kevin Benson > wro

Re: [sqlite] apostrophes in strings...

2011-02-20 Thread Scott Hess
You can also convert: ATTACH DATABASE x AS y KEY z to: SELECT sqlite_attach(x, y, z) where the parameters can be turned into bind arguments. Then embedded quotes won't be an issue. -scott On Sun, Feb 20, 2011 at 11:31 AM, Pavel Ivanov wrote: > I believe doubling the single quote inside th

Re: [sqlite] will FTS speed up [LIKE '%foo%'] searches? or: how does iTunes do it?

2011-02-18 Thread Scott Hess
On Fri, Feb 18, 2011 at 12:08 AM, David M. Cotter wrote: > so i am still left wondering if searching substrings is really any faster > using FTS. You may want to search the archives, as this has come up before. I don't recall if anyone had an inspired solution. You could possibly use a custom

Re: [sqlite] integer or string in fts3 table

2011-01-31 Thread Scott Hess
On Mon, Jan 31, 2011 at 9:15 AM, Samuel Adam wrote: > On Mon, 31 Jan 2011 11:46:21 -0500, Samuel Adam wrote: >> On Mon, 31 Jan 2011 11:05:13 -0500, Simon Slavin >> wrote: >>> In the definition given in the original post, which I quoted, the table >>> was defined as follows: >>> CREATE VIRT

Re: [sqlite] Deleting the sqlite journal file?

2011-01-21 Thread Scott Hess
On Fri, Jan 21, 2011 at 5:33 PM, Pavel Ivanov wrote: >>        (2) Are there any changes that can/should be made >> to sqlite3 so that it can identify the bogus journal in this >> scenario and discard it? > > Define the word "bogus". How should SQLite understand that the journal > and the database

Re: [sqlite] WAL on a separate filesystem?

2011-01-19 Thread Scott Hess
On Wed, Jan 19, 2011 at 12:15 PM, Dustin Sallings wrote: > This isn't a question so much about value judgment (I've already > argued that some, though mentioning maintenance tools is helpful > there, too).  It comes down to whether reliability of SQLite itself > would be reduced if a WAL existed o

Re: [sqlite] EXTERNAL:Re: Rounding Strategy

2011-01-05 Thread Scott Hess
On Wed, Jan 5, 2011 at 8:27 AM, Jon Polfer wrote: >> Butthe more important questoin is "Does it make a difference to > your application?" > > Not really.  I've talked with the PMs and such about it, and this > doesn't seem to be a problem.  I wanted to know how SQLite3 worked in > this departm

Re: [sqlite] Rounding Strategy

2011-01-04 Thread Scott Hess
SQLite handles rounding by running the value through the internal printf with the appropriate precision. As best I can tell the internal printf adds half a unit at the appropriate position, then truncates. Since the 3.05 isn't precisely represented (with format %.16f, I get "3.0498"),

Re: [sqlite] pragma vs select for introspection

2010-12-13 Thread Scott Hess
On Mon, Dec 13, 2010 at 1:27 PM, Puneet Kishor wrote: > Wols Lists wrote: >> On 13/12/10 01:38, Darren Duncan wrote: >>> Darren Duncan wrote: Wols Lists wrote: > Dunno how well that approach translates into a relational engine, > because Pick has several very non-relational quirks (ev

Re: [sqlite] Increase the datafile file size to limit the file fragmentation

2010-12-10 Thread Scott Hess
On Fri, Dec 10, 2010 at 12:20 PM, Vander Clock Stephane wrote: > to limit the file fragmentation i want to increase the size of the > database file (with the windows API function). Consider http://www.sqlite.org/c3ref/c_fcntl_chunk_size.html . -scott _

Re: [sqlite] Stored procedures (was: Question about SQLite features.)

2010-11-12 Thread Scott Hess
On Fri, Nov 12, 2010 at 5:50 AM, BareFeetWare wrote: > IMO, if you're implementing database logic (ie constraints and triggers) in > application code, then you're reinventing the wheel, making your package > unnecessarily complex and grossly inefficient. If you're just using SQLite > to store your

Re: [sqlite] Full text search FTS3 of files

2010-10-18 Thread Scott Hess
On Sun, Oct 17, 2010 at 11:13 PM, Dami Laurent (PJ) wrote: >>Is it possible to use FTS3 for search without storing the actual file >>contents/search terms/keywords in a row. In other words, create a FTS3 >>tables with rows that only contains an ID and populate the B-Tree with >>keywords for search

Re: [sqlite] Many many tables... performance wise?

2010-10-15 Thread Scott Hess
Having a table with an owner_id, key, and value, with a unique index on (owner_id, key) will probably be more efficient than having a separate table per owner. Also, it will be easier to code safely, because bind parameters don't work on table names (I'm assuming you're using dynamic table names i

Re: [sqlite] Registering a custom tokenizer per database rather than per connection

2010-10-12 Thread Scott Hess
On Tue, Oct 12, 2010 at 8:40 AM, Drake Wilson wrote: > Just to clarify, a trigger isn't useful for this in general, because > you still have nowhere to get the function pointer from.  Above I was > worrying that the current SQLite code might be less safe than it could > be as a result of allowing

Re: [sqlite] Speed up DELETE of a lot of records

2010-10-12 Thread Scott Hess
On Mon, Oct 11, 2010 at 8:13 AM, Jay A. Kreibich wrote: > On Mon, Oct 11, 2010 at 02:08:54PM +0200, Michele Pradella scratched on the > wall: >>   Ok so the main idea it's always the same: split the DELETE to make the >> operation on less records, but do it more often. > >  Another thought occurs

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 10:20 PM, Max Vlasov wrote: >> I wonder whether it would be possible to extend the VFS in a way that would >> make our use case possible (transferring a file handle over process >> boundary). Please note that we do it on all platforms. On POSIX we pass an >> integer file des

Re: [sqlite] Feature request: copying vacuum

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 10:09 PM, Max Vlasov wrote: >> I agree with Jay - while it is tempting to have SQLite bite off >> optimizing this kind of thing, it's pretty far out of scope.  Next >> we'll be talking about running SQLite on raw partitions! >> > > Scott, thought about it, ironically sqlite

Re: [sqlite] Feature request: copying vacuum

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 12:46 PM, Jay A. Kreibich wrote: > On Wed, Sep 01, 2010 at 11:41:00AM -0700, Taras Glek scratched on the wall: >> Currently VACUUM takes care of sqlite-level fragmentation. Unfortunately >> it does little for fs-level fragmentation since the same file is being >> reused. It

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-18 Thread Scott Hess
I'm not sure Chromium has any fts1 databases, I think the original patch was applied there for completeness. The change from fts2 to fts3 has been made in the history system, but it only applies to new data, and hasn't yet rolled out to stable. So we wouldn't be able to even start to cease using

Re: [sqlite] [PATCH] cache preloading

2010-08-17 Thread Scott Hess
On Mon, Aug 16, 2010 at 7:58 PM, Shawn Wilsher wrote: > On Mon, Aug 16, 2010 at 5:13 PM, Paweł Hajdan, Jr. > wrote: >> Is it something you'd like to include in SQLite? If so, does the patch need >> any adjustments before that's possible? > I'm slightly concerned about licensing here - do we know

Re: [sqlite] Substring (LIKE "%key%") searches, would FTS3 with suffix-tree tokenizer be the fast way?

2010-08-06 Thread Scott Hess
On Fri, Aug 6, 2010 at 6:08 PM, Sam Roberts wrote: > On Fri, Aug 6, 2010 at 11:32 AM, Scott Hess wrote: >> On Thu, Aug 5, 2010 at 12:42 PM, Sam Roberts wrote: >>> FTS3 only searches full terms/words by default, but I think if I built a >>> custom >>> to

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-06 Thread Scott Hess
chard Hipp wrote: > FTS3 updated here:  http://www.sqlite.org/src/ci/b8b465ed2c > > On Fri, Aug 6, 2010 at 2:24 PM, Scott Hess wrote: > >> This bug comment describes the problem: >>   http://code.google.com/p/chromium/issues/detail?id=15261#c20 >> >> excerpt:

Re: [sqlite] Substring (LIKE "%key%") searches, would FTS3 with suffix-tree tokenizer be the fast way?

2010-08-06 Thread Scott Hess
On Thu, Aug 5, 2010 at 12:42 PM, Sam Roberts wrote: > FTS3 only searches full terms/words by default, but I think if I built a > custom > tokenizer that returned all the suffix trees for a name: FTS3 can do prefix searches, MATCH 'a*'. Also, it aimed to support multiple hits at the same positio

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-06 Thread Scott Hess
This bug comment describes the problem: http://code.google.com/p/chromium/issues/detail?id=15261#c20 excerpt: > Apparently the problem is caused by tolower(), whose behavior is affected by > current > locale. Under locale tr_TR.UTF-8, tolower('I') returns 'I' rather than 'i', > because > lower

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-04 Thread Scott Hess
Pawel, You should probably pull the current SQLite code and make sure the patch even applies, and if not, check to make sure that the problem hasn't already been fixed. ext/fts3 should no longer have the flaw in question, as that code was heavily rewritten. Chromium's SQLite was last synced from

Re: [sqlite] Optimizing Songbird

2010-08-02 Thread Scott Hess
[Sorry for the mis-fire.] On Mon, Aug 2, 2010 at 11:11 AM, Simon Slavin wrote: > On 2 Aug 2010, at 7:06pm, Richard Hipp wrote: >> Most of the slow queries seem to be "SELECT count(*) FROM ".  Such >> queries have to visit every row in the table (in order to count the rows) >> and thus get pro

Re: [sqlite] Optimizing Songbird

2010-08-02 Thread Scott Hess
On Mon, Aug 2, 2010 at 11:11 AM, Simon Slavin wrote: > On 2 Aug 2010, at 7:06pm, Richard Hipp wrote: >> Most of the slow queries seem to be "SELECT count(*) FROM ".  Such >> queries have to visit every row in the table (in order to count the rows) >> and thus get progressively slower as the nu

Re: [sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-07-30 Thread Scott Hess
Ah, OK (I was finding an older different version on the sqlite.org site). Pawel, we could probably modify our patch to use SQLite's change, then on next merge it will go away. Thanks, scott On Thu, Jul 29, 2010 at 10:53 PM, Dan Kennedy wrote: > > On Jul 30, 2010, at 12:44 PM, Scott

Re: [sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-07-29 Thread Scott Hess
On Thu, Jul 29, 2010 at 10:05 PM, Dan Kennedy wrote: > On Jul 30, 2010, at 8:39 AM, Paweł Hajdan, Jr. wrote: >> I'm attaching a suggested patch to verify number of arguments >> in icuRegexpFunc. Please review it. >> >> This is upstreaming of >> http://src.chromium.org/viewvc/chrome/trunk/src/third

Re: [sqlite] Database corruption on Linux ext3

2010-07-13 Thread Scott Hess
On Tue, Jul 13, 2010 at 10:03 PM, Scott Hess wrote: > On Tue, Jul 13, 2010 at 8:24 PM, Simon Slavin wrote: >> It might be useful to figure out whether we're aiming for >> detection or correction.  By 'correction' I don't mean recovery >> of all informati

Re: [sqlite] Database corruption on Linux ext3

2010-07-13 Thread Scott Hess
On Tue, Jul 13, 2010 at 8:24 PM, Simon Slavin wrote: > It might be useful to figure out whether we're aiming for > detection or correction. By 'correction' I don't mean recovery > of all information, I mean restoring the database to some state > it was in just after a COMMIT took effect. There's

Re: [sqlite] linear behavior on INSERT (was Re: unexpected large journal file)

2010-06-22 Thread Scott Hess
Eric, You should reduce your demonstration case to something you'd be willing to post the code for. Probably using synthetic data (preferably generated data). There's something missing in the thread right now, and it's unlikely to be exposed by random shots in the dark. -scott On Tue, Jun 22,

Re: [sqlite] unexpected large journal file

2010-06-18 Thread Scott Hess
On Fri, Jun 18, 2010 at 5:24 PM, Eric Smith wrote: > Jay A.  Kreibich wrote: >> Yes.  Hence the "and this is the important part" comment.  Most of >> the time when people are building billion-row files, they're building >> a new DB by importing a static source of data.  If things go wrong, >> you

Re: [sqlite] Is it possible to generate tf-idf matrix from the FTS3 table?

2010-06-07 Thread Scott Hess
On Sat, Jun 5, 2010 at 10:29 AM, Han-Teng Liao wrote: >    I intend to use my existing datasets stored in sqlite3 database for some > linguistic analysis for Chinese language. After I have successfully > installed and run the FTS3 Extension and ICU Extension, I am curious whether > it is theoretic

Re: [sqlite] MySQL vs. SQLite

2010-06-01 Thread Scott Hess
On Tue, Jun 1, 2010 at 2:37 PM, Simon Slavin wrote: > On 1 Jun 2010, at 7:11pm, Israel Lins Albuquerque wrote: >> About that future release functionality. Will be possible to know whats >> temporary index are created? >> Using that information will be easy to know what indexes we need create to

Re: [sqlite] create virtual table if not exists table_id???

2010-05-12 Thread Scott Hess
On Wed, May 12, 2010 at 10:40 AM, Roger Binns wrote: > On 05/11/2010 11:50 AM, Matt Young wrote: >> sqlite> create virtual table if not exists words using fts3  (f1 ); >> Error: near "not": syntax error > >  http://www.sqlite.org/cvstrac/tktview?tn=2604 > > To fix it requires code changes to SQLit

Re: [sqlite] FTS3 Appropriate usage

2010-04-16 Thread Scott Hess
On Fri, Apr 16, 2010 at 3:24 AM, Alexey Pechnikov wrote: > And you can use my patches for zlib-compression for FTS3. I'm planning to make > the "fts3z" extension because I want to use as original FTS3 > as FTS3 with compression together. Back when I was working up fts1, I experimented with compre

Re: [sqlite] VACUUM & journal size

2010-03-15 Thread Scott Hess
On Mon, Mar 15, 2010 at 11:18 AM, Scott Hess wrote: > AFAICT, the operation to copy the pages back _is_ journaled, and the > journal will get any pages which are overwritten in the front of the > main database.  If the initial database has half of the pages used, it > seems like the j

Re: [sqlite] VACUUM & journal size

2010-03-15 Thread Scott Hess
On Sun, Mar 14, 2010 at 5:18 PM, Jay A. Kreibich wrote: > On Sun, Mar 14, 2010 at 07:19:59PM -0400, Matthew L. Creech scratched on the > wall: >> I have a SQLite database with one large table, and I'd like to shrink >> the size of that table to free up space in the filesystem.  My problem >> is t

Re: [sqlite] Min() ; was: Re: if exist

2010-03-09 Thread Scott Hess
On Tue, Mar 9, 2010 at 10:46 AM, Tim Romano wrote: > On 3/9/2010 10:56 AM, Scott Hess wrote: >> On Tue, Mar 9, 2010 at 7:15 AM, Tim Romano  wrote: >>> Of these three: >>> >>> select c from T where 1=2                 // returns 0 rows >>> select min

Re: [sqlite] Min() ; was: Re: if exist

2010-03-09 Thread Scott Hess
On Tue, Mar 9, 2010 at 7:15 AM, Tim Romano wrote: > Of these three: > > select c from T where 1=2                 // returns 0 rows > select min(c) from T where 1=2         // returns 1 row > select min(88,99) from T where 1=2  // returns 0 rows > > the only case that "threw" me is the second one,

Re: [sqlite] FTS3 bug with MATCH plus OR

2010-03-03 Thread Scott Hess
I can't speak to the question of whether it's a real problem, but I bet you can work around with a sub-select. Something like: select * from myfts where rowid = 1 OR rowid IN (select rowid from myfts where (myfts MATCH 'one')); -scott On Wed, Mar 3, 2010 at 3:26 AM, Ralf Junker wrote: > The r

Re: [sqlite] why is underscore like dash?

2010-03-02 Thread Scott Hess
On Tue, Mar 2, 2010 at 9:41 AM, Wilson, Ronald wrote: > sqlite> select * from test where text like '_'; from http://www.sqlite.org/lang_expr.html > An underscore ("_") in the LIKE pattern matches any single character in the > string. ___ sqlite-users m

Re: [sqlite] Is there an optimization for "IS NULL" operator in a SELECT query ?

2010-03-01 Thread Scott Hess
On Mon, Mar 1, 2010 at 3:12 PM, Jean-Christophe Deschamps wrote: >>NULL = 12345 is NULL, NOT NULL is NULL, so subset N is not part of NOT >>(col = 12345). > > You're right of course!  (and I was even saying about nulls treated apart) > > But, in your view, that the set can be non-contiguous for >

Re: [sqlite] Is there an optimization for "IS NULL" operator in a SELECT query ?

2010-03-01 Thread Scott Hess
On Mon, Mar 1, 2010 at 2:44 PM, Jean-Christophe Deschamps wrote: > The actual reason for the way NOT works as for now may be due to the > fact that negating a condition may cause the resulting set to be in > fact itself the union of two subsets. > Say the "where" condition K is "col = 12345".  We

Re: [sqlite] Bug in porter stemmer

2010-02-24 Thread Scott Hess
Actually, I think a new version of the tokenizer would have to be a distinct tokenizer (ie, "porter" versus "porter1" versus "porter2", whatever). fts4 should not interpret the meaning of an explicit tokenizer differently from fts3, but it could use a different default tokenizer. [Don't take this

Re: [sqlite] Interrupt first sqlite3_step in FTS3 query

2010-02-20 Thread Scott Hess
On Sat, Feb 20, 2010 at 4:28 AM, Max Vlasov wrote: > you mentioned full-text search. > I just tried to search for mentioning of sqlite3_interrupt in the sqlite > sources > The main is the implemention of the function itself that just sets the > isInterrupted variable: > > void sqlite3_interrupt(sq

Re: [sqlite] Is there a way to "predict" autoincrement rowid range

2010-01-07 Thread Scott Hess
As mentioned, don't use raw rowid, define a "my_id INTEGER PRIMARY KEY AUTOINCREMENT". This currently acts as an alias to rowid, but if the implementation changes in the future, it will continue to work as documented. So things will be correct, but they might not be as efficient as before. Split

Re: [sqlite] Unique column constrained to 0 .. 2^32?

2009-12-30 Thread Scott Hess
Since you're asking the question, it seems safe to assume that you expect the table to have on the order of 2^32 values? And that you'll be freeing up old values by deleting rows from the table? First off, don't use rowid. Use "my_id INTEGER PRIMARY KEY" or something like that. rowids are an in

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-28 Thread Scott Hess
On Mon, Dec 28, 2009 at 8:19 PM, Dan Kennedy wrote: >> If you want the data for other purposes, you could almost implement a >> hack to get is.  For instance, "SELECT count(docid) FROM fts_table >> WHERE fts_table MATCH 'x';" is kind of close, and you could change the >> match to handle prefix stu

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-28 Thread Scott Hess
segments/nodes) while the latter tokens themselves. > > One of the problems (if such queries can be implemented technically) is the > proper language sintax. As the tokens are not columns of the virtual table, > this syntax have to be different from general Select. > > On Wed, Dec 23, 20

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-23 Thread Scott Hess
collect data from MATCH, because it may contain irrelevant, non-exising > words (but ironically it would be helpful for collecting "hit" data). If we > know inside xNext that the call from a real data appending then at least we > will be able to maintain a dictionary of words

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-23 Thread Scott Hess
The tokens are all there, so it is "theoretically possible". But the performance would be very bad because it would require iterating over all the segment data, basically a worst-case query. Even then, you wouldn't have information about hit counts. To do it reasonably would require redesigning

Re: [sqlite] The next release of SQLite....

2009-12-03 Thread Scott Hess
BTW, this is why I added docid as an alias for rowid for fts3 (rowid-versus-VACUUM was the initial reason for revving fts2 to fts3). Saying "rowid is just like rowid in other tables, except different" seemed more confusing than it was worth. Though since I'm not writing the new documentation, I'm

[sqlite] Is the dbFileVers written to journal files?

2009-11-10 Thread Scott Hess
Someone was asking me a question about what happens if the wrong journal file gets applied to a database. My answer was that terrible things happen. I found myself wondering if the "File change counter" value is stored in the journal (*). It would seem like a cheap(ish) way to give some reasonab

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
fts3 columns are all implicitly TEXT, no matter how you dress them up. There's already docid as a primary-key alias for rowid. -scott On Wed, Nov 4, 2009 at 11:03 AM, Simon Slavin wrote: > > On 4 Nov 2009, at 5:05pm, sorka wrote: > >> Hmm. Have you actually tried this yourself? >> >> Here's wh

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
way be a few duplicates out of the hundreds of > records so it will always fail. For whatever reason, the delete, even though > it's just 2 or 3 records is taking 10 times longer than just the insert > alone where I can artificially make a case where there are no duplicates. > >

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
On Tue, Nov 3, 2009 at 9:12 PM, sorka wrote: > Is there any way to have an intsert into an FTS3 table ignore a row if the > ROWID being inserted already exists? This is turning out to be quite > troublesome because I'm inserting thousands of records where just a few like > 3 or 4 will have the sam

<    1   2   3   4   5   >