Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Max Vlasov
On Sun, Feb 13, 2011 at 8:53 PM, Jim Wilcoxson wrote: > > But I thought about how it would be possible to test this explanation . > I'm > > going to do some tests that works like this. The same id updating, but in > > the middle of 10,000 operation I will unplug the cord, the sqlite will > say >

Re: [sqlite] basic rtree question

2011-02-13 Thread Jay A. Kreibich
On Sun, Feb 13, 2011 at 04:18:25PM -0600, Puneet Kishor scratched on the wall: > On Sunday, February 13, 2011 at 4:15 PM, Jay A. Kreibich wrote: > > > CREATE VIRTUAL TABLE points_rtree USING rtree ( > > > id, min_lon, min_lat, max_lon, max_lat > > > ); > > > > Almost. You need to match min/max p

Re: [sqlite] BUG: capi3e.test when compiled with SQLITE_OMIT_UTF16

2011-02-13 Thread Shane Harrelson
Thanks.Hopefully corrected here: http://www.sqlite.org/src/info/b04304b967 -Shane On Fri, Feb 11, 2011 at 4:59 PM, Noah Hart wrote: > > capi3e.test needs >      ifcapable utf16 logic before capi3e-2.1.$i > to properly pass tests when compiled with SQLITE_OMIT_UTF16 > > ~ Noah Hart > > > --

Re: [sqlite] Question about having 2 db connections on same thread

2011-02-13 Thread Pavel Ivanov
> I do a transaction on connection 1 using BEGIN IMMEDIATE, some rows updated, > COMMIT. > Then, when I attempt same sequence on connection 2, when I do first call to > step > to execute begin transaction and it never returns. Do you check result code from COMMIT and is it successful? My guess is

Re: [sqlite] Question about having 2 db connections on same thread

2011-02-13 Thread Simon Slavin
On 14 Feb 2011, at 12:30am, Vannus wrote: > Are you not able to use the same db connection in both classes? > > something like > FirstTablesClass->dbConnection = MyDBConnectionClass > SecondTablesClass->dbConnection = MyDBConnectionClass Nevertheless, having two connections should work too. Hi

Re: [sqlite] Question about having 2 db connections on same thread

2011-02-13 Thread Vannus
Are you not able to use the same db connection in both classes? something like FirstTablesClass->dbConnection = MyDBConnectionClass SecondTablesClass->dbConnection = MyDBConnectionClass On 13 February 2011 23:04, jeff archer wrote: > I thought this would be OK but now that I am seeing it fail I

Re: [sqlite] Speed up count(distinct col)

2011-02-13 Thread Random Coder
On Feb 13, 2011, at 3:38 PM, BareFeetWare wrote: > On 13/02/2011, at 1:04 AM, Yuzem wrote: > >> I am grabbing the data from the each movie imdb webpage. > > Does IMDB allow use of their data this way? After my brief reading of their > site, I thought they charge a $15k minimum per year for dat

Re: [sqlite] Speed up count(distinct col)

2011-02-13 Thread BareFeetWare
On 13/02/2011, at 1:04 AM, Yuzem wrote: > I am grabbing the data from the each movie imdb webpage. Does IMDB allow use of their data this way? After my brief reading of their site, I thought they charge a $15k minimum per year for data. > The script is written in bash and I can give you the cod

[sqlite] Question about having 2 db connections on same thread

2011-02-13 Thread jeff archer
I thought this would be OK but now that I am seeing it fail I am having my doubts.  I have created 2 db connections on the same thread, both on the same file.  i.e. 2 calls to sqlite3_open16() (use exact same filename both times) have created 2 sqlite3 instances.  I do this because I have 2

Re: [sqlite] basic rtree question

2011-02-13 Thread Puneet Kishor
On Sunday, February 13, 2011 at 4:15 PM, Jay A. Kreibich wrote: > I have a simple table like so > > > > CREATE TABLE points ( > > id INTEGER PRIMARY KEY, > > lon REAL, > > lat REAL, > > tile INTEGER > > ); > > > > It has about 13.25 million rows. I want to be able to return rows > > given a bo

Re: [sqlite] basic rtree question

2011-02-13 Thread Jay A. Kreibich
On Sun, Feb 13, 2011 at 11:47:48AM -0600, Puneet Kishor scratched on the wall: > I have a simple table like so > > CREATE TABLE points ( > id INTEGER PRIMARY KEY, > lon REAL, > lat REAL, > tile INTEGER > ); > > It has about 13.25 million rows. I want to be able to return rows > given a bounding b

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Simon Slavin
On 13 Feb 2011, at 5:53pm, Jim Wilcoxson wrote: > I also found this page, used to force a Linux system crash: > > http://evuraan.blogspot.com/2005/06/how-to-force-paniccrash-in-linux.html > > I seem to remember a post that SQLite commit/sync is tested with the kill > command, but it seems like

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Jim Wilcoxson
On Sun, Feb 13, 2011 at 11:55 AM, Max Vlasov wrote: > On Sun, Feb 13, 2011 at 5:31 PM, Jim Wilcoxson wrote: > > > On Sun, Feb 13, 2011 at 1:15 AM, Max Vlasov > wrote: > > > So the final results: > > > - the db was always ok and contains the correct value (id=10001 for > > initial > > > 1). > >

[sqlite] basic rtree question

2011-02-13 Thread Puneet Kishor
I have a simple table like so CREATE TABLE points ( id INTEGER PRIMARY KEY, lon REAL, lat REAL, tile INTEGER ); It has about 13.25 million rows. I want to be able to return rows given a bounding box (min_lon, min_lat, max_lon, max_lat). Is the following the right strategy? CREATE VIRTUAL TABLE

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Jean-Christophe Deschamps
>But what I postulate is that you can't physically write *the same* record >over and over more than 90 times per second on a 5400 rpm drive, >unless the >drive, OS, or filesystem implements something like wear-leveling, >where the >physical location of sectors is constantly changing. It's still

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Max Vlasov
On Sun, Feb 13, 2011 at 5:31 PM, Jim Wilcoxson wrote: > On Sun, Feb 13, 2011 at 1:15 AM, Max Vlasov wrote: > > So the final results: > > - the db was always ok and contains the correct value (id=10001 for > initial > > 1). > > - the speed was about 227 commits per second so very close to my syst

[sqlite] Query optimizer with attached databases

2011-02-13 Thread Felix Schlesinger
Hello, I have come across several situations in which the sqlite query optimizer chooses clearly suboptimal query plans when a query spans several attached databases. If I copy all involved tables into one sqlite database file, a better query plan is chosen. The difference can be quite dramatic (o

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Jim Wilcoxson
On Sun, Feb 13, 2011 at 7:07 AM, Jean-Christophe Deschamps wrote: > > >So if the drive is 5400 rpm, 227 is much more than 5400/60=90 and even if > >it's 7200 (manufacturers sometimes upgrade drives inside portable hd > >without > >prior notice), it's still twice as much as 7200/60=120. > > 5400/6

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Jim Wilcoxson
On Sun, Feb 13, 2011 at 1:15 AM, Max Vlasov wrote: > On Sat, Feb 12, 2011 at 7:11 PM, Jim Wilcoxson wrote: > > > > > > > Unless I'm missing something, SQLite has to update the first page of the > > database on every commit, to update the change counter. Assuming you are > > using rotating media

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-13 Thread Jean-Christophe Deschamps
>So if the drive is 5400 rpm, 227 is much more than 5400/60=90 and even if >it's 7200 (manufacturers sometimes upgrade drives inside portable hd >without >prior notice), it's still twice as much as 7200/60=120. 5400/60, 7200/60 ... those values rely on the assumption that successive LBAs are ma

Re: [sqlite] SQLite version 3.7.6 testing

2011-02-13 Thread Sven L
Lol... we just had a long discussion about sqlite_stat2 and ANALYZE command. I removed these from my code, since the command should not be used except for rare cases :P Should I re-add my ANALYZE-code and ask my original question again? Confused :P > From: d...@sqlite.org > Date: Sat, 12 Feb