[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread R Smith
On 2016/04/16 9:25 PM, Cecil Westerhof wrote: > 2016-04-16 20:36 GMT+02:00 R Smith : > > I am filling the database again, but now with text UUID instead of blob > UUID. That takes a ?little? more time. When it is filled I try again. Don't forget to copy the DB file once it is populated, that way

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 1:03 GMT+02:00 Keith Medcalf : > > Have another problem also. My CPU is about 15%, but the load average is > > also about 15. (This is on a Linux system.) This results (sometimes) in a > > very sluggish system. Can the load be a SQLite problem, or is it a Java > > problem? (When the pro

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 4:16 GMT+02:00 R Smith : > > > On 2016/04/16 9:25 PM, Cecil Westerhof wrote: > >> 2016-04-16 20:36 GMT+02:00 R Smith : >> >> I am filling the database again, but now with text UUID instead of blob >> UUID. That takes a ?little? more time. When it is filled I try again. >> > > Don't forg

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Rob Willett
Cecil,, Linux reporting 8 cores is due to hyper threading on the four cores. Thats normal. One of the cores reporting a high usage is normal if you have single threaded app that simply cannot be moved, e.g. many perl programs exhibit this behaviour. Thats fine and to be expected. I do not hav

[sqlite] The LIKE optimization breaks user defined like functions

2016-04-17 Thread Clemens Ladisch
Domingo Alvarez Duarte wrote: > Today I decided to see why my applications stop using index when using > my user defined "like" function The optimization replaces the LIKE with two comparisons. Installing a user-defined function implies that your new LIKE works differently, so this optimization m

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 10:13 GMT+02:00 Rob Willett : > I do not have the whole thread for what you reported but I did read > somebody ask if you have put your inserts between a BEGIN/END transaction. > That will make a massive difference to your speed. Also I?m unclear as to > how Java fits in all of this. P

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Simon Slavin
On 17 Apr 2016, at 10:38am, Cecil Westerhof wrote: > I start with a: >? > > ?conn.setAutoCommit(false); > but that is not the same? Yes, that does the same as BEGIN ... END. At least, according to the documentation it does. But you caused me to look up how the JDBC works, especially for

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 12:03 GMT+02:00 Simon Slavin : > > On 17 Apr 2016, at 10:38am, Cecil Westerhof > wrote: > > > I start with a: > >? > > > > ?conn.setAutoCommit(false); > > but that is not the same? > > Yes, that does the same as BEGIN ... END. At least, according to the > documentation it does. >

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
All, Any thoughts on using the first byte of a BLOB to indicate what kind of data that BLOB contains? For example: 0 = plain old bytes 1 = 16 bit integers 2 = 32 bit integers 3 = singles 4 = doubles, etc. I am forced to consider this approach because my function will be receiving blobs but wil

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Ketil Froyn
I'm not sure I understand why you can't specify the type in a separate column, rather than prepending it to the blob? That seems like a more flexible way to have access to the information. Regards, Ketil On 17 Apr 2016 2:02 p.m., "William Drago" wrote: > All, > > Any thoughts on using the first

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Simon Slavin
On 17 Apr 2016, at 1:02pm, William Drago wrote: > I am forced to consider this approach because my function will be receiving > blobs but will have no idea what those blobs are. Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's an integer, a string, a car, a peace treaty or

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Brad Stiles
An ADO.NET DataTable *can* have type information for table columns, if you add a parameter to the WriteXml method to have it write the schema along with the data. Obviously, *their* data must be typed from their source, or be inferred, for this to work. See if your provider can do this for you

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Keith Medcalf
Perfectly linear. The time waster in creating the records is the index with the completely separate copy of all the data and the native primary key (record number) into a duplicate structure (the index btree). Creating the index and the base table at the same time, while linear as well, is ev

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:20 AM, Ketil Froyn wrote: > I'm not sure I understand why you can't specify the type in a separate > column, rather than prepending it to the blob? That seems like a more > flexible way to have access to the information. I have no control over the database schema, table structure,

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:45 AM, Simon Slavin wrote: > On 17 Apr 2016, at 1:02pm, William Drago wrote: > >> I am forced to consider this approach because my function will be receiving >> blobs but will have no idea what those blobs are. > Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 10:13 AM, Brad Stiles wrote: > An ADO.NET DataTable *can* have type information for table columns, if you > add a parameter to the WriteXml method to have it write the schema along with > the data. Obviously, *their* data must be typed from their source, or be > inferred, for this t

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 17:13 GMT+02:00 Keith Medcalf : > > Perfectly linear. The time waster in creating the records is the index > with the completely separate copy of all the data and the native primary > key (record number) into a duplicate structure (the index btree). Creating > the index and the base t

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Simon Slavin
On 17 Apr 2016, at 5:13pm, Cecil Westerhof wrote: > Or is this normal > in Java programs? Yes. You're discovering that Java is rubbish. 'futex' for Java is what most environments call 'mutex' and Java doesn't do locking well. It's this which is causing your delays, not SQLite or anything a

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Cecil Westerhof
2016-04-17 18:13 GMT+02:00 Cecil Westerhof : > 2016-04-17 17:13 GMT+02:00 Keith Medcalf : > The strange thing is that the blob variant takes a lot of time now also. > First it took only 4? hour, now it is already busy for eight hours and only > has come to 8.9E7. > > 14:36:01: Inserted8.40

[sqlite] Is it possible that dropping a big table takes very long

2016-04-17 Thread Scott Robison
On Sun, Apr 17, 2016 at 1:44 PM, Cecil Westerhof wrote: > 2016-04-17 18:13 GMT+02:00 Cecil Westerhof : > > > 2016-04-17 17:13 GMT+02:00 Keith Medcalf : > > The strange thing is that the blob variant takes a lot of time now also. > > First it took only 4? hour, now it is already busy for eight hou

[sqlite] Avoid duplicate sets with join table

2016-04-17 Thread James K. Lowden
On Sat, 16 Apr 2016 01:20:55 +0200 Ketil Froyn wrote: > I have two tables and a join table, in principle like this: > > CREATE TABLE records (id INTEGER PRIMARY KEY, data TEXT); > CREATE TABLE features (id INTEGER PRIMARY KEY, data TEXT UNIQUE); > CREATE TABLE records_features (id_r INTEGER, id_

[sqlite] Caveat entry

2016-04-17 Thread James K. Lowden
On Fri, 15 Apr 2016 22:53:57 +0100 Simon Slavin wrote: > On 15 Apr 2016, at 10:05pm, Cecil Westerhof > wrote: > > * SQLite datatypes and how SQLite decides which datatype you want > * SQLite uses affinities not column types > * thinking you should index each column instead of indexes for queries

[sqlite] Caveat entry

2016-04-17 Thread Keith Medcalf
> The best database performance advice I ever got still applies: think > about how the DBMS will traverse the data, and minimize the work it > will do. It's all about I/O. No number of additional threads will > bring the data into memory sooner. The only way to optimize I/O is to not do it.

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:20 AM, Ketil Froyn wrote: > I'm not sure I understand why you can't specify the type in a separate > column, rather than prepending it to the blob? That seems like a more > flexible way to have access to the information. I have no control over the database schema, table structure,

[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:45 AM, Simon Slavin wrote: > On 17 Apr 2016, at 1:02pm, William Drago wrote: > >> I am forced to consider this approach because my function will be receiving >> blobs but will have no idea what those blobs are. > Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's