Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Chris Locke
I don't know - that's why I asked. Thanks for the clarification. On Mon, Jan 28, 2019 at 10:10 PM Warren Young wrote: > On Jan 28, 2019, at 2:44 PM, Chris Locke wrote: > > > >> The table name should not be meaningful to your application; nothing in > >> your application should conjure up a ta

Re: [sqlite] Claimed vulnerability in SQLite: Info or Intox?

2019-01-29 Thread Vladimir Barbu
We do use FTS3 and don't provide execution of arbitrary SQL in our product code (of course, SQL injection is also not possible), but clients could write their own customizations via plugins. --- Vladimir -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread mzzdvd
Dear all, what happens if I put all data in a single table and this table become very huge (for example millions of rows)? Will I have same performace problems? Thanks. Regards. > > Il 28 gennaio 2019 alle 17.28 Simon Slavin ha > scritto: > > On 28 Jan 2019, at 4:17pm, mzz...@lib

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Chris Locke
Ryan Smith has already covered this scenario. "And to add to the slew of "Few-tables-many-rows rather than Many-tables-few-rows" solutions offered, one thing to note: After a single table contains a few million rows, INSERTing will become slightly slower, but the difference will still be measured

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Simon Slavin
On 29 Jan 2019, at 11:00am, mzz...@libero.it wrote: > what happens if I put all data in a single table and this table become very > huge (for example millions of rows)? > > Will I have same performace problems? I used to use a 43 Gigabyte SQLite database of which 42 Gigabytes was taken up with

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Andy Bennett
Hi, what happens if I put all data in a single table and this table become very huge (for example millions of rows)? Will I have same performace problems? The INSERTing profile has been covered by others but I'll just add something about SELECT as well. It depends on how you query it. i.e

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Rob Willett
Millions of rows is not even large, never mind huge or very huge :) We have tables with hundreds of millions of rows, we got to billions of rows in a single table before we changed the logic. From memory we had 67GB for a single database and I reckon 60GB was one table. Not many issues at all

Re: [sqlite] Custom aggregate functions in Tcl

2019-01-29 Thread Richard Hipp
On 1/29/19, Andy Goth wrote: > > Question: does xFinal() get called if an error occurs during (or between) > calling xStep()? Are errors even possible? I'm curious if there's any way > to leak the Tcl_Obj pointed to by the aggregate context. xFinal() gets called by sqlite3_reset() or sqlite3_fina

[sqlite] SQLITE_DBCONFIG_DEFENSIVE documentation looks confusing

2019-01-29 Thread Chris Brody
I am very sorry to say that I have found the usage of the SQLITE_DBCONFIG_DEFENSIVE option to be somewhat confusing. From my first reading of https://www.sqlite.org/releaselog/3_26_0.html I thought SQLITE_DBCONFIG_DEFENSIVE was a compile-time option. (I was proven wrong pretty quickly.) Then I fo

[sqlite] Displaying hierarchical structure

2019-01-29 Thread Bart Smissaert
Working on an Android app and part of that is storing SQL in a virtual folder system in SQLite. For this I want to use a so-called closure table as explained nicely here: http://technobytz.com/closure_table_store_hierarchical_data.html I have a table holder the folder details: ID PARENT_ID Folde

Re: [sqlite] Custom aggregate functions in Tcl

2019-01-29 Thread Andy Goth
Oh yeah, I meant to say that I was going to leave window functions for future expansion. First I need to get more familiar with their use. Yesterday was my first time implementing an aggregate function, and I need to work my way up. On Tue, Jan 29, 2019, 07:46 Richard Hipp On 1/29/19, Andy Goth

Re: [sqlite] SQLITE_DBCONFIG_DEFENSIVE documentation looks confusing

2019-01-29 Thread Chris Brody
I think my sample code should have read as follows: To activate the "defensive" flag for a database connection: sqlite3_db_config(db, SQLITE_DBCONFIG_DEFENSIVE, 1, NULL); (I got the wrong prefix before, and I discovered that it crashes if I do not add the NULL argument.) I hope I got this right,

Re: [sqlite] Custom aggregate functions in Tcl

2019-01-29 Thread Gerry Snyder
I hope your work makes into the SQLite source code. It will be useful. Gerry Snyder On Tue, Jan 29, 2019 at 12:16 AM Andy Goth wrote: > I wish to define custom aggregate functions in Tcl, but this capability is > currently not exposed through the Tcl interface. Thus I am thinking about > how be

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread mzzdvd
thanks a lot for fast answers. Now I'll try to modify my DB structure to use one table. Thanks to all. Bye. > > Il 29 gennaio 2019 alle 12.29 Simon Slavin ha > scritto: > > On 29 Jan 2019, at 11:00am, mzz...@libero.it wrote: > > > > > > what happens if I put all da

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread David Raymond
I may be missing something, but with recursive CTE's, why do you need the second table? I'm assuming it's just to speed things up once the counts get large? sqlite> create table folders (id integer primary key, parent_id int references folders, name text not null collate nocase, check (not (pa

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread Bart Smissaert
Thanks for that, very nice indeed. The second table is used for other purposes. I think the depth column speeds up certain queries. RBS On Tue, Jan 29, 2019 at 5:09 PM David Raymond wrote: > I may be missing something, but with recursive CTE's, why do you need the > second table? I'm assuming i

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread Bart Smissaert
Had another look at your solution and not sure now it is quite OK. The output comes out fine, but that seems to be due to the lucky fact that it just sort OK by folder. If I change the folder names then the output is not OK. RBS On Tue, Jan 29, 2019 at 5:09 PM David Raymond wrote: > I may be mi

Re: [sqlite] SEE Temp Files

2019-01-29 Thread Jens Alfke
> On Jan 28, 2019, at 3:35 PM, Richard Hipp wrote: > > On the other > hand, if you set SQLITE_TEMP_STORE=2 so that intermediate results are > held in memory, then intermediate results may be written to swap space > when the device gets under memory pressure. Mobile OSs don’t swap. (iOS for ce

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread David Raymond
Yup, you're right. Will have to think some more on that then. -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Bart Smissaert Sent: Tuesday, January 29, 2019 1:06 PM To: SQLite mailing list Subject: Re: [sqlite] Displaying hierarchi

Re: [sqlite] SQLITE_DBCONFIG_DEFENSIVE documentation looks confusing

2019-01-29 Thread Dan Kennedy
On 29/1/62 23:18, Chris Brody wrote: I think my sample code should have read as follows: To activate the "defensive" flag for a database connection: sqlite3_db_config(db, SQLITE_DBCONFIG_DEFENSIVE, 1, NULL); (I got the wrong prefix before, and I discovered that it crashes if I do not add the N

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread J Decker
Probably just the order by just do 'order by parent_id,name' to group folders together and then alphabetcal? (was expanding it... I would also start with where parent_id=0 ) with recursive foo (id, parent_id, name) as ( select * from folders where parent_id = 0 union all select folders.i

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread Keith Medcalf
See https://sqlite.org/lang_with.html which includes how to traverse the recursive tree in either depth-first or breadth-first order. Why do you need the closure table at all? create table folders ( idinteger primary key, parent_id integer references folders, name tex

Re: [sqlite] Displaying hierarchical structure

2019-01-29 Thread Bart Smissaert
Yes, thanks, -- breadth first does the job nicely indeed. Not sure the closure table is needed. There are some complex tasks though, that need thinking of, eg copying one folder into an other folder. I am not familiar with recursive queries and it looks complex to me. There might arise a problem wi

Re: [sqlite] SEE Temp Files

2019-01-29 Thread Scott Perry
> On Jan 29, 2019, at 10:12, Jens Alfke wrote: > >> On Jan 28, 2019, at 3:35 PM, Richard Hipp wrote: >> >> On the other >> hand, if you set SQLITE_TEMP_STORE=2 so that intermediate results are >> held in memory, then intermediate results may be written to swap space >> when the device gets und

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread James K. Lowden
On Tue, 29 Jan 2019 12:00:49 +0100 (CET) mzz...@libero.it wrote: > what happens if I put all data in a single table and this table > become very huge (for example millions of rows)? Big tables are your friend, actually. A binary search on 1 million rows requires at most 20 operations. A bi

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Wout Mertens
I always have to explain to people that there's no magic sauce that "real databases" add versus SQLite. SQLite uses the same techniques all databases use, and thanks to the absence of a network later, you avoid a lot of latency, so it can actually be faster. (I do believe that SQLite optimizes fo

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Keith Medcalf
On Tuesday, 29 January, 2019 16:28, Wout Mertens wrote: >To: SQLite mailing list >Subject: Re: [sqlite] SQLite slow when lots of tables > > I always have to explain to people that there's no magic sauce that > "real databases" add versus SQLite. > SQLite uses the same techniques all databases use

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Igor Korot
Hi, On Tue, Jan 29, 2019 at 7:06 PM Keith Medcalf wrote: > > On Tuesday, 29 January, 2019 16:28, Wout Mertens wrote: > > >To: SQLite mailing list > >Subject: Re: [sqlite] SQLite slow when lots of tables > > > > I always have to explain to people that there's no magic sauce that > > "real database

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Peter da Silva
On Tue, Jan 29, 2019, 7:46 PM Igor Korot You can install mySQL/MariaDB for free and use it for your needs. > I believe that if you pay to Oracle/MariaDB Foundation, it will be > just for support. (I may be wrong though). > Or better, PostgreSQL. We have a system that uses SQLite as an optional ca

Re: [sqlite] SQLite slow when lots of tables

2019-01-29 Thread Igor Korot
Peter, On Tue, Jan 29, 2019 at 7:53 PM Peter da Silva wrote: > > On Tue, Jan 29, 2019, 7:46 PM Igor Korot > > You can install mySQL/MariaDB for free and use it for your needs. > > I believe that if you pay to Oracle/MariaDB Foundation, it will be > > just for support. (I may be wrong though). >