Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Looks interesting, but if I'm reading the descriptions right I don't think those would help: my partitioning can be on a hash of a unique text ID or on a geographic area, and both of those seem to be based on unique rowid ranges. On Thu, Aug 22, 2019 at 1:23 PM Keith Medcalf wrote: > > Have you

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Keith Medcalf
Have you looked at the swarmvtab or unionvtab extension? https://www.sqlite.org/unionvtab.html https://www.sqlite.org/swarmvtab.html Which can "avoid" having to write your own unions. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Still a bit over 3x slower on queries but that's a 7x performance improvement. On Thu, Aug 22, 2019 at 11:40 AM Peter da Silva wrote: > Legit. I'll try that. > > On Thu, Aug 22, 2019 at 11:33 AM David Raymond > wrote: > >> I don't know how smart the planner is, but as a thought, would UNION ALL

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Originally Tcl/native Tcl binding, now a C++ extension calling the C-binding that's a Tcl extension itself. On Thu, Aug 22, 2019 at 11:17 AM test user wrote: > What language/binding library are you using? > > On Thu, 22 Aug 2019 at 16:45, Peter da Silva wrote: > > > Database is on tmpfs and per

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Legit. I'll try that. On Thu, Aug 22, 2019 at 11:33 AM David Raymond wrote: > I don't know how smart the planner is, but as a thought, would UNION ALL > make any improvement over just UNION? With just UNION it has to > de-duplicate all the subquery results whereas with UNION ALL it would be > fr

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread David Raymond
I don't know how smart the planner is, but as a thought, would UNION ALL make any improvement over just UNION? With just UNION it has to de-duplicate all the subquery results whereas with UNION ALL it would be free to separate all the various subqueries from each other. Or do you actually need

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread test user
What language/binding library are you using? On Thu, 22 Aug 2019 at 16:45, Peter da Silva wrote: > Database is on tmpfs and periodically snapshotted to SSD. There are > bottlenecks upstream of sqlite that we can see in traces. > > On Thu, Aug 22, 2019 at 10:36 AM Warren Young wrote: > > > On Au

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Database is on tmpfs and periodically snapshotted to SSD. There are bottlenecks upstream of sqlite that we can see in traces. On Thu, Aug 22, 2019 at 10:36 AM Warren Young wrote: > On Aug 22, 2019, at 9:27 AM, Peter da Silva wrote: > > > > Have an existing application that's pushing the limit >

Re: [sqlite] Attached databases and union view.

2019-08-22 Thread Warren Young
On Aug 22, 2019, at 9:27 AM, Peter da Silva wrote: > > Have an existing application that's pushing the limit If the limit is in hardware, shards won’t help. For example, a SQLite DB on a 7200 RPM spinning disk is limited to about 60 transactions per second under the stock SQLite fsync logic,

[sqlite] Attached databases and union view.

2019-08-22 Thread Peter da Silva
Have an existing application that's pushing the limit on how fast it can read data and add it to the database, and thinking of sharding the database file so I can have multiple writers writing to shards of the main tables. ATTACH DATABASE 'shard0.sqlite' as shard0; ATTACH DATABASE 'shard1.sqlite'

Re: [sqlite] conditional trigger

2019-08-22 Thread David Raymond
Your create view had 2 typos in it by the way. Here's a fixed version for those copying and pasting: CREATE VIEW select_file_dirs AS SELECT dir_paths.dir_path AS dir_path, files.f_name FROM dir_paths INNER JOIN files ON files.dir_id = dir_paths.id;

[sqlite] conditional trigger

2019-08-22 Thread InAsset.Michele Petrazzo
Hi all, I'm trying to create a conditional trigger for add some data to my sqlite db only when it isn't inside (yet). My tables: CREATE TABLE dir_paths ( id integer PRIMARY KEY, dir_path TEXT NOT NULL UNIQUE ); CREATE TABLE files ( id INTEGER PRIMARY KEY, f_name TEXT NOT NULL,