Re: [sqlite] Copy-on-write VFS

2019-10-07 Thread Fredrik Larsen
ill look more into the WAL-system when I get some time. If someone know for sure what I'm planning to do will crash and burn, I will appriciate a heads-up on why. Fredrik On Mon, Oct 7, 2019 at 11:42 AM Simon Slavin wrote: > On 7 Oct 2019, at 10:34am, Fredrik Larsen wrote: > > >

Re: [sqlite] Copy-on-write VFS

2019-10-07 Thread Fredrik Larsen
n, Oct 7, 2019 at 11:16 AM Simon Slavin wrote: > On 7 Oct 2019, at 9:56am, Fredrik Larsen wrote: > > > nowhere does it warn about severe regression with > > unbounded WAL-size [snip] > > There are tons of bad stuff the documentation doesn't warn you about. You > migh

Re: [sqlite] Copy-on-write VFS

2019-10-07 Thread Fredrik Larsen
sable checkpointing? > > -- > The fact that there's a Highway to Hell but only a Stairway to Heaven says > a lot about anticipated traffic volume. > > >-Original Message- > >From: sqlite-users On > >Behalf Of Fredrik Larsen > >Sent: Friday, 4 October, 2019

Re: [sqlite] Copy-on-write VFS

2019-10-04 Thread Fredrik Larsen
, simplifying backups and the effect of data-corrupting errors. Fredrik On Fri, Oct 4, 2019 at 3:44 PM Simon Slavin wrote: > On 4 Oct 2019, at 12:17pm, Fredrik Larsen wrote: > > > A copy-on-write IO-path where data is split into static and dynamic > parts (think snapshots for

Re: [sqlite] Copy-on-write VFS

2019-10-04 Thread Fredrik Larsen
else had explored this path. From the feedback so far it seems not. Fredrik On Fri, Oct 4, 2019 at 3:23 PM test user wrote: > Hello Fredrik, > > Why does it need to be part of a VFS instead of using a file system with > COW like ZFS? > > On Fri, 4 Oct 2019 at 12:18, Fre

[sqlite] Copy-on-write VFS

2019-10-04 Thread Fredrik Larsen
Hi A copy-on-write IO-path where data is split into static and dynamic parts (think snapshots for storage) would be very helpful for our project, . This would simplify backups, testing, moving data around in a multinode environment, etc. Does something like this exist for sqlite? In my head this

Re: [sqlite] Lookup join

2019-10-01 Thread Fredrik Larsen
I have run analyze on production data, that should work better, right? Or do you propose to "trick" the query-planner using some kind of staging-data? Anyway, I use INDEXED BY hints now, and this solves my problem. Fredrik On Tue, Oct 1, 2019 at 8:57 PM Simon Slavin wrote: > When trying to

Re: [sqlite] Lookup join

2019-10-01 Thread Fredrik Larsen
queries are below 10ms on fairly big data-sets, from 1-2 seconds. That is an improvement of several orders of magnitude :) Fredrik On Tue, Oct 1, 2019 at 8:49 PM Keith Medcalf wrote: > > On Tuesday, 1 October, 2019 11:58, Fredrik Larsen > wrote: > > >Thanks Keith! I have

Re: [sqlite] Lookup join

2019-10-01 Thread Fredrik Larsen
the indexing system would be better. But when the SQL-translation works as expected, it is nice to have sql available as a tool for general reporting. Fredrik On Tue, Oct 1, 2019 at 8:05 PM Richard Hipp wrote: > On 10/1/19, Fredrik Larsen wrote: > > > > The declarative model

Re: [sqlite] Lookup join

2019-10-01 Thread Fredrik Larsen
Thanks Keith! I have spent several days trying to tune my query towards expected performance, without luck. I somehow missed your fairly straight forward solution. I still have some problems making sqlite use the correct indexes, but this can at least be fixed by well-placed INDEXED-BY-hints. The

[sqlite] Lookup join

2019-09-30 Thread Fredrik Larsen
Consider query below; SELECT key FROM t1 LEFT JOIN ( SELECT key,max(rev),data FROM t2 WHERE rev < ? GROUP BY key ) USING (key) ORDER BY key ? LIMIT ? In above query sqlite will materialize the t2-sub-query and then start working on the outer query. I have a lot of data in t2 so this will

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
Good to know that I was not to far off target then. But fixing issues in less than a day of reporting? On a Saturday? Who does that? I was planning to feel happy about solving this issue.. :) Fredrik On Sat, Sep 21, 2019 at 9:31 PM Dan Kennedy wrote: > > On 22/9/62 02:25, Fredrik Larsen

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
something :) Fredrik On Sat, Sep 21, 2019 at 8:49 PM Keith Medcalf wrote: > > See Dan's checkin on trunk for this issue. > > https://www.sqlite.org/src/info/20f7951bb238ddc0 > > >-Original Message- > >From: sqlite-users On > >Behalf Of Fredrik Larse

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
red to stop SQLite from assuming to much about GROUP-BY queries. Fredrik On Sat, Sep 21, 2019 at 4:12 PM Fredrik Larsen wrote: > Your last sentence got me thinking. So I downloaded the source, modified > the ordering of the GROUP-BY expression to match ORDER-BY and it works! > This will of

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
Your last sentence got me thinking. So I downloaded the source, modified the ordering of the GROUP-BY expression to match ORDER-BY and it works! This will offcourse only work if the GROUP-BY and ORDER-BY matches generally expect for the direction. This fix only improves performance for relevant

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Fredrik Larsen
M R Smith wrote: > On 2019/09/20 2:49 PM, Fredrik Larsen wrote: > > Hi Ryan > > > > Nobody is proposing that QP should automagically add an index, I'm only > > asking why the QP does not use already added index, that is specially > added > > for this specific case. I

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Fredrik Larsen
Hi Ryan Nobody is proposing that QP should automagically add an index, I'm only asking why the QP does not use already added index, that is specially added for this specific case. I don't thinks this is a very "obscurest of use-case" or to much to ask for, in fact, this is the expected behavior

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-19 Thread Fredrik Larsen
gt; -Ursprüngliche Nachricht- > Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] > Im Auftrag von Fredrik Larsen > Gesendet: Donnerstag, 19. September 2019 14:14 > An: sqlite-users@mailinglists.sqlite.org > Betreff: [EXTERNAL] [sqlite] Group-by and order

[sqlite] Group-by and order-by-desc does not work as expected

2019-09-19 Thread Fredrik Larsen
/questions/58009898/sqlite-group-by-with-sort-by-desc-does-not-work-as-expected Fredrik Larsen ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users