Re: [Cuirass] Missing database indexes?

2018-12-19 Thread Amirouche Boubekki
Did you consider using wiredtiger? Le lun. 19 nov. 2018 à 11:47, Danny Milosavljevic a écrit : > Hi Ludo, > > >It doesn’t seem to help much, perhaps because the query is too complex? > > Yeah, probably. > > According to the docs a log message is supposed to appear when it is doing > it. > > We

Re: [Cuirass] Missing database indexes?

2018-11-19 Thread Danny Milosavljevic
Hi Ludo, >It doesn’t seem to help much, perhaps because the query is too complex? Yeah, probably. According to the docs a log message is supposed to appear when it is doing it. We should just special-case the common queries so the optimizer has a easier life. pgp8qd5F8FXLx.pgp Description:

Re: [Cuirass] Missing database indexes?

2018-11-19 Thread swedebugia
Hi On 2018-11-10 18:33, Ludovic Courtès wrote: snip --8<---cut here---start->8--- sqlite> CREATE INDEX Builds_index_evaluation ON Builds(evaluation); snip Anything else we should do? I woke up today with an idea. :) Now on berlin it seems that all

Re: [Cuirass] Missing database indexes?

2018-11-16 Thread Björn Höfling
On Tue, 13 Nov 2018 09:10:30 +0100 Clément Lassieur wrote: > Hi Björn, > > Björn Höfling writes: > > > We are directly relying on the rowid here, there is no explicit > > id-column. > > > > This could lead to unpredicted results and reorderings (6th Quirk in > > document): > > > >

Re: [Cuirass] Missing database indexes?

2018-11-16 Thread Björn Höfling
On Tue, 13 Nov 2018 00:27:15 +0100 Danny Milosavljevic wrote: > Hi Björn, > > On Mon, 12 Nov 2018 19:50:44 +0100 > Björn Höfling wrote: > > > Hm. This code smells ... It looks too complicated. > > I was trying to cut down the number of prepared statements in use and > prevent a

Re: [Cuirass] Missing database indexes?

2018-11-14 Thread Ludovic Courtès
Hello, Danny Milosavljevic skribis: > On Mon, 12 Nov 2018 19:50:44 +0100 > Björn Höfling wrote: > >> Hm. This code smells ... It looks too complicated. > > I was trying to cut down the number of prepared statements in use and prevent > a > combinatorial explosion while keeping the kinds of

Re: [Cuirass] Missing database indexes?

2018-11-14 Thread Ludovic Courtès
Hi, Danny Milosavljevic skribis: > On Sun, 11 Nov 2018 18:06:00 +0100 > l...@gnu.org (Ludovic Courtès) wrote: > >> I don’t really know what additional index to create (and I’d rather let >> SQLite do it for me, if it were possible). > > Not exactly what you mean but there's this: > >

Re: [Cuirass] Missing database indexes?

2018-11-13 Thread Clément Lassieur
Hi Björn, Björn Höfling writes: > We are directly relying on the rowid here, there is no explicit > id-column. > > This could lead to unpredicted results and reorderings (6th Quirk in > document): > > https://www.sqlite.org/rowidtable.html > > We should add a column: > > id INTEGER PRIMARY KEY

Re: [Cuirass] Missing database indexes?

2018-11-12 Thread Danny Milosavljevic
On Tue, 13 Nov 2018 00:31:40 +0100 Danny Milosavljevic wrote: > Hi Ludo, > > On Sun, 11 Nov 2018 18:06:00 +0100 > l...@gnu.org (Ludovic Courtès) wrote: > > > I don’t really know what additional index to create (and I’d rather let > > SQLite do it for me, if it were possible). > > Not

Re: [Cuirass] Missing database indexes?

2018-11-12 Thread Danny Milosavljevic
Hi Ludo, On Sun, 11 Nov 2018 18:06:00 +0100 l...@gnu.org (Ludovic Courtès) wrote: > I don’t really know what additional index to create (and I’d rather let > SQLite do it for me, if it were possible). Not exactly what you mean but there's this: https://www.sqlite.org/lang_analyze.html It does

Re: [Cuirass] Missing database indexes?

2018-11-12 Thread Danny Milosavljevic
Hi Björn, On Mon, 12 Nov 2018 19:50:44 +0100 Björn Höfling wrote: > Hm. This code smells ... It looks too complicated. I was trying to cut down the number of prepared statements in use and prevent a combinatorial explosion while keeping the kinds of queries we can do open. Either the value of

Re: [Cuirass] Missing database indexes?

2018-11-12 Thread Amirouche Boubekki
Hello all, Le lun. 12 nov. 2018 à 19:51, Björn Höfling < bjoern.hoefl...@bjoernhoefling.de> a écrit : > Hi Ludo, > > On Sun, 11 Nov 2018 18:06:00 +0100 > l...@gnu.org (Ludovic Courtès) wrote: > > > Indeed, that solves the problem for this simple example, thanks! > > > > Now, if I go back to the

Re: [Cuirass] Missing database indexes?

2018-11-12 Thread Björn Höfling
Hi Ludo, On Sun, 11 Nov 2018 18:06:00 +0100 l...@gnu.org (Ludovic Courtès) wrote: > Indeed, that solves the problem for this simple example, thanks! > > Now, if I go back to the big query that /api/latestbuilds makes¹, the > result is still pretty bad: > > --8<---cut

Re: [Cuirass] Missing database indexes?

2018-11-11 Thread Ludovic Courtès
Hi Björn, Björn Höfling skribis: > The link you provided explains it: The column over which you are sorting > (stoptime) is not indexed. Add it to the (same) index: > > sqlite> DROP INDEX Builds_index_evaluation; > sqlite> CREATE INDEX Builds_index_evaluation ON Builds(evaluation, stoptime); >

Re: [Cuirass] Missing database indexes?

2018-11-10 Thread Björn Höfling
On Sat, 10 Nov 2018 18:33:23 +0100 l...@gnu.org (Ludovic Courtès) wrote: > Now, ‘db-get-builds’ in Cuirass uses a more complex query. In > particular, it orders things, very roughly along these lines: > > --8<---cut here---start->8--- > sqlite> EXPLAIN QUERY

[Cuirass] Missing database indexes?

2018-11-10 Thread Ludovic Courtès
Hello! I was investigating the slowness of our /api/latestbuilds requests on berlin. I found that if we have just the two indexes currently defined in ‘schema.sql’, basically everything involves a table scan: --8<---cut here---start->8--- sqlite> EXPLAIN