Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-23 Thread Dominique Devienne
On Sat, Sep 21, 2019 at 10:17 PM Fredrik Larsen wrote: > [...] But fixing issues in less than a day of reporting? [...] > That's not unusual at all for SQLite. Either it gets "fixed" quickly, or it doesn't. The hard part is making the case with Richard (and Dan) about the merit of the change,

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
Good to know that I was not to far off target then. But fixing issues in less than a day of reporting? On a Saturday? Who does that? I was planning to feel happy about solving this issue.. :) Fredrik On Sat, Sep 21, 2019 at 9:31 PM Dan Kennedy wrote: > > On 22/9/62 02:25, Fredrik Larsen wrote:

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Dan Kennedy
On 22/9/62 02:25, Fredrik Larsen wrote: Interesting, very similar change but not fully idenctial. In my patch, I created a sqlite3ExprListCompareIgnoreButUpdateSort, and used this function from line 6239. This function ignores the sort part when comparing expressions, but will update the

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
Interesting, very similar change but not fully idenctial. In my patch, I created a sqlite3ExprListCompareIgnoreButUpdateSort, and used this function from line 6239. This function ignores the sort part when comparing expressions, but will update the GroupBy sortOrder field if expressions are found

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Keith Medcalf
See Dan's checkin on trunk for this issue. https://www.sqlite.org/src/info/20f7951bb238ddc0 >-Original Message- >From: sqlite-users On >Behalf Of Fredrik Larsen >Sent: Saturday, 21 September, 2019 08:12 >To: SQLite mailing list >Subject: Re: [sqlite] [EXTERNAL] Group-by and

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
To clarify; GROUP-BY does not really have ordering, but in the SQLite implementation, GROUP-BY and ORDER-BY is very closely related as expected, and it is possible to set a GROUP-BY direction in code (it is default 0 -> ASC). So thats what I did. Also, some other modifications very required to

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-21 Thread Fredrik Larsen
Your last sentence got me thinking. So I downloaded the source, modified the ordering of the GROUP-BY expression to match ORDER-BY and it works! This will offcourse only work if the GROUP-BY and ORDER-BY matches generally expect for the direction. This fix only improves performance for relevant

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Keith Medcalf
>We can observe GROUP BY works ASCending only as of now. Why it can't work >DESCending to avoid ordering, that's a different question. >From https://www.sqlite.org/lang_select.html we can observe that >GROUP BY takes an expr on the RHS, while ORDER BY takes an expr >followed by optional COLLATE

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Fredrik Larsen
I´m not sure why you think group_by + order_by_desc + limit N queries are so obscure? Useful for lots of tail-statistics (number of transactions last N hours if group_key is time-based, etc). In my case I'm implementing a event-store using sqlite, where I need to be able to retrieve entity data

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread R Smith
On 2019/09/20 2:49 PM, Fredrik Larsen wrote: Hi Ryan Nobody is proposing that QP should automagically add an index, I'm only asking why the QP does not use already added index, that is specially added for this specific case. I don't thinks this is a very "obscurest of use-case" or to much to

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Fredrik Larsen
Hi Ryan Nobody is proposing that QP should automagically add an index, I'm only asking why the QP does not use already added index, that is specially added for this specific case. I don't thinks this is a very "obscurest of use-case" or to much to ask for, in fact, this is the expected behavior

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread R Smith
On 2019/09/20 11:12 AM, Dominique Devienne wrote: But who says the GROUP BY must return rows in ASCending order? A lot of us "oldies" of this ML well know the order is arbitrary and subject to change w/o an explicit ORDER BY. So the GROUP BY is allowed, AFAIK, to return rows in DESCending

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Dominique Devienne
On Fri, Sep 20, 2019 at 12:33 PM Hick Gunter wrote: > The dialogue from the stackoverflow discussion shows this quite clearly. > Shows what clearly Gunter? I'm not sure to follow. I've read the SO post, and I don't get your point. We can observe GROUP BY works ASCending only as of now. Why it

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Hick Gunter
The dialogue from the stackoverflow discussion shows this quite clearly. "The code for looping over an index goes backwards only when needed. For implementing GROUP BY itself, going backwards is never needed, so it is never tried. It is possible that a future SQLite version might add code to

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-20 Thread Dominique Devienne
On Thu, Sep 19, 2019 at 6:15 PM Hick Gunter wrote: > -Ursprüngliche Nachricht- > Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] > Im Auftrag von Fredrik Larsen > Gesendet: Donnerstag, 19. September 2019 17:29 > An: SQLite mailing list > Betreff: Re: [sqlite]

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-19 Thread Hick Gunter
-Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Fredrik Larsen Gesendet: Donnerstag, 19. September 2019 17:29 An: SQLite mailing list Betreff: Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-19 Thread Fredrik Larsen
Simen; ANALYZE and PRAGMA reverse_unordered_selects = YES does not affect results. Hick; ORDER BY x DESC >is< covered by index. Btree-indexes allows traversal both ways. You can see this if you remove GROUP_BY. Got an answer on StackOverflow that seems to be from somebody that knows internal

Re: [sqlite] [EXTERNAL] Group-by and order-by-desc does not work as expected

2019-09-19 Thread Hick Gunter
An ORDER BY clause will omit sorting only if the visitation order exactly fulfills the clause. A GROUP BY clause is able to avoid creating a temporary table if the visitation order exactly fulfills the clause. If a SELECT references only fields present in an index, that (covering) index may