Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread Keith Medcalf

On Thursday, 7 February, 2019 22:47, li...@herger.net wrote:

>> COLLATE affects SORTING, it does not transmorgify the "value" of
>the thing to which it is applied.  That is, name COLLATE  means
>that the item name is SORTED using the collating sequence , not
>that the result of "name COLLATE " is transmorgified into tha
>value that is used for sorting.

>Understood. But wouldn't a GROUP BY sort the data internally in order
>to be able to group records? Or would you not at least expect it to
>follow the same rules grouping as when sorting? In sorting it seems to
>consider Š "the same" as S, but it doesn't in grouping. I'm not too concerned
>about the representation.

But you are not using the same "expression" for selecting, sorting, and 
grouping.  That is, you need to specify:

  SELECT expression, count(distinct id)
FROM artists
GROUP BY expression
ORDER BY expression;

where expression is the expression that you want to use

  SELECT substr(name collate de_DE, 1, 1), count(distinct id)
FROM artists
GROUP BY substr(name collate de_DE, 1, 1)
ORDER BY substr(name collate de_DE, 1, 1);

If you do not do so then you cannot expect the "GROUP BY" to be using the same 
ordering as the "ORDER BY" nor the select to be returning the same value that 
was used to do the grouping and sorting.

(This substitutes the collation NOCASE for the de_DE since I no have a de_DE 
collation:

sqlite> select substr(name collate nocase, 1, 1), count(distinct id)
   ...>   FROM artists
   ...> group by substr(name collate nocase, 1, 1)
   ...> order by substr(name collate nocase, 1, 1);
QUERY PLAN
|--SCAN TABLE artists (~1048576 rows)
`--USE TEMP B-TREE FOR GROUP BY
addr  opcode p1p2p3p4 p5  comment
  -        -  --  -
0 Init   0 57000  Start at 57
1 Noop   1 4 000
2 SorterOpen 3 3 0 k(1,NOCASE)00
3 Integer0 5 000  r[5]=0; clear abort 
flag
4 Null   0 8 800  r[8..8]=NULL
5 Gosub  7 52000
6 OpenRead   0 3 0 2  00  root=3 iDb=0; artists
7 ColumnsUsed0 0 0 3  00
8 Explain8 0 0 SCAN TABLE artists (~1048576 rows)  00
9 Noop   0 0 000  Begin WHERE-loop0: 
artists
10Rewind 0 20000
11  Noop   0 0 000  Begin WHERE-core
12  Column 0 1 13   00  r[13]=artists.name
13  Function0  6 1310substr(3)  03  
r[10]=func(r[13..15])
14  Column 0 1 11   00  r[11]=artists.name
15  Column 0 0 12   00  r[12]=artists.id
16  MakeRecord 103 16   00  
r[16]=mkrec(r[10..12])
17  SorterInsert   3 16000  key=r[16]
18  Noop   0 0 000  End WHERE-core
19Next   0 11001
20Noop   0 0 000  End WHERE-loop0: 
artists
21OpenPseudo 4 16300  3 columns in r[16]
22SorterSort 3 56000  GROUP BY sort
23  SorterData 3 16400  r[16]=data
24  Column 4 0 900  r[9]=
25  Compare8 9 1 k(1,NOCASE)00  r[8] <-> r[9]
26  Jump   273127   00
27  Move   9 8 100  r[8]=r[9]
28  Gosub  6 44000  output one row
29  IfPos  5 56000  if r[5]>0 then 
r[5]-=0, goto 56; check abort flag
30  Gosub  7 52000  reset accumulator
31  Column 4 2 17   00  r[17]=
32  Found  2 36171  00  key=r[17]
33  MakeRecord 171 18   00  r[18]=mkrec(r[17])
34  IdxInsert  2 18171  10  key=r[18]
35  AggStep0 172 count(1)   01  accum=r[2] 
step(r[17])
36  If 4 38000
37  Column 4 1 100  r[1]=
38  Integer1 4 000  r[4]=1; indicate 
data in accumulator
39SorterNext 3 23000
40Gosub  6 44000  output final row
41Goto   0 56000
42Integer1 5 000  r[5]=1; set abort flag
43Return 6 0 

Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread Simon Slavin
On 8 Feb 2019, at 5:47am, li...@herger.net wrote:

> In sorting it seems to consider Š "the same" as S, but it doesn't in 
> grouping. 

That may be a bug.  Can you construct a small test database, where your data 
features, say, R, S, T, and Š, and show how ORDER BY behaves differently to 
GROUP BY ?

Please be clear which version of SQLite you're using for your demo.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread lists

COLLATE affects SORTING, it does not transmorgify the "value" of the thing to which it is 
applied.  That is, name COLLATE  means that the item name is SORTED using the collating 
sequence , not that the result of "name COLLATE " is transmorgified into tha 
value that is used for sorting.


Understood. But wouldn't a GROUP BY sort the data internally in order to 
be able to group records? Or would you not at least expect it to follow 
the same rules grouping as when sorting? In sorting it seems to consider 
Š "the same" as S, but it doesn't in grouping. I'm not too concerned 
about the representation.


Michael




That is

select name collate nocase, count(distinct id) from x group by name collate 
nocase order by name collate nocase

whill produce cased output not the value that was used for the sorting.


select lower(name collate nocase), count(distinct id) from x group by name 
collate nocase order by name collate nocase;

to transmorgificate name into a "caseless" representation.  So you would need 
to do something like this:

select de_DE(substr(name collate de_DE,1,1)), count(distinct id) from artists
group by substr(name collate de_DE,1,1)
order by by substr(name collate de_DE,1,1)

and the function de_DE would have to transmorgificate its value to the result 
you want to see.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.


-Original Message-
From: sqlite-users [mailto:sqlite-users-
boun...@mailinglists.sqlite.org] On Behalf Of li...@herger.net
Sent: Thursday, 7 February, 2019 05:12
To: sqlite-users@mailinglists.sqlite.org
Subject: [sqlite] GROUP BY and ICU collation


Hi there,

I'm trying to create a list with an index list. Eg. I have

artists:


Sting
Šuma Čovjek
Suzanne Vega

That's the sort order I'd get using an ICU collation. "Šuma

Čovjek"

would be sorted as "Suma..." as expected.

Now I'd like to create an index bar by providing groups of the

first

character:

SELECT SUBSTR(name,1,1), count(distinct id) FROM artists GROUP BY
SUBSTR(name,1,1) ORDER BY name COLLATE de_DE>>

Aren't you missing a COLLATE clause after the GROUP BY term?

  ... GROUP BY SUBSTR(name,1,1) COLLATE de_DE ...


TBH: I didn't even know about this. I thought the COLLATE at the end
of
the statement would do it for all.

Alas, tried again to no avail. No matter whether I add it after the
GROUP BY or not, the result is the same.

I should probably have added some version information: I'm using the
Perl DBD::SQLite 1.58 (sqlite 3.22.0). I didn't see any mention of
related changes in the changelog for SQLite. What would be the
easiest
(and most reliable) way to try to reproduce this without Perl? Is
there
a HowTo use collations with the CLI sqlite?

--

Michael
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users




___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Multiple SELECTs in one call

2019-02-07 Thread Jose Isaias Cabrera

They are related by a, but I see that Keith prepared a nice answer.  Thanks.  
And yes, UNION is the key here.  I thought that it was to UNITE other tables, 
but I can see that I can use it with the same table.  Thanks.



From: sqlite-users  on behalf of 
Simon Slavin 
Sent: Wednesday, February 6, 2019 11:30 PM
To: SQLite mailing list
Subject: Re: [sqlite] Multiple SELECTs in one call

On 7 Feb 2019, at 4:21am, Jose Isaias Cabrera  wrote:

>  want to use the result of (SELECT a from t where e != 1); to run another 
> select (SELECT a from t where d > 3); and then, one more select (SELECT a 
> from t where c != 1 AND b != 1);

How are these related to each other ?

Do you want one big result which has the rows returned by the first SELECT, and 
also the rows returned by the second SELECT, and also the rows returned by the 
third SELECT ?

If so, you use UNION:



Or did I get it wrong ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Multiple SELECTs in one call

2019-02-07 Thread Jose Isaias Cabrera

Thanks, Keith.  Yep, exactly what I was looking for.  Gracias.


From: sqlite-users  on behalf of 
Keith Medcalf 
Sent: Wednesday, February 6, 2019 11:37 PM
To: SQLite mailing list
Subject: Re: [sqlite] Multiple SELECTs in one call


You mean something like

select *
  from t
 where a in (select a from t where e != 1
 union
 select a from t where d > 3
 union
 SELECT a from t where c != 1 AND b != 1);

or more succinctly:

select *
  from t
 where a in (select a
  from t
 where e != 1
or d > 3
or (c != 1
and b != 1));

or do you mean:

select *
  from t
 where a in (select a from t where e != 1
 intersect
 select a from t where d > 3
 intersect
 SELECT a from t where c != 1 AND b != 1);

or more succinctly:

select *
  from t
 where a in (select a
  from t
 where e != 1
   and d > 3
   and c != 1
   and b != 1);

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.


>-Original Message-
>From: sqlite-users [mailto:sqlite-users-
>boun...@mailinglists.sqlite.org] On Behalf Of Jose Isaias Cabrera
>Sent: Wednesday, 6 February, 2019 21:22
>To: sqlite-users@mailinglists.sqlite.org
>Subject: [sqlite] Multiple SELECTs in one call
>
>
>Greetings.
>
>I need some help from you gurus to have multiple selects, but the
>sequence is important.  For example,
>
>create table t (a, b, c, d, e);
>insert into t values (1,2,3,4,5);
>insert into t values (2,2,3,4,5);
>insert into t values (3,3,3,3,3);
>insert into t values (4,1,1,1,1);
>insert into t values (5,1,1,2,2);
>insert into t values (6,2,3,2,2);
>
>what I want to do is to do a select that first chooses all items
>where e != 1, and then the result of that select be used to create
>more selects based on other criteria. For example,
>
>SELECT * from t WHERE a IN (SELECT a from t where e != 1);
>
>But I want to use the result of (SELECT a from t where e != 1); to
>run another select (SELECT a from t where d > 3); and then, one more
>select (SELECT a from t where c != 1 AND b != 1); and the final
>select would be "SELECT * from t WHERE a IN" which would have the
>result.  I hope someone understands. :-)  Thanks for the help.
>
>josé
>
>___
>sqlite-users mailing list
>sqlite-users@mailinglists.sqlite.org
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread Keith Medcalf


COLLATE affects SORTING, it does not transmorgify the "value" of the thing to 
which it is applied.  That is, name COLLATE  means that the item name is 
SORTED using the collating sequence , not that the result of "name COLLATE 
" is transmorgified into tha value that is used for sorting.  

That is

select name collate nocase, count(distinct id) from x group by name collate 
nocase order by name collate nocase

whill produce cased output not the value that was used for the sorting.


select lower(name collate nocase), count(distinct id) from x group by name 
collate nocase order by name collate nocase;

to transmorgificate name into a "caseless" representation.  So you would need 
to do something like this:

select de_DE(substr(name collate de_DE,1,1)), count(distinct id) from artists 
group by substr(name collate de_DE,1,1)
order by by substr(name collate de_DE,1,1)

and the function de_DE would have to transmorgificate its value to the result 
you want to see.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

>-Original Message-
>From: sqlite-users [mailto:sqlite-users-
>boun...@mailinglists.sqlite.org] On Behalf Of li...@herger.net
>Sent: Thursday, 7 February, 2019 05:12
>To: sqlite-users@mailinglists.sqlite.org
>Subject: [sqlite] GROUP BY and ICU collation
>
>>> Hi there,
>>>
>>> I'm trying to create a list with an index list. Eg. I have
>artists:
>>>
>>> Sting
>>> Šuma Čovjek
>>> Suzanne Vega
>>>
>>> That's the sort order I'd get using an ICU collation. "Šuma
>Čovjek"
>>> would be sorted as "Suma..." as expected.
>>>
>>> Now I'd like to create an index bar by providing groups of the
>first
>>> character:
>>>
>>> SELECT SUBSTR(name,1,1), count(distinct id) FROM artists GROUP BY
>>> SUBSTR(name,1,1) ORDER BY name COLLATE de_DE>>
>> Aren't you missing a COLLATE clause after the GROUP BY term?
>>
>>  ... GROUP BY SUBSTR(name,1,1) COLLATE de_DE ...
>
>TBH: I didn't even know about this. I thought the COLLATE at the end
>of
>the statement would do it for all.
>
>Alas, tried again to no avail. No matter whether I add it after the
>GROUP BY or not, the result is the same.
>
>I should probably have added some version information: I'm using the
>Perl DBD::SQLite 1.58 (sqlite 3.22.0). I didn't see any mention of
>related changes in the changelog for SQLite. What would be the
>easiest
>(and most reliable) way to try to reproduce this without Perl? Is
>there
>a HowTo use collations with the CLI sqlite?
>
>--
>
>Michael
>___
>sqlite-users mailing list
>sqlite-users@mailinglists.sqlite.org
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread Simon Slavin
On 7 Feb 2019, at 5:02pm, li...@herger.net wrote:

>> If you don't want to have to keep specifying the COLLATE, put it in the 
>> table definition:
>> CREATE TABLE ... ( ... artist TEXT COLLATE de_DE ...)
> 
> Thanks for the hint. But the application is localized to the user's language. 
> Therefore the collation can have different values.

This is currently a problem with SQLite.  You can't specify a collation as a 
string or a variable.  In other words, neither of the following work:

  *** CREATE TABLE ... ( ... artist TEXT COLLATE 'de_DE' ...) ***

  *** SELECT a FROM t ORDER BY a COLLATE 'de_DE' ***

Also, if you use ALTER TABLE RENAME COLUMN other references to that column in 
the schema get changed accordingly.  This makes it impossible to change a 
column's COLLATE easily.

So you end up making your CREATE TABLE command in software once you know the 
preferred collation for that particular installation.  The advantage of doing 
this is that it only needs to be done once, at installation or first run, and 
nothing else in your code needs to worry about which collation to use.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread lists

TBH: I didn't even know about this. I thought the COLLATE at the end of the 
statement would do it for all.


If you don't want to have to keep specifying the COLLATE, put it in the table 
definition:

CREATE TABLE ... ( ... artist TEXT COLLATE de_DE ...)


Thanks for the hint. But the application is localized to the user's 
language. Therefore the collation can have different values.


--

Michael
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Inconsistent behavior in sqlite3_set_authorizer() and error messages

2019-02-07 Thread Joshua Thomas Wise
Is the SQLite team aware of these issues?


> On Feb 4, 2019, at 7:00 AM, sqlite-users-requ...@mailinglists.sqlite.org 
> wrote:
> 
> Hello,
> 
> I found some behaviors that should probably be considered bugs (and should be 
> fixed and/or documented).
> 
> Let’s start the explanation by observing some behavior that actually is 
> correct and consistent. Below, we observe which type of action is reported by 
> sqlite3_set_authorizer(), given some SQL input:
> 
> "CREATE TEMP TABLE foo(x)” -> SQLITE_CREATE_TEMP_TABLE
> "CREATE TEMP VIEW foo AS SELECT 1” -> SQLITE_CREATE_TEMP_VIEW
> "CREATE TEMP TRIGGER foo BEFORE INSERT ON t BEGIN SELECT 1; END" -> 
> SQLITE_CREATE_TEMP_TRIGGER
> 
> So far so good. But what happens when we use the “temp.foo” syntax instead of 
> the TEMP keyword?
> 
> "CREATE TABLE temp.foo(x)" -> SQLITE_CREATE_TABLE
> "CREATE INDEX temp.foo ON t(x)" -> SQLITE_CREATE_TEMP_INDEX
> "CREATE VIEW temp.foo AS SELECT 1" -> SQLITE_CREATE_VIEW
> "CREATE TRIGGER temp.foo BEFORE INSERT ON t BEGIN SELECT 1; END" -> 
> SQLITE_CREATE_TEMP_TRIGGER
> 
> Uh-oh. We have some inconsistencies there. CREATE TABLE and CREATE VIEW 
> report their regular CREATE_* variants, while CREATE INDEX and CREATE TRIGGER 
> do report their TEMP_* variants.
> 
> I recommend that either all or none of those statements should report their 
> TEMP_* variants to the sqlite3_set_authorizer() callback. If the decision is 
> made to not use TEMP_* variants in these cases, the schema name (“temp”, in 
> this case) should ideally be included as an argument to the 
> sqlite3_set_authorizer() callback.
> 
> I also found strange inconsistencies regarding error messages. If we execute 
> the 4 statements above, but this time with the SQLITE_OMIT_TEMPDB 
> compile-time option present, we get the following error messages 
> (respectively):
> 
> 'SQL logic error'
> 'cannot create a TEMP index on non-TEMP table “t”'
> 'SQL logic error’
> 'SQL logic error’
> 
> However, if we replace “temp.foo” with “miss.foo” in each of those 
> statements, we get the much better error message:
> 
> 'unknown database miss’
> 
> All of the observations described in this email were very surprising to me. 
> Hopefully they can be fixed and/or documented.
> 
> Best regards,
> 
> Josh

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Problem to delete all data with a LIKE

2019-02-07 Thread Simon Slavin
On 7 Feb 2019, at 3:51pm, Ludovic Gasc Lemaire  wrote:

> When I launch this SQL query:
> DELETE FROM astdb WHERE key LIKE '%SIP%';
> 
> It deletes between 3 and 40 lines, but not all lines.
> And each time I launch this SQL query, it removes between 3 and 40 other
> lines, but not all lines.

Please run

PRAGMA integrity_check

on the database.  If that doesn't spot anythnig wrong, can you prepare a small 
sample database so we can reproduce the problem ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] GROUP BY and ICU collation

2019-02-07 Thread Simon Slavin
On 7 Feb 2019, at 12:12pm, li...@herger.net wrote:

> TBH: I didn't even know about this. I thought the COLLATE at the end of the 
> statement would do it for all.

If you don't want to have to keep specifying the COLLATE, put it in the table 
definition:

CREATE TABLE ... ( ... artist TEXT COLLATE de_DE ...)

Then it will be used for every operation involving that column, unless it 
collides with another specified COLLATE.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Problem to delete all data with a LIKE

2019-02-07 Thread Ludovic Gasc Lemaire
Hi,

With this package: https://packages.ubuntu.com/cosmic/sqlite3

In the sqlite file of Asterisk (
https://wiki.asterisk.org/wiki/display/AST/Asterisk+Internal+Database ) you
have this schema:
CREATE TABLE IF NOT EXISTS `astdb` (
`key` VARCHAR ( 256 ),
`value` VARCHAR ( 256 ),
PRIMARY KEY(`key`)
);

You can have hundred lines like that:

INSERT INTO `astdb` (key,value) VALUES
('/SIP/Registry/','XXX.XXX.XX.XXX:62467:360:XXX@192.168.0.85:2051
');

When I launch this SQL query:
DELETE FROM astdb WHERE key LIKE '%SIP%';

It deletes between 3 and 40 lines, but not all lines.
And each time I launch this SQL query, it removes between 3 and 40 other
lines, but not all lines.

It's a very strange behavior, I have never see that.

It looks like a bug.

Thanks for your help.

Regards.
--
Ludovic Gasc Lemaire (GMLudo)
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] GROUP BY and ICU collation

2019-02-07 Thread lists

Hi there,

I'm trying to create a list with an index list. Eg. I have artists:

Sting
Šuma Čovjek
Suzanne Vega

That's the sort order I'd get using an ICU collation. "Šuma Čovjek"
would be sorted as "Suma..." as expected.

Now I'd like to create an index bar by providing groups of the first
character:

SELECT SUBSTR(name,1,1), count(distinct id) FROM artists GROUP BY
SUBSTR(name,1,1) ORDER BY name COLLATE de_DE


Aren't you missing a COLLATE clause after the GROUP BY term?

 ... GROUP BY SUBSTR(name,1,1) COLLATE de_DE ...


TBH: I didn't even know about this. I thought the COLLATE at the end of 
the statement would do it for all.


Alas, tried again to no avail. No matter whether I add it after the 
GROUP BY or not, the result is the same.


I should probably have added some version information: I'm using the 
Perl DBD::SQLite 1.58 (sqlite 3.22.0). I didn't see any mention of 
related changes in the changelog for SQLite. What would be the easiest 
(and most reliable) way to try to reproduce this without Perl? Is there 
a HowTo use collations with the CLI sqlite?


--

Michael
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users