Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-13 Thread Dominique Devienne
On Wed, Feb 12, 2020 at 9:02 PM Eric Grange wrote: > [...] This is completely safe vs SQL injection, and IME quite efficient. [...] I disagree that this is efficient enough. I'd much rather have native support in SQLite for array binding, in the public API, than this. That public API could wrap

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-12 Thread Eric Grange
If the trouble comes from a big "IN()", an approach can be to pass all the values in a JSON array (one parameter) and use json_each in the query. This is completely safe vs SQL injection, and IME quite efficient. IME using JSON + json_each is also very efficient to fill temporary tables (indexed

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-11 Thread J. King
On February 11, 2020 1:43:30 p.m. EST, Jens Alfke wrote: >I ran into this a few months ago. I ended up just biting the bullet and >constructing a SQL statement by hand, concatenating comma-separated >values inside an "IN (…)" expression. > >Yes, SQL injection is a danger. But if you're being bad

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-11 Thread Richard Hipp
On 2/11/20, J. King wrote: > SQLite also has a 1M byte statement > length limit ... The statement length limit is yet another defense against mischief caused by SQL injections. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-11 Thread Jens Alfke
> On Feb 11, 2020, at 2:10 AM, Digital Dog wrote: > > Thanks for enlighening again. It was just a thought. It seems it would be a > lot of design and code to maintain the performance while preventing the > original problem from happening. Not worth the trouble. But maybe > increasing the

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-11 Thread Digital Dog
On Mon, Feb 10, 2020 at 9:03 PM Richard Hipp wrote: > On 2/10/20, Digital Dog wrote: > > Maybe they should be treated as a > > dictionary/hashtable/linked list or similar? > > > > > Parameter look-ups are on the critical path. How much performance are > you willing to give up in order to have

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Keith Medcalf
On Monday, 10 February, 2020 14:36, Simon Slavin wrote: >Does this problem affect unnumbered indexes too ? In other words if I >have >(?,?,?,?,?) >and bind to the fifth one using the index do I have the same problems as >having >(?1,?2,?3,?4,?5) >and bind to the fifth one using its number

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Simon Slavin
Does this problem affect unnumbered indexes too ? In other words if I have (?,?,?,?,?) and bind to the fifth one using the index do I have the same problems as having (?1,?2,?3,?4,?5) and bind to the fifth one using its number ? ___ sqlite-users

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Richard Hipp
On 2/10/20, Digital Dog wrote: > Maybe they should be treated as a > dictionary/hashtable/linked list or similar? > Parameter look-ups are on the critical path. How much performance are you willing to give up in order to have parameters with larger numbers? -- D. Richard Hipp d...@sqlite.org

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Digital Dog
On Mon, Feb 10, 2020 at 8:27 PM Richard Hipp wrote: > On 2/10/20, Digital Dog wrote: > > > > Nobody bothered to actually show the downside of increasing this value to > > e.g. 10 thousands but everybody immediately proceeded to grumble. > > What is the justifiable rationale to not change the

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Richard Hipp
On 2/10/20, Digital Dog wrote: > > Nobody bothered to actually show the downside of increasing this value to > e.g. 10 thousands but everybody immediately proceeded to grumble. > What is the justifiable rationale to not change the default? The maximum number of variables used to be unlimited

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-10 Thread Digital Dog
Huh so the typical ranting for a valid use case has happened. > Another alternative is to construct the command as a string. begging for SQL injection. Thanks, no. > all the suggestions and examples with temporary tables and that's what you call easy for the programmer? Nobody bothered to

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-07 Thread Alex Bronstein
Thanks, everyone, for your responses on Tuesday. I wasn't on the mailing list, so didn't receive them in my email, but I am now, and I can see the responses on https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg118979.html. Much appreciated! > Can you show us some place where

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Keith Medcalf
On Tuesday, 4 February, 2020 17:23, J. King wrote: >Not everyone has access to carrays and intarrays, either, such as PHP >users like myself. Then you should probably be creating a temporary table and using that/ begin immediate; create temporary table inlist(x primary key(x)) without rowid;

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Scott Robison
On Tue, Feb 4, 2020, 5:23 PM J. King wrote > Not everyone has access to carrays and intarrays, either, such as PHP > users like myself. > But everyone has access to temp tables, and I think the idea of creating a temp table, inserting 1000 items in a loop, and using that temp table in the

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread J. King
On February 4, 2020 7:10:52 p.m. EST, Scott Perry wrote: >On Feb 4, 2020, at 12:26 PM, Simon Slavin wrote: >> >> On 4 Feb 2020, at 7:13pm, Deon Brewis wrote: >> >>> WHERE x IN (?1,?2,?3,?4...,?1000 ) >> >> People really do this ? Sheesh. > >It's a pretty common pattern. Sure, queries that

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Scott Perry
On Feb 4, 2020, at 12:26 PM, Simon Slavin wrote: > > On 4 Feb 2020, at 7:13pm, Deon Brewis wrote: > >> WHERE x IN (?1,?2,?3,?4...,?1000 ) > > People really do this ? Sheesh. It's a pretty common pattern. Sure, queries that are likely to use very large IN groups ought to be using carray or

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Keith Medcalf
On Tuesday, 4 February, 2020 12:14, Deon Brewis wrote: >WHERE x IN (?1,?2,?3,?4...,?1000 ) That would be a really silly construct to use. Why are you bothering to name all the parameters? Anonymous parameters are merely an array of pointers to values. When you give the parameters names

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Simon Slavin
On 4 Feb 2020, at 7:13pm, Deon Brewis wrote: > WHERE x IN (?1,?2,?3,?4...,?1000 ) People really do this ? Sheesh. > The alternate to this is to create a virtual table wrapper over the internal > datasets of the app. Which is of course better, but harder. (We need an STL > for SQLite.

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Deon Brewis
AM To: SQLite mailing list Subject: Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER On 4 Feb 2020, at 6:27pm, Alex Bronstein wrote: > In such cases, you can easily end > up with more than 999 parameters. I'm curious. Can you show us some place where using 999 para

Re: [sqlite] Please increase the default for SQLITE_MAX_VARIABLE_NUMBER

2020-02-04 Thread Simon Slavin
On 4 Feb 2020, at 6:27pm, Alex Bronstein wrote: > In such cases, you can easily end > up with more than 999 parameters. I'm curious. Can you show us some place where using 999 parameters is a reasonable way to use SQLite ? > PostgreSQL and some other databases support a 16 bit parameter