[sqlite] Version 3.20.0 release candidate 2

[Richard Hipp]

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[David Raymond]

Union extension page has weird numbering on it for each sentence: 
https://sqlite.org/draft/unionvtab.html

For the query in making a union vtab, is there a shorthand for 
"smallest/largest possible rowid" for columns 3 and 4 to say "just look at 
everything"?

Definitely looks like an extension I could use, thanks.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: Tuesday, July 25, 2017 11:23 AM
To: General Discussion of SQLite Database; sqlite-dev
Subject: [sqlite] Version 3.20.0 release candidate 2

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Stephen Chrzanowski]

On Tue, Jul 25, 2017 at 12:25 PM, petern 
wrote:

> You're trying to change the topic to the security model. This thread is
> supposed to be about a lengthy beyond the pale proposal that named all
> manner of hypothetical boogie men before concluding the only way is a
> "nuclear solution" as in: "Let's just nuke it, that's only way to be
> safe".   I'll ask again. Did you read the proposal itself? If not, why are
> you responding on this thread?  What is your expertise exactly?
>
>
My expertise extends to the knowledge that what Dr Hipp has written is of
the best interest of all the BILLIONS of devices that are out in the field,
actively used knowingly, or unknowingly by the users who use the library.

Your attitude towards a public forum and bully attempts isn't required
here.  I'd ask YOU to leave based on the fact that your behavior is
anything but professional, as I'm not interested in your self proclaimed
expertise.  Perhaps in the software development area, you are an expert in
your own products you deploy, but absolutely NOT in the area of what this
public domain library is being used for.

If there is an aspect of this software you do not like, fork it, and write
your own code to remove what Dr Hipp and his team has put into the code.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Jens Alfke]

> On Jul 25, 2017, at 9:39 AM, Stephen Chrzanowski  wrote:
> 
> Your attitude towards a public forum and bully attempts isn't required
> here.  I'd ask YOU to leave based on the fact that your behavior is
> anything but professional, as I'm not interested in your self proclaimed
> expertise.

Stephen, I don’t think this is called for. Peter is being somewhat vehement (as 
are you) but I don’t think anything he’s said counts as bullying or is 
otherwise out of bounds in a technical discussion.

> My expertise extends to the knowledge that what Dr Hipp has written is of
> the best interest of all the BILLIONS of devices that are out in the field

Argument by appeal to authority isn’t a good idea here. I’m sure Dr Hipp 
doesn’t consider himself infallible (even if he is a doctor), and part of the 
reason for open development is to catch the mistakes and incorrect assumptions 
that all of us are prone to.

I have some modest knowledge of computer security, and I know it’s vulnerable 
to very, very subtle design errors that can leave openings for attack. 
(Security bugs are unique in that they are not triggered randomly, but are 
actively hunted down and exploited by human-level intelligences.) It’s in 
everyone’s best interest that any security related issues in the design be 
thoroughly debated before this is released.

> If there is an aspect of this software you do not like, fork it, and write
> your own code to remove what Dr Hipp and his team has put into the code.

Again, this is unhelpful. We all understand the basics of open source 
development. Forking isn’t a solution, it’s an admission of failure.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] High memory usage for in-memory db with nested transactions

[Brendan E. Coughlan]

Hello everybody,

The included C-Program takes a database file name as a command line 
parameter.


I compiled it with SQLite 3.19.3 under Visual Studio 2017.

If I supply test.db as a command line parameter / file name (and test.db 
doesn't exist yet) the size of test.db, the size of test.db.journal and 
the program's memory usage reach a combined maximum a little short of 2MB.


If I replace the supplied file name by :memory:, the program uses all 
available memory and eventually runs into a memory allocation error.


Am I doing something wrong and if yes what?

Program:

#include 
#include 
#include "sqlite3.h"

void runOrCrash_str(sqlite3 *db, const char *sql)
{
int err = sqlite3_exec(db, sql, NULL, NULL, NULL);
if (SQLITE_OK != err)
{
printf("Error running %s: %s.", sql, sqlite3_errmsg(db));
exit(1);
}
}

void runOrCrash_stmt(sqlite3_stmt * const stmt)
{
int err = sqlite3_reset(stmt);
if (SQLITE_OK != err)
{
printf(
"Error resetting %s: %s.",
sqlite3_expanded_sql(stmt),
sqlite3_errmsg(sqlite3_db_handle(stmt))
);
exit(1);
}
err = sqlite3_step(stmt);
if (SQLITE_DONE != err)
{
printf(
"Error running %s: %s.",
sqlite3_expanded_sql(stmt),
sqlite3_errmsg(sqlite3_db_handle(stmt))
);
exit(1);
}
}

void finalizeOrCrash(sqlite3_stmt *stmt)
{
int err = sqlite3_finalize(stmt);
if (SQLITE_OK != err)
{
printf(
"Error resetting %s: %s.",
sqlite3_expanded_sql(stmt),
sqlite3_errmsg(sqlite3_db_handle(stmt))
);
exit(1);
}
}

sqlite3_stmt *getStmtOrCrash(sqlite3 *db, const char *sql)
{
sqlite3_stmt *rv = NULL;
int err = sqlite3_prepare_v2(db, sql, -1, , NULL);
if (SQLITE_OK != err)
{
printf("Error preparing statement %s: %s.", sql, 
sqlite3_errmsg(db));

exit(1);
}
return rv;
}

int main(int argc, char* argv[])
{
sqlite3 *db = NULL;
int err = SQLITE_OK;

if (2 == argc)
{
const char *fileName = argv[1];
err = sqlite3_open(fileName, );
}
else
{
printf("The database file name should be the only command line 
parameter.");

exit(1);
}
if (SQLITE_OK != err)
{
printf("Error opening the database: %d", err);
exit(0);
}

runOrCrash_str(db, "CREATE TABLE bla(content INTEGER);");
runOrCrash_str(db, "INSERT INTO Bla(content) VALUES(1)");
sqlite3_stmt * sp2 = getStmtOrCrash(db, "SAVEPOINT SP2;");
sqlite3_stmt * sp2Release = getStmtOrCrash(db, "RELEASE SP2;");
sqlite3_stmt * update = getStmtOrCrash(db, "UPDATE bla SET content 
= 1;");


runOrCrash_str(db, "BEGIN IMMEDIATE;");
runOrCrash_str(db, "SAVEPOINT SP1;");


for (int ii = 0; ii < 50; ++ii)
{
runOrCrash_stmt(sp2);
runOrCrash_stmt(update);
runOrCrash_stmt(sp2Release);
}


runOrCrash_str(db, "RELEASE SP1;");
runOrCrash_str(db, "COMMIT;");

finalizeOrCrash(sp2);
sp2 = NULL;
finalizeOrCrash(sp2Release);
sp2Release = NULL;
finalizeOrCrash(update);
update = NULL;
err = sqlite3_close(db);
if (SQLITE_OK != err)
{
printf("Error closing database: %s", sqlite3_errmsg(db));
exit(1);
}
db = NULL;

return 0;
}

Thanks in advance,
Brendan E. Coughlan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[David Wellman]

Also, I've just spotted the following (** characters surround the changes):

1) 
On the same page under " 2.1. Upping The Threat Level", the fourth paragraph 
starts:
 Because the pointer is passed in the t1.t1 column...

I think this is meant to be:
 Because the pointer is passed in the ** t1.cx ** column...

2)
On the same page under " 2.1. Upping The Threat Level", the first bullet under 
the sixth paragraph starts:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do not harm even if exploited

I thin this is meant to say:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do ** no ** harm even if exploited

3)
On the same page under " 2.3. Pointer Leaks"
The second sentence of the first para says:
   In other words, subtypes on pointer values prevents attacks using SQL 
statements like this:

I think it should say:
   In other words, subtypes on pointer values ** prevent ** attacks using SQL 
statements like this:

Or possibly:
   In other words, ** using ** subtypes on pointer values prevents attacks 
using SQL statements like this:

4)
On the same page, under " 5. Summary"

The second sentence of key take-awy #3 starts:
   Instead, use the interfaces designed to facility secure pointer passing:

I think it should be:
   Instead, use the interfaces designed to facilitate secure pointer passing:


Cheers,
Dave


Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of David Wellman
Sent: 25 July 2017 16:31
To: 'SQLite mailing list'
Subject: Re: [sqlite] Version 3.20.0 release candidate 2

Hi,

Minor doc error at https://sqlite.org/draft/bindptr.html 

The first paragraph under "2. A Brief History Of Pointer Passing In SQLite" 
starts with:
   It is sometimes convenient for SQLite extensions to communicatin non-SQL 
values...

I'm not sure what it is meant to say, but probably not that :-) Possibly 
something like:
It is sometimes convenient for SQLite extensions to communicate non-SQL 
values...

Cheers,
Dave



Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: 25 July 2017 16:23
To: General Discussion of SQLite Database; sqlite-dev
Subject: [sqlite] Version 3.20.0 release candidate 2

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Keith Medcalf]

Richard,

Your description of the rational for the new pointer values is sound and quite 
reasonable.  I believe that it meets all the requirements you have set out to 
prevent SQL from being used to either retrieve or set arbitrary internal use 
(that is internal to both the application and SQLite3) pointers that was not 
directly intended (and pre-coded) by the application developer.  How much of a 
security risk being able to do so would present is not particularly relevant to 
the implementation and is difficult to estimate because it certainly varies 
considerably by host OS.  Simply eliminating the possibility of vulnerability 
via this method is adequate and commendable.

It should be noted however that this does not in any way prevent or mitigate 
that a "malicious application" will in fact be able to access or set pointer 
values, however, this is not within the scope of the change.  The scope of the 
change is to protect existing and future applications from being used to 
perform side channel pointer attacks via crafted SQL injections that were not 
intended by the application author.  At this it will succeed brilliantly.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

>-Original Message-
>From: sqlite-users [mailto:sqlite-users-
>boun...@mailinglists.sqlite.org] On Behalf Of Richard Hipp
>Sent: Monday, 24 July, 2017 05:54
>To: General Discussion of SQLite Database
>Subject: [sqlite] New draft document on the new pointer-passing
>interfaces
>
>https://www.sqlite.org/draft/bindptr.html
>
>--
>D. Richard Hipp
>d...@sqlite.org
>___
>sqlite-users mailing list
>sqlite-users@mailinglists.sqlite.org
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Peter Da Silva]

On 7/25/17, 11:25 AM, "sqlite-users on behalf of petern" 
 wrote:
> You're trying to change the topic to the security model.

All I was doing was pointing out that hiding the type information from 
attackers is not a requirement, so the fact that it’s visible if you examine 
the binary or source is not relevant.

Now you’ve acknowledged that you’re not pointing out any actual security flaw, 
we’re done, right?

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Stephen Chrzanowski]

My tone isn't about the technical development discussion.  Its about my
subscribing to this forum and seeing my 11 year olds mentality shine
through with his "I'm not getting the attention I want, so I'm going to
yell and scream and pout until I get what I want".  Perhaps it is a
language barrier, or a cultural difference I don't know about, but, in the
last SEVERAL notes (Pretty much the post right after Dr Hipps notice of the
draft in this particular thread), this hasn't been levied to the technical
side of things, but an individual who is screaming and making demands.
Making demands isn't a discussion.

No one is perfect, I get that.  And please, do discuss the possible or
impossibility of the security faults of the library.  I'm interested in
that.  I'm not interested in seeing the demands.


On Tue, Jul 25, 2017 at 1:00 PM, Jens Alfke  wrote:

>
> > On Jul 25, 2017, at 9:39 AM, Stephen Chrzanowski 
> wrote:
> >
> > Your attitude towards a public forum and bully attempts isn't required
> > here.  I'd ask YOU to leave based on the fact that your behavior is
> > anything but professional, as I'm not interested in your self proclaimed
> > expertise.
>
> Stephen, I don’t think this is called for. Peter is being somewhat
> vehement (as are you) but I don’t think anything he’s said counts as
> bullying or is otherwise out of bounds in a technical discussion.
>
> > My expertise extends to the knowledge that what Dr Hipp has written is of
> > the best interest of all the BILLIONS of devices that are out in the
> field
>
> Argument by appeal to authority isn’t a good idea here. I’m sure Dr Hipp
> doesn’t consider himself infallible (even if he is a doctor), and part of
> the reason for open development is to catch the mistakes and incorrect
> assumptions that all of us are prone to.
>
> I have some modest knowledge of computer security, and I know it’s
> vulnerable to very, very subtle design errors that can leave openings for
> attack. (Security bugs are unique in that they are not triggered randomly,
> but are actively hunted down and exploited by human-level intelligences.)
> It’s in everyone’s best interest that any security related issues in the
> design be thoroughly debated before this is released.
>
> > If there is an aspect of this software you do not like, fork it, and
> write
> > your own code to remove what Dr Hipp and his team has put into the code.
>
> Again, this is unhelpful. We all understand the basics of open source
> development. Forking isn’t a solution, it’s an admission of failure.
>
> —Jens
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[David Wellman]

Sorry, I should have said that item#4 in my last email should change 'facility' 
to 'facilitate'.

Dave


Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of David Wellman
Sent: 25 July 2017 16:46
To: 'SQLite mailing list'
Subject: Re: [sqlite] Version 3.20.0 release candidate 2

Also, I've just spotted the following (** characters surround the changes):

1) 
On the same page under " 2.1. Upping The Threat Level", the fourth paragraph 
starts:
 Because the pointer is passed in the t1.t1 column...

I think this is meant to be:
 Because the pointer is passed in the ** t1.cx ** column...

2)
On the same page under " 2.1. Upping The Threat Level", the first bullet under 
the sixth paragraph starts:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do not harm even if exploited

I thin this is meant to say:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do ** no ** harm even if exploited

3)
On the same page under " 2.3. Pointer Leaks"
The second sentence of the first para says:
   In other words, subtypes on pointer values prevents attacks using SQL 
statements like this:

I think it should say:
   In other words, subtypes on pointer values ** prevent ** attacks using SQL 
statements like this:

Or possibly:
   In other words, ** using ** subtypes on pointer values prevents attacks 
using SQL statements like this:

4)
On the same page, under " 5. Summary"

The second sentence of key take-awy #3 starts:
   Instead, use the interfaces designed to facility secure pointer passing:

I think it should be:
   Instead, use the interfaces designed to facilitate secure pointer passing:


Cheers,
Dave


Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of David Wellman
Sent: 25 July 2017 16:31
To: 'SQLite mailing list'
Subject: Re: [sqlite] Version 3.20.0 release candidate 2

Hi,

Minor doc error at https://sqlite.org/draft/bindptr.html 

The first paragraph under "2. A Brief History Of Pointer Passing In SQLite" 
starts with:
   It is sometimes convenient for SQLite extensions to communicatin non-SQL 
values...

I'm not sure what it is meant to say, but probably not that :-) Possibly 
something like:
It is sometimes convenient for SQLite extensions to communicate non-SQL 
values...

Cheers,
Dave



Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: 25 July 2017 16:23
To: General Discussion of SQLite Database; sqlite-dev
Subject: [sqlite] Version 3.20.0 release candidate 2

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org

Re: [sqlite] Version 3.20.0 release candidate 2

[Jens Alfke]

In the release notes themselves:

The sentence fragment “tab-completions…interfaces” was accidentally 
copied/pasted from the previous line:
• Added the UNION virtual table extension. tab-completions for 
interactive user interfaces. This is a work in progress. Expect further 
enhancements in future releases.

This sentence is technically correct but reads really awkwardly; I suggest 
deleting the first “interfaces”:
• Add new interfaces pointer passing interfaces.

Typo: “Futher":
• The built-in date and time functions have been enhanced so that they 
can be used within CHECK constraints, indexes on expressions, and in the WHERE 
clause of a partial index, provided that they do not use the 'now', 
'localtime', or 'utc' keywords.  Futher information.

The various list items also disagree on tense, with some in present (“Add …”) 
and some in past (“Added…”), but I understand if you feel this is too nit-picky 
to be worth cleaning up :)

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[Simon Slavin]

On 25 Jul 2017, at 6:06pm, Brendan E. Coughlan  wrote:

> I compiled it with SQLite 3.19.3 under Visual Studio 2017.

We’re going to presume Window 10 with all updates to date installed unless you 
tell us otherwise.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[David Wellman]

Hi,

Minor doc error at https://sqlite.org/draft/bindptr.html 

The first paragraph under "2. A Brief History Of Pointer Passing In SQLite" 
starts with:
   It is sometimes convenient for SQLite extensions to communicatin non-SQL 
values...

I'm not sure what it is meant to say, but probably not that :-) Possibly 
something like:
It is sometimes convenient for SQLite extensions to communicate non-SQL 
values...

Cheers,
Dave



Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: 25 July 2017 16:23
To: General Discussion of SQLite Database; sqlite-dev
Subject: [sqlite] Version 3.20.0 release candidate 2

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[Graham Holden]

Going on the snippet in David's correction only:
 That arbitrary SQL was suppose to be run inside a sandbox where it could do ** 
no ** harm even if exploited
It probably should be "was supposed".
Graham.
 Original message From: David Wellman 
 Date: 25/07/2017  16:46  (GMT+00:00) To: 'SQLite 
mailing list'  Subject: Re: [sqlite] 
Version 3.20.0 release candidate 2 
Also, I've just spotted the following (** characters surround the changes):

1) 
On the same page under " 2.1. Upping The Threat Level", the fourth paragraph 
starts:
 Because the pointer is passed in the t1.t1 column...

I think this is meant to be:
 Because the pointer is passed in the ** t1.cx ** column...

2)
On the same page under " 2.1. Upping The Threat Level", the first bullet under 
the sixth paragraph starts:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do not harm even if exploited

I thin this is meant to say:
 The WebSQL interface to webkit allowed any webpage to to run arbitrary SQL 
in the browser for Chrome and Safari. That arbitrary SQL was suppose to be run 
inside a sandbox where it could do ** no ** harm even if exploited

3)
On the same page under " 2.3. Pointer Leaks"
The second sentence of the first para says:
   In other words, subtypes on pointer values prevents attacks using SQL 
statements like this:

I think it should say:
   In other words, subtypes on pointer values ** prevent ** attacks using SQL 
statements like this:

Or possibly:
   In other words, ** using ** subtypes on pointer values prevents attacks 
using SQL statements like this:

4)
On the same page, under " 5. Summary"

The second sentence of key take-awy #3 starts:
   Instead, use the interfaces designed to facility secure pointer passing:

I think it should be:
   Instead, use the interfaces designed to facilitate secure pointer passing:


Cheers,
Dave


Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of David Wellman
Sent: 25 July 2017 16:31
To: 'SQLite mailing list'
Subject: Re: [sqlite] Version 3.20.0 release candidate 2

Hi,

Minor doc error at https://sqlite.org/draft/bindptr.html 

The first paragraph under "2. A Brief History Of Pointer Passing In SQLite" 
starts with:
   It is sometimes convenient for SQLite extensions to communicatin non-SQL 
values...

I'm not sure what it is meant to say, but probably not that :-) Possibly 
something like:
    It is sometimes convenient for SQLite extensions to communicate non-SQL 
values...

Cheers,
Dave



Ward Analytics Ltd - information in motion
Tel: +44 (0) 118 9740191
Fax: +44 (0) 118 9740192
www: http://www.ward-analytics.com

Registered office address: The Oriel, Sydenham Road, Guildford, Surrey, United 
Kingdom, GU1 3SR
Registered company number: 3917021 Registered in England and Wales.

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: 25 July 2017 16:23
To: General Discussion of SQLite Database; sqlite-dev
Subject: [sqlite] Version 3.20.0 release candidate 2

We are restarting the test process for SQLite 3.20.0.  There is a new
release candidate on the website at https://sqlite.org/download.html
and on the newly reopened branch-3.20 of the source tree at
https://sqlite.org/src/timeline?r=branch-3.20

A new draft change log is at https://sqlite.org/draft/releaselog/3_20_0.html

Changes in the release candidate since the previous release attempt include:

* Improved documentation for the new pointer-passing interface
* Date/time functions can now be used in CHECK constraints and indexes
* Added the UNION virtual table extension

The target release date is one week from today.  If you have concerns
with anything in this release, please raise them now.

The checklist at https://sqlite.org/checklists/320/index has been
reset.  The release will occur when that checklist goes all-green.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org

Re: [sqlite] New draft document on the new pointer-passing interfaces

[petern]

You're trying to change the topic to the security model. This thread is
supposed to be about a lengthy beyond the pale proposal that named all
manner of hypothetical boogie men before concluding the only way is a
"nuclear solution" as in: "Let's just nuke it, that's only way to be
safe".   I'll ask again. Did you read the proposal itself? If not, why are
you responding on this thread?  What is your expertise exactly?

Also, where is your critique of my suggestion to simply and logically
expand the well proven and secure subtype mechanism?  Why do you presume
that Richard's idea to suddenly create a parallel API is automatically the
best plan?  What is your expertise on the main issue of how to pass around
runtime object pointers?  If you reply to this thread again, please
demonstrate some expertise on the issue at hand.

Now I'll address the question of remote attack path in case somebody
mistakes a superficial question for a valid point.  I don't know any exact
attack path, local or remote, and neither does Richard.  That's the point.
However, all unknowns being equal, if there is a local way to inject
pointers then there could be a remote way.   Whatever the method, the
attacker can then use the supposedly secure constant space names everywhere
the application is deployed without further opposition.   If you recall, I
pointed out that the programmer is free to rotate or randomize integer
subtypes of the current API at runtime if it's keeping them from sleep.  So
in this vein of these hypothetical threats, constant space names are no
magic bullet.  They have drawbacks too.

That being said, let me be very clear once again for short attention
spans.  There is no attack path in the current API.  It works fine and that
was proven in the field!  So please, BEFORE RESPONDING TO THIS POST, DO
READ THE PROPOSAL WHERE THIS IS CLEARLY STATED!



On Tue, Jul 25, 2017 at 6:11 AM, Peter Da Silva <
peter.dasi...@flightaware.com> wrote:

> On 7/24/17, 7:20 PM, "sqlite-users on behalf of petern" <
> sqlite-users-boun...@mailinglists.sqlite.org on behalf of
> peter.nichvolo...@gmail.com> wrote:
> > Justifications presented in the proposal claim hardwired constants for
> mandatory lock and key style pointer value receiving are a great idea
> because SQL can't generate constant space strings.
>
> This has nothing to do with the secrecy or otherwise of these strings, but
> to prevent developers from _even in principle_ implementing a mechanism for
> passing these strings in from SQL. If it is not possible to inject the
> string, then it doesn’t matter if they’re secret or not. Think of it as
> preventing the creation of a “cross domain” attack from an insecure module.
>
> You might as well complain that the names of internal functions in SQLite
> are known. There’s no way from SQL to call them, so it’s not an attack
> surface.
>
> > On the local device the hacker attack space would be immediately
> narrowed to constants listed in the executable which, I might add, are
> guaranteed to work on remote copy of the same application!
>
> How do you figure? What is the attack path you’re postulating?
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[Richard Hipp]

On 7/25/17, David Wellman  wrote:
> Hi,
>
> Minor doc error at https://sqlite.org/draft/bindptr.html
>

Typo fixed.  Thanks.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Version 3.20.0 release candidate 2

[Simon Slavin]

For the pragmas which an be used as functions, would it be possible to mark 
these somehow in the "List Of PRAGMAs" ?

Perhaps PRAGMAs whih an be used as functions can have a normal bullet point and 
those which can’t have a "WHITE BULLET" instead.  Or any other method of 
marking which appeals to the author.  I don’t mind as long as they’re marked.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[Richard Hipp]

On 7/25/17, Brendan E. Coughlan  wrote:
>
> I compiled it with SQLite 3.19.3 under Visual Studio 2017.
>
> If I supply test.db as a command line parameter / file name (and test.db
> doesn't exist yet) the size of test.db, the size of test.db.journal and
> the program's memory usage reach a combined maximum a little short of 2MB.

I get 288KB on Linux...

>
> If I replace the supplied file name by :memory:, the program uses all
> available memory and eventually runs into a memory allocation error.
>

Memory usage tops out at 2,066,287,984 for me.  That is 7000 times
more memory, though...

> Am I doing something wrong and if yes what?
>

Because of the way you have structured your SAVEPOINTs, the statement
log (used to ROLLBACK TO a prior savepoint) must add at least one new
page for each of your 500K UPDATEs.  When the database file is on
disk, the statement log is a temporary file on disk which you are not
noticing.  But when the database file is ":memory:" the statement log
is also in memory.  500K transactions at 4KB per page accounts for
most of the 2GB of memory used.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] New draft document on the new pointer-passing interfaces

[Richard Hipp]

On 7/24/17, petern  wrote:
> Great.  But, if this is an ultimate replacement for BLOB'ed pointers, these
> new pseudo-null pointers must support SQLITE_STATIC and destructor function
> pointer lifetime disposition for those migrating their code.

Nobody is forcing you to migrate your legacy code to the new API.
Anything that worked for you in 3.19.3 (or earlier) will continue to
work just as well in 3.20.0.  If what you are doing now works for you,
then you are welcomed to keep doing exactly the same in the future.

>
> Why can't the producer destructor disposition be preserved within a chain
> of application functions by subsequent consumers passing SQLITE_STATIC
> disposition as they do now?

I cannot say with precision because your proposal is vague.  But what
you want to do will very likely use extra memory and extra CPU cycles
for the overwhelmingly common case where pointers are not being
passed.  We do not want to burden the common case for the convenience
of an outlier.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[Brendan E. Coughlan]

> Memory usage tops out at 2,066,287,984 for me.

That's pretty much all there is for a normal 32bit Windows process (2GB 
in user mode, a litte of which may be taken by user mode parts of the 
OS, the other 2GB reserved for Kernel mode). I hear 32bit Linux 
programmers get another GB of user mode memory, but I've never done 
anything memory-intensive on Linux.



> Because of the way you have structured your SAVEPOINTs, the statement
> log (used to ROLLBACK TO a prior savepoint) must add at least one new
> page for each of your 500K UPDATEs.

So a RELEASEd savepoint can still take a page in the log? Is there an 
approved way to work around this?


Since this happens rarely in the original context I'm thinking about 
making it a temporary database as described at 
https://www.sqlite.org/inmemorydb.html. When most of the cache is unused 
(as it usually is) that would be (roughly) as fast as an in-memory db, 
right? If so, is there any way to influence where in the filesystem the 
empty temporary database is created? And, just to make sure, a temporary 
database would also get an on-disk statement log, right?


Greetings,
Brendan E. Coughlan
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[David Raymond]

Would you elaborate a bit more on that? To my untrained-in-C eyes it looks like 
there's the outer transaction, then a savepoint1, then a loop of (savepoint2, 
update the only record, release savepoint2). Is savepoint2 there not actually 
getting released each time? Wouldn't the outer transaction and savepoint1 each 
only need to hold the original 2 pages? Which of those is/are eating the 
memory? Is it an SQL problem or a C problem that's causing it? And what's the 
correct way?


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: Tuesday, July 25, 2017 1:35 PM
To: SQLite mailing list
Subject: Re: [sqlite] High memory usage for in-memory db with nested 
transactions


Because of the way you have structured your SAVEPOINTs, the statement
log (used to ROLLBACK TO a prior savepoint) must add at least one new
page for each of your 500K UPDATEs.  When the database file is on
disk, the statement log is a temporary file on disk which you are not
noticing.  But when the database file is ":memory:" the statement log
is also in memory.  500K transactions at 4KB per page accounts for
most of the 2GB of memory used.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[Dan Kennedy]

On 07/26/2017 12:58 AM, David Raymond wrote:

Would you elaborate a bit more on that? To my untrained-in-C eyes it looks like 
there's the outer transaction, then a savepoint1, then a loop of (savepoint2, 
update the only record, release savepoint2). Is savepoint2 there not actually 
getting released each time? Wouldn't the outer transaction and savepoint1 each 
only need to hold the original 2 pages? Which of those is/are eating the 
memory? Is it an SQL problem or a C problem that's causing it? And what's the 
correct way?


It's an implementation artifact really.

Each time a page is modified within a savepoint, if the page has not 
already been journalled within the current savepoint, a copy of it is 
appended to the statement journal. In case you do "ROLLBACK TO" to 
revert the change. But pages are never removed from the statement 
journal - except that if the number of open savepoint transactions drops 
to zero, the statement journal is truncated to zero bytes in size.


So opening and closing lots of nested savepoints while writing to the 
database without ever closing the outermost savepoint can leave you with 
a statement journal many times the size of the original database. With 
the current implementation, the only way to avoid this is to ensure the 
outermost savepoint is closed from time to time.


Dan.








-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Richard Hipp
Sent: Tuesday, July 25, 2017 1:35 PM
To: SQLite mailing list
Subject: Re: [sqlite] High memory usage for in-memory db with nested 
transactions


Because of the way you have structured your SAVEPOINTs, the statement
log (used to ROLLBACK TO a prior savepoint) must add at least one new
page for each of your 500K UPDATEs.  When the database file is on
disk, the statement log is a temporary file on disk which you are not
noticing.  But when the database file is ":memory:" the statement log
is also in memory.  500K transactions at 4KB per page accounts for
most of the 2GB of memory used.



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[David Raymond]

Thank you for the explanation.


-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Dan Kennedy
Sent: Tuesday, July 25, 2017 2:21 PM
To: sqlite-users@mailinglists.sqlite.org
Subject: Re: [sqlite] High memory usage for in-memory db with nested 
transactions

On 07/26/2017 12:58 AM, David Raymond wrote:
> Would you elaborate a bit more on that? To my untrained-in-C eyes it looks 
> like there's the outer transaction, then a savepoint1, then a loop of 
> (savepoint2, update the only record, release savepoint2). Is savepoint2 there 
> not actually getting released each time? Wouldn't the outer transaction and 
> savepoint1 each only need to hold the original 2 pages? Which of those is/are 
> eating the memory? Is it an SQL problem or a C problem that's causing it? And 
> what's the correct way?

It's an implementation artifact really.

Each time a page is modified within a savepoint, if the page has not 
already been journalled within the current savepoint, a copy of it is 
appended to the statement journal. In case you do "ROLLBACK TO" to 
revert the change. But pages are never removed from the statement 
journal - except that if the number of open savepoint transactions drops 
to zero, the statement journal is truncated to zero bytes in size.

So opening and closing lots of nested savepoints while writing to the 
database without ever closing the outermost savepoint can leave you with 
a statement journal many times the size of the original database. With 
the current implementation, the only way to avoid this is to ensure the 
outermost savepoint is closed from time to time.

Dan.





>
>
> -Original Message-
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
> Behalf Of Richard Hipp
> Sent: Tuesday, July 25, 2017 1:35 PM
> To: SQLite mailing list
> Subject: Re: [sqlite] High memory usage for in-memory db with nested 
> transactions
>
>
> Because of the way you have structured your SAVEPOINTs, the statement
> log (used to ROLLBACK TO a prior savepoint) must add at least one new
> page for each of your 500K UPDATEs.  When the database file is on
> disk, the statement log is a temporary file on disk which you are not
> noticing.  But when the database file is ":memory:" the statement log
> is also in memory.  500K transactions at 4KB per page accounts for
> most of the 2GB of memory used.
>

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] High memory usage for in-memory db with nested transactions

[Brendan E. Coughlan]

We’re going to presume Window 10 with all updates to date installed unless you 
tell us otherwise.


Correct.

I don't think it really matters though, since the situation I simplified 
it from runs on Window 7 and is compiled with VS2008, so it's probably 
not very new.


Greetings,
Brendan
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users