[sqlite] hard links and SQLite

2016-01-12 Thread Rowan Worth
On 12 January 2016 at 13:12, Felipe Gasper wrote: > Same code, just different processes. > > We?ve just noted over the years with race conditions that that ?if it can > fail, it will?, so we try to be as bulletproof as we can. > Good policy :) After you unlink the temp file, I presume the

[sqlite] hard links and SQLite

2016-01-12 Thread Rowan Worth
On 12 January 2016 at 13:26, Scott Hess wrote: > On Mon, Jan 11, 2016 at 9:12 PM, Felipe Gasper > wrote: > > > On 11 Jan 2016 9:06 PM, Rowan Worth wrote: > > > >> * if it returns SQLITE_OK and zero rows, the schema hasn't been created > >> yet > >> > > > > Sure; however, by the time you do the

[sqlite] hard links and SQLite

2016-01-12 Thread Rowan Worth
On 12 January 2016 at 03:00, Felipe Gasper wrote: > On 11 Jan 2016 1:45 PM, Scott Hess wrote: > >> >> As far as preventing the other process from using it before the schema >> exists, do "SELECT count(*) FROM sqlite_master", and if the result is 0, >> the schema does not exist. If you create

[sqlite] hard links and SQLite

2016-01-12 Thread Zia Khatri
m which doesn't propagate POSIX file region locks across hard links. -Rowan ___ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] hard links and SQLite

2016-01-12 Thread Felipe Gasper
On 11 Jan 2016 9:06 PM, Rowan Worth wrote: > On 12 January 2016 at 03:00, Felipe Gasper wrote: > >> On 11 Jan 2016 1:45 PM, Scott Hess wrote: >> >>> >>> As far as preventing the other process from using it before the schema >>> exists, do "SELECT count(*) FROM sqlite_master", and if the result is

[sqlite] hard links and SQLite

2016-01-11 Thread Igor Tandetnik
On 1/11/2016 11:52 PM, Scott Doctor wrote: > TOCTTOU? What is that? A friendly local search engine suggests it means "time of check to time of use" ( https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use ) -- Igor Tandetnik

[sqlite] hard links and SQLite

2016-01-11 Thread Keith Medcalf
> On 12 January 2016 at 13:26, Scott Hess wrote: > > > On Mon, Jan 11, 2016 at 9:12 PM, Felipe Gasper > > wrote: > > > > > On 11 Jan 2016 9:06 PM, Rowan Worth wrote: > > > > > >> * if it returns SQLITE_OK and zero rows, the schema hasn't been > created > > >> yet > > >> > > > > > > Sure;

[sqlite] hard links and SQLite

2016-01-11 Thread R Smith
On 2016/01/11 9:37 PM, Warren Young wrote: > The OP was vague about that, but I think the point of his current > gymnastics is to prevent the other process from creating a rogue > schema, or to insert compromising data into a correct schema. To make > it concrete, you could probably write a

[sqlite] hard links and SQLite

2016-01-11 Thread R Smith
So, if I understand correct, you create a file and then start to add a schema etc. Before the schema is committed (or before the exclusive transaction is started) you assume that other threads/processes /might/ be opening the file and trying to write some data to it (because reading would be

[sqlite] hard links and SQLite

2016-01-11 Thread Scott Hess
On Mon, Jan 11, 2016 at 9:12 PM, Felipe Gasper wrote: > On 11 Jan 2016 9:06 PM, Rowan Worth wrote: > >> * if it returns SQLITE_OK and zero rows, the schema hasn't been created >> yet >> > > Sure; however, by the time you do the next action it?s possible that > something else will be creating the

[sqlite] hard links and SQLite

2016-01-11 Thread Clemens Ladisch
Felipe Gasper wrote: > To prevent race conditions where a 2nd process accesses a newly-created > SQLite file before the creator process can set up the schema, ... And what happens when the 2nd process tries to access the database just before the first creates it? > I?m seeing some corruption in

[sqlite] hard links and SQLite

2016-01-11 Thread Scott Doctor
On 01/11/2016 18:06, Rowan Worth wrote: > On 12 January 2016 at 03:00, Felipe Gasper wrote: > >> On 11 Jan 2016 1:45 PM, Scott Hess wrote: >> >>> As far as preventing the other process from using it before the schema >>> exists, do "SELECT count(*) FROM sqlite_master", and if the result is 0,

[sqlite] hard links and SQLite

2016-01-11 Thread Stephan Beal
On Mon, Jan 11, 2016 at 7:57 PM, Stephan Beal wrote: > On Mon, Jan 11, 2016 at 7:55 PM, Warren Young wrote: > >> On POSIX systems, you can securely create a temp file that only your user >> can see via the mkstemp(3) C library call. SQLite will happily open the >> resulting 0-byte file,

[sqlite] hard links and SQLite

2016-01-11 Thread Stephan Beal
On Mon, Jan 11, 2016 at 7:55 PM, Warren Young wrote: > On POSIX systems, you can securely create a temp file that only your user > can see via the mkstemp(3) C library call. SQLite will happily open the > resulting 0-byte file, allowing you to create your schema inside it. Then > when the file

[sqlite] hard links and SQLite

2016-01-11 Thread Simon Slavin
On 11 Jan 2016, at 7:06pm, Warren Young wrote: > On Jan 11, 2016, at 11:57 AM, Simon Slavin wrote: > >> Just do BEGIN IMMEDIATE immediately after you open the database. > > Doesn?t that set up a race condition? What prevents the other process from > opening the DB and running its own SQL

[sqlite] hard links and SQLite

2016-01-11 Thread Simon Slavin
On 11 Jan 2016, at 6:25pm, Felipe Gasper wrote: > - create SQLite file as a .tmp beside permanent location I think you can omit everything to do with the link. Just do BEGIN IMMEDIATE immediately after you open the database. This should lock other processes out until you have finished

[sqlite] hard links and SQLite

2016-01-11 Thread Bernardo Sulzbach
> What have folks here done to avoid the race condition originally described? > I?ve seen some documentation of SQLite and hard links but Your message appears to be truncated. What if you move the hard link creation part to right after the commit?

[sqlite] hard links and SQLite

2016-01-11 Thread Felipe Gasper
On 11 Jan 2016 1:45 PM, Scott Hess wrote: > > As far as preventing the other process from using it before the schema > exists, do "SELECT count(*) FROM sqlite_master", and if the result is 0, > the schema does not exist. If you create the schema as a transaction, that > will be atomic. But in

[sqlite] hard links and SQLite

2016-01-11 Thread Felipe Gasper
On 11 Jan 2016 1:30 PM, Bernardo Sulzbach wrote: >> What have folks here done to avoid the race condition originally described? >> I?ve seen some documentation of SQLite and hard links but > > Your message appears to be truncated. Oops! Nothing earth-shattering ? I meant to put

[sqlite] hard links and SQLite

2016-01-11 Thread Felipe Gasper
seeing some corruption in our SQLite files now but have had a hard time tracking down the cause. Could it be because of the above logic? What have folks here done to avoid the race condition originally described? I?ve seen some documentation of SQLite and hard links but -FG

[sqlite] hard links and SQLite

2016-01-11 Thread Warren Young
On Jan 11, 2016, at 12:27 PM, Simon Slavin wrote: > > On 11 Jan 2016, at 7:06pm, Warren Young wrote: > >> On Jan 11, 2016, at 11:57 AM, Simon Slavin wrote: >> >>> Just do BEGIN IMMEDIATE immediately after you open the database. >> >> Doesn?t that set up a race condition? What prevents the

[sqlite] hard links and SQLite

2016-01-11 Thread Warren Young
On Jan 11, 2016, at 11:57 AM, Simon Slavin wrote: > > Just do BEGIN IMMEDIATE immediately after you open the database. Doesn?t that set up a race condition? What prevents the other process from opening the DB and running its own SQL if the OS?s scheduler happens to take the CPU away from the

[sqlite] hard links and SQLite

2016-01-11 Thread Scott Hess
On Mon, Jan 11, 2016 at 11:00 AM, Felipe Gasper wrote: > On 11 Jan 2016 1:45 PM, Scott Hess wrote: > >> As far as preventing the other process from using it before the schema >> exists, do "SELECT count(*) FROM sqlite_master", and if the result is 0, >> the schema does not exist. If you create

[sqlite] hard links and SQLite

2016-01-11 Thread Warren Young
On Jan 11, 2016, at 11:57 AM, Stephan Beal wrote: > > On Mon, Jan 11, 2016 at 7:55 PM, Warren Young wrote: > >> There must be an equivalent of mkstemp() on Windows, doubtless taking 3 >> times as many parameters and with a function name 4 times as long. :) > > sqlite exposes the functionality

[sqlite] hard links and SQLite

2016-01-11 Thread Warren Young
On Jan 11, 2016, at 11:25 AM, Felipe Gasper wrote: > > To prevent race conditions where a 2nd process accesses a newly-created > SQLite file before the creator process can set up the schema On POSIX systems, you can securely create a temp file that only your user can see via the

[sqlite] hard links and SQLite

2016-01-11 Thread Scott Hess
ail > to open the DB because it?s locked or open it and see the schema. > > I?m seeing some corruption in our SQLite files now but have had a > hard time tracking down the cause. Could it be because of the above logic? > What have folks here done to avoid the race condition orig