Re: [sqlite] [SOLVED] Re: [sqlite] nfs 'sillynames'
On Thu, 16 Sep 2004 [EMAIL PROTECTED] wrote:
If this were implemented, it would remove the capability to go into the
background. A common thing for a daemon process to do upon start-up is to
get everything initialized (e.g. open databases and do other things that
could fail and would need to be reported to the user), and then fork() and
the parent process exits, leaving the child running.
??
do a man 2 fcntl and you'll see:
...
F_GETFD
Read the close-on-exec flag. If the FD_CLOEXEC bit is 0, the
file will remain open across exec, otherwise it will be closed.
...
the close-on-exec bit has nothing to do with forking, only exec'ing. therefore
daemon code would operate exactly as before. if fact the cose in question IS a
daemon and i'm doing this manually using /proc - but it's not portable.
There's nothing wrong with *either* the parent process continuing to use an
open database handle _or_ the child process continuing to use an open
database handle. It just can't be used by both, and should be closed by the
one not using it.
sure. the problem is that, when a fork occurs, the sqlite api may have many
files opens, not only the db handle, such as the db-journal file or
/var/tmp/ files and those files are not exposed to the api client.
therfore they have no reliable way to know which OTHER files to
close/keep-open. for example:
harp:~ > strace sqlite db 'create table t(f);begin;insert into t values (42);commit;'
2>&1| grep open | tail
open("/etc/passwd", O_RDONLY) = 3
open("/home/ahoward/.sqliterc", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/ahoward/db", O_RDWR|O_CREAT|O_LARGEFILE, 0644) = 3
open("/var/tmp/sqlite_ZRnHLHGmAfplF87", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4
open("/home/ahoward/db-journal", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 5
open("/home/ahoward", O_RDONLY|O_LARGEFILE) = 6
open("/var/tmp/sqlite_ZRnHLHGmAfplF87-journal", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
0600) = 7
open("/var/tmp", O_RDONLY|O_LARGEFILE) = 8
open("/home/ahoward/db-journal", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 5
open("/home/ahoward", O_RDONLY|O_LARGEFILE) = 6
so here we see that, if a child was created during a transaction it would need
to close quite a few files other than the db itself - it would be nearly
impossible for a client to know which ones to close before execing (not
forking).
Note that if your child does not need the sqlite database handle, it should
call sqlite_close() to release of all resources associated with that handle
(not just close the file). That should solve your problem.
that IS good advice ;-) i'll see if that does the trick. i'm only worried
that closing the database in the child may do something like flush file handles
or commit/abort the current transaction - that would be VERY bad if it occured
in both parent and child.
kind regards.
-a
--
===
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===
Re: [sqlite] [SOLVED] Re: [sqlite] nfs 'sillynames'
"Ara.T.Howard" <[EMAIL PROTECTED]> writes: > the problem is that the child inherits all the open fds from the 'begin > transaction' (sqlite db-journal, etc.). in my case, the child never uses > these at all. regardless, when the parent does the unlink the sillyname is > created. > > the child could attempt to close this fd, but it's difficult for the child > to determine which files to close. i think a better solution would be for > the sqlite api to set the close-on-exec bit of any files created during a > transaction - that way a fork/exec will not inherit all those open files. > in fact - it seem as if setting the close-on-exec bit on ALL files the > sqlite api opens is the right thing to do since you'd never want them open > after an exec right? If this were implemented, it would remove the capability to go into the background. A common thing for a daemon process to do upon start-up is to get everything initialized (e.g. open databases and do other things that could fail and would need to be reported to the user), and then fork() and the parent process exits, leaving the child running. There's nothing wrong with *either* the parent process continuing to use an open database handle _or_ the child process continuing to use an open database handle. It just can't be used by both, and should be closed by the one not using it. Note that if your child does not need the sqlite database handle, it should call sqlite_close() to release of all resources associated with that handle (not just close the file). That should solve your problem. Derrell
[sqlite] [SOLVED] Re: [sqlite] nfs 'sillynames'
On Fri, 3 Sep 2004, John LeSueur wrote:
a followup on this post:
i was running some straces on sqlite while it using transactions and think
i
may have found the source of the problem, we see this in in the strace
output:
...
...
close(5)= 0
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
...
...
i think what is happening is
...
...
close(5)= 0
here remote client opens db-journal
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
I think this file is created everytime you start a transaction. Maybe even if
you only perform read only commands.
The journal is what is used to keep track of the changes to the database.
Anyone else have any thoughts?
this is, of course, the case. i finally tracked down the cause: in my code i
do
begin transaction
get a job from database
fork to run job
update db with pid of job
commit transation
the problem is that the child inherits all the open fds from the 'begin
transaction' (sqlite db-journal, etc.). in my case, the child never uses
these at all. regardless, when the parent does the unlink the sillyname is
created.
the child could attempt to close this fd, but it's difficult for the child to
determine which files to close. i think a better solution would be for the
sqlite api to set the close-on-exec bit of any files created during a
transaction - that way a fork/exec will not inherit all those open files. in
fact - it seem as if setting the close-on-exec bit on ALL files the sqlite api
opens is the right thing to do since you'd never want them open after an exec
right?
thoughts?
-a
--
===
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===
Re: [sqlite] nfs 'sillynames'
On Fri, 3 Sep 2004, John LeSueur wrote: I think this file is created everytime you start a transaction. Maybe even if you only perform read only commands. The journal is what is used to keep track of the changes to the database. Anyone else have any thoughts? yes - it is. the thing is, unless a remote client unlinks it while another client has it open the sillynames would not appear. i can post a copy of one of the sillynamed files - but they are definitely the contents of db-journal. the question is why this is happening. running strace and looking at the fcntl lock calls lead me to think it __should__ be impossible for any client to unlink this file before someone else closed it - but that IS what i'm seeing happen... -a -- === | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov | PHONE :: 303.497.6469 | A flower falls, even though we love it; | and a weed grows, even though we do not love it. | --Dogen ===
Re: [sqlite] nfs 'sillynames'
a followup on this post:
i was running some straces on sqlite while it using transactions and
think i
may have found the source of the problem, we see this in in the strace
output:
...
...
close(5)= 0
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
...
...
i think what is happening is
...
...
close(5)= 0
here remote client opens db-journal
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
I think this file is created everytime you start a transaction. Maybe
even if you only perform read only commands.
The journal is what is used to keep track of the changes to the
database. Anyone else have any thoughts?
John LeSueur
Re: [sqlite] nfs 'sillynames'
On Fri, 3 Sep 2004, Ara.T.Howard wrote:
if you are unfamiliar with nfs sillynames, they occur when a file that is
open
on one client is removed or renamed on another.
i am seeing alot of these appear in an NFS directory i'm using to store a
sqlite database acessed by many clients. the access protocol is a
meta-transaction wrapped around an actual sqlite transaction using an
additional empty lockfile (db.lock). this is to ensure single writer
multiple
reader semantics for the entire network, eg.
lock_type = read # or perhaps write
aquire fcntl lock of lock_type on db.lock
open db.lock
start_transaction(lock_type)
db.execute(sql)
end_transaction
close db.lock
release fcntl lock
NONE of the applications has exited for about 30 days. ALL of the
application
are aquiring write locks on the db.lock so one process only is accessing the
db. NONE of the uses removes, renames, etc. the file - only reads and writes
are done via the sqlite api. the semantics are definitely single writer (and
potentially) many reader, if this we not so the application would crash
horribly almost instantly. the api i'm using throws an exception if it gets
SQLITE_BUSY and i have no busy handler so i am positive that the locking
works, and that no readers ever attempt to write (upgrade lock) and vice
versa.
i'm simply mystified as to what's creating the sillynames. they all appear
to
be the product of the sqlite api: opening them up in vi shows them to be a
binary file that's obviously part of the database since i can recognize many
strings from the database in them.
is worries me - it seems to imply that sqlite, at some point does a rename or
remove when some remote client has an open file handle. could this be
because
ALL of my operations (even reads) is inside a transaction?
more info...
the sillynamed files are only ever a minute or so old and disappear almost as
quickly (this would be after the last client called a close). the
application
is working great but i'd like to understand this as it concerns me somewhat.
the sqlite lib version is the latest 2.8 branch. the nfs server/client impl
are the latest patched redhat enterprise versions.
kind regards.
a followup on this post:
i was running some straces on sqlite while it using transactions and think i
may have found the source of the problem, we see this in in the strace output:
...
...
close(5)= 0
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
...
...
i think what is happening is
...
...
close(5)= 0
here remote client opens db-journal
unlink("/dmsp/moby-1-1/ahoward/shared/silly/db-journal") = 0
and the sillyname is created, only to disappear when the remote client closes
the db-journal.
...
...
the locking seems like it should prevent this - but the files themselves are
definitely subsets of the original - so i'm thinking there is some race
condition here.
-a
--
===
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===
