Re: [sqlite] Reading across processes on Solaris
Joe Wilson <[EMAIL PROTECTED]> wrote: > > I'll continue investigating and try to get to the bottom of this. Thanks > > for the sanity-check. > > Might there be some autocommit/sql statement batching difference for the > driver or your app on different platforms? No; one of the beauties of developing in Python is that it minimizes the amount of OS-specific code you need to write. I have to apologize for the misdiagnosis. I accepted my colleague's diagnosis of "writes from writer not appearing in view of database from reader on Solaris only" at face value. After some hours of attempting to reproduce the problem, it now appears they were showing up all along. So: sqlite list 1, list subscriber 0. Thanks again, Joe. Charles -- --- Charles Cazabon <[EMAIL PROTECTED]> --- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Reading across processes on Solaris
> I'd have thought so, too. I've confirmed fdatasync shows up in the symbols in > the compiled sqlite library, and that two instances of the `sqlite3` SQL shell > don't show the problem. Unfortunately, two minimal Python programs don't show > the problem either. > > That points to the application code -- except that I only see this behaviour > on Solaris, and there's no OS-specific code in the database-related portions > of the application. I'll continue investigating and try to get to the bottom > of this. Thanks for the sanity-check. Might there be some autocommit/sql statement batching difference for the driver or your app on different platforms? If all else fails, try truss on the "failing" Solaris processes and compare its output to strace for the same "correct" application on Linux. Sucker-punch spam with award-winning protection. Try the free Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/features_spam.html - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Reading across processes on Solaris
Joe Wilson <[EMAIL PROTECTED]> wrote: > --- Charles Cazabon <[EMAIL PROTECTED]> wrote: > > > > On most platforms, the reader sees new data appear in the database > > periodically as the writer creates new records. But on Solaris, the > > reader never sees any updates -- it only ever sees whatever data was in > > the database when the reader first opened it, even though the writer is > > continuing to insert new data periodically. [...] > > I thought that SQLite's use of fdatasync on Solaris should be enough to > synchronize reads and writes from various processes: > > The fdatasync() function forces all currently queued I/O > operations associated with the file indicated by file > descriptor fildes to the synchronized I/O completion state. I'd have thought so, too. I've confirmed fdatasync shows up in the symbols in the compiled sqlite library, and that two instances of the `sqlite3` SQL shell don't show the problem. Unfortunately, two minimal Python programs don't show the problem either. That points to the application code -- except that I only see this behaviour on Solaris, and there's no OS-specific code in the database-related portions of the application. I'll continue investigating and try to get to the bottom of this. Thanks for the sanity-check. Charles -- --- Charles Cazabon <[EMAIL PROTECTED]> --- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Reading across processes on Solaris
--- Charles Cazabon <[EMAIL PROTECTED]> wrote: > I'm using sqlite (through the pysqlite wrapper, but this behaviour seems > unrelated to the wrapper) in an application on various platforms. One process > create an sqlite database and starts writing data to it; another process opens > the sqlite database and reads from it. > > On most platforms, the reader sees new data appear in the database > periodically as the writer creates new records. But on Solaris, the reader > never sees any updates -- it only ever sees whatever data was in the database > when the reader first opened it, even though the writer is continuing to > insert new data periodically. > > I've seen similar behaviour with non-database files on Solaris -- writes to a > file across processes aren't seen by the reader unless the reader supplies the > O_RSYNC flag to the open(2) call. It seems to be a Solaris peculiarity, as I > don't see this behaviour on Linux, *BSD, or other commercial Unices. > > I've looked at the sqlite source code, and it does not appear to be supplying > the O_RSYNC or O_SYNC flags to open(2). I thought that SQLite's use of fdatasync on Solaris should be enough to synchronize reads and writes from various processes: The fdatasync() function forces all currently queued I/O operations associated with the file indicated by file descriptor fildes to the synchronized I/O completion state. The functionality is as described for fsync(3C) (with the symbol _XOPEN_REALTIME defined), with the exception that all I/O operations are completed as defined for synchronised I/O data integrity completion. As far as I know, this ought to have the same effect as O_RSYNC: O_RSYNC If this flag is set, reading the data will block until any pending writes which affect the data are complete. Consider the situation where we want to read a block of data, which another process is updating. If this flag is not set, it is indeterminate whether the data returned will be that which is on the disk, or that which is scheduled to be written. Can you confirm that the pysqlite wrapper that you're using compiled sqlite with fdatasync? nm your_python_sqlite_wrapper.so | grep fdatasync or nm sqlite3.so | grep fdatasync Sucker-punch spam with award-winning protection. Try the free Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/features_spam.html - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Reading across processes on Solaris
--- Charles Cazabon <[EMAIL PROTECTED]> wrote: > I'm using sqlite (through the pysqlite wrapper, but this behaviour seems > unrelated to the wrapper) in an application on various platforms. One process > create an sqlite database and starts writing data to it; another process opens > the sqlite database and reads from it. > > On most platforms, the reader sees new data appear in the database > periodically as the writer creates new records. But on Solaris, the reader > never sees any updates -- it only ever sees whatever data was in the database > when the reader first opened it, even though the writer is continuing to > insert new data periodically. > > I've seen similar behaviour with non-database files on Solaris -- writes to a > file across processes aren't seen by the reader unless the reader supplies the > O_RSYNC flag to the open(2) call. It seems to be a Solaris peculiarity, as I > don't see this behaviour on Linux, *BSD, or other commercial Unices. > > I've looked at the sqlite source code, and it does not appear to be supplying > the O_RSYNC or O_SYNC flags to open(2). > > So, my questions are: > > 1) Has anyone else run into this issue? Is there a known way to work around > it? > > 2) Should the open(2) call be modified to provide the O_RSYNC flag? Would > this have nasty side effects? One would think that if such an issue existed in sqlite under Solaris someone would have reported it long ago - but you never know. I don't have access to Solaris at the moment, but you might try testing with 2 instances of the commandline shell, sqlite3, to make sure it's not a python driver issue. If you have any trouble building sqlite3 from sources, just post any compile issue to the list. Make sure you are running sqlite3 on a database file on a _local_ file system. Perform updates/inserts/deletes in one sqlite3 instance and see what happens when you do an appropriate select from the other sqlite3 instance. If it is proven that Solaris sqlite3 does indeed require the O_RSYNC flag in its open, I'm sure it could be put into os_unix.c with an appropriate #ifdef. Need Mail bonding? Go to the Yahoo! Mail Q for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list=396546091 - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Reading across processes on Solaris
Greetings, I searched the web and the list archives but found nothing relevant to this problem, so here goes... I'm using sqlite (through the pysqlite wrapper, but this behaviour seems unrelated to the wrapper) in an application on various platforms. One process create an sqlite database and starts writing data to it; another process opens the sqlite database and reads from it. On most platforms, the reader sees new data appear in the database periodically as the writer creates new records. But on Solaris, the reader never sees any updates -- it only ever sees whatever data was in the database when the reader first opened it, even though the writer is continuing to insert new data periodically. I've seen similar behaviour with non-database files on Solaris -- writes to a file across processes aren't seen by the reader unless the reader supplies the O_RSYNC flag to the open(2) call. It seems to be a Solaris peculiarity, as I don't see this behaviour on Linux, *BSD, or other commercial Unices. I've looked at the sqlite source code, and it does not appear to be supplying the O_RSYNC or O_SYNC flags to open(2). So, my questions are: 1) Has anyone else run into this issue? Is there a known way to work around it? 2) Should the open(2) call be modified to provide the O_RSYNC flag? Would this have nasty side effects? I appreciate any responses. I'm subscribed to the list, so I don't need to be cc'd. Charles -- --- Charles Cazabon <[EMAIL PROTECTED]> --- - To unsubscribe, send email to [EMAIL PROTECTED] -
