RE: [sqlite] Your Concurrency Idea
Dan, The journal file contains the "before" image of a modified database block not the after image. during write operations: before images are written to the journal file for recovery. Then the actual change is written to the .db file. A commit is an atomic operation due to unlinking the journal file. During recovery the journal file is played onto its corresponding .db file. This has the affect of a ROLLBACK command return the .db to a consistent state. So using the journal file to allow reads to continue is the correct place, given the current design of sqlite journalling and transactions. Dan Petitt <[EMAIL PROTECTED]> wrote: The journal file sounds like the wrong place to put it, the journal seems to be the place to store information that needs writing to the database on completion; storing some read-only information in the same file seems at odds with its current purpose. Perhaps a separate file(s) might be more appropriate to store this information, this might resolve your file format issues but also improve (but not eliminate) any performance considerations; perhaps the information could even (optionally) be stored in memory to greatly improve things. Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 30 October 2007 17:15 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Your Concurrency Idea "Dan Petitt" wrote: > Richard, i noticed this ticket in the system: > http://www.sqlite.org/cvstrac/tktview?tn=2417,8 > > And wondered if its something that is getting any serious thought or > something that is just a faint possibility? > Seems to be an incompatibly file format change, which more or less rules it out for any consideration. Also unaddressed in the proposal is how to locate a particular page within the journal file without having to do (performance killing) sequential scan of the possible very large file. -- D. Richard Hipp - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
RE: [sqlite] Your Concurrency Idea
The journal file sounds like the wrong place to put it, the journal seems to be the place to store information that needs writing to the database on completion; storing some read-only information in the same file seems at odds with its current purpose. Perhaps a separate file(s) might be more appropriate to store this information, this might resolve your file format issues but also improve (but not eliminate) any performance considerations; perhaps the information could even (optionally) be stored in memory to greatly improve things. Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 30 October 2007 17:15 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Your Concurrency Idea "Dan Petitt" <[EMAIL PROTECTED]> wrote: > Richard, i noticed this ticket in the system: > http://www.sqlite.org/cvstrac/tktview?tn=2417,8 > > And wondered if its something that is getting any serious thought or > something that is just a faint possibility? > Seems to be an incompatibly file format change, which more or less rules it out for any consideration. Also unaddressed in the proposal is how to locate a particular page within the journal file without having to do (performance killing) sequential scan of the possible very large file. -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Your Concurrency Idea
DRH, I seemed to recall posting that ticket. I don't have any good solutions for the txn journal lookup at the moment. Maybe a form of block chaining from the sourcing DB to the txn journal offset. Each txn journal modification would point to the prior versions offset. So that multiple changes to a block within a transaction could be efficiently traversed within the TXN journal. Only the last change offset is needed to be maintained, ie a tail pointer. When a change is written to the journal first the current pending offset would need to be made to the block. Then the original version would be written. The on disk txn version (writer) would contain a reference to the location in the journal, Along with a global TXN change number from the Master block. Readers would determine that the on disk version does not match the Master Block version and would use the on disks journal offset to access the journal file. Locks on the journal file would need to be maintained indicating a shared read request so that the jouranl would not go away whilst the read is in progress. Just my blue sky idea of resolving read/write concurrency. At a minimum either the original block would need to be made accessible (this idea) or a mechanism to perform block versioning and block level rollback to a prior version would need to be developed in order to improve read/write concurrency. Block level rollback to a version can be accomplished but would be a major change in direction and philosophy. Regards, Ken [EMAIL PROTECTED] wrote: "Dan Petitt" wrote: > Richard, i noticed this ticket in the system: > http://www.sqlite.org/cvstrac/tktview?tn=2417,8 > > And wondered if its something that is getting any serious thought or > something that is just a faint possibility? > Seems to be an incompatibly file format change, which more or less rules it out for any consideration. Also unaddressed in the proposal is how to locate a particular page within the journal file without having to do (performance killing) sequential scan of the possible very large file. -- D. Richard Hipp - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Your Concurrency Idea
"Dan Petitt" <[EMAIL PROTECTED]> wrote: > Richard, i noticed this ticket in the system: > http://www.sqlite.org/cvstrac/tktview?tn=2417,8 > > And wondered if its something that is getting any serious thought or > something that is just a faint possibility? > Seems to be an incompatibly file format change, which more or less rules it out for any consideration. Also unaddressed in the proposal is how to locate a particular page within the journal file without having to do (performance killing) sequential scan of the possible very large file. -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
