Re: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
On Thu, 29 Jun 2000 17:19:32 -0400, Shane Hathaway <[EMAIL PROTECTED]> wrote: >Quite simply, Data.fs is appended, not modified, except during a pack >operation. Think of it like a binary log file. Therefore there are >only two ways a simple, live backup of Data.fs can fail: > >1) You do the copy in the middle of a pack operation. For most, >packing is a rare operation. But if this happened, your backup would >be worthless. I think its better than that. FileStorage packs itself into a temporary file, which is renamed to data.fs only when the pack is complete. >2) You get a partly truncated record at the very end. This is actually >not a big deal, and AFAIK in most cases ZODB is capable of >automatically rolling back a partially committed transaction. I think the other exception is during an Undo, where Zope will rewrite one byte in the middle of the file. Toby Dickenson [EMAIL PROTECTED] ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
On Thu, 29 Jun 2000, Wilkinson Charlie E wrote: > This would be consistent with "atomic updates." What it means is > that a transaction is not written to the Data.fs file until it has > completed, and when it is finally written out, it's done as an > atomic operation, i.e. at the system level it is a single write > operation and nothing else happens on the system until it completes. Brilliant! No worries, then :-) (by the way, loose the HTML) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
Title: RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?) > -Original Message- > From: Erik Enge [mailto:[EMAIL PROTECTED]] > > On Thu, 29 Jun 2000, Jeff K. Hoffman wrote: > > > It is my understanding that the ZODB on disk is always in a > consistent > > state; Jim has said, in the past, that you can simply make a copy of > > Data.fs, even while Zope is running, without problems. > > But the Data.fs has to be updated sometimes. And if you read > from a file > that is changing, the integrity of your data isn't. How can > you make sure > that the file is not written to during a certain period of time? You > can't? You can. This would be consistent with "atomic updates." What it means is that a transaction is not written to the Data.fs file until it has completed, and when it is finally written out, it's done as an atomic operation, i.e. at the system level it is a single write operation and nothing else happens on the system until it completes. From that you can infer that if you were reading the Data.fs file (making your backup copy) at the same time, it would be impossible to read the file "in the middle of an update." If both read and write requests hit Data.fs at essentially the same time, one goes first and the other waits until the first completes. So if the write gets first shot, the read will block until the write (meaning a completed transaction/update) finishes, and your read will include that last update. If your read gets first shot, then it will not include the update. You will under no circumstances get *half* an update. One of the other things that makes this possible is ZODB being of a "journaling" nature. i.e., all updates are *appended* to Data.fs. This is why you must periodically pack the database, which conglomerates all the updates back into the base datastore (or whatever those wacky Zopistas call it. ;) that comprises the first part of the Data.fs file. I think that's all correct. Someone please throttle me senseless if not. :-) -cw-
Re: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
"Jeff K. Hoffman" wrote: > > On Thu, 29 Jun 2000, Erik Enge wrote: > > > On Thu, 29 Jun 2000, Jeff K. Hoffman wrote: > > > > > It is my understanding that the ZODB on disk is always in a consistent > > > state; Jim has said, in the past, that you can simply make a copy of > > > Data.fs, even while Zope is running, without problems. > > > > But the Data.fs has to be updated sometimes. And if you read from a file > > that is changing, the integrity of your data isn't. How can you make sure > > that the file is not written to during a certain period of time? You > > can't? > > Now that I'm in the spotlight I can't seem to find the message I was > referring to in the archives; I sure hope I didn't misquote anyone (i.e. > Jim). > > Unfortunately, I don't have an answer to your question. I still believe I > am correct, but cannot tell you, technically, why. I have some theories, > but I don't want to cloud the issue anymore with my hypotheses. > > Let's hope someone who does know chimes in. Quite simply, Data.fs is appended, not modified, except during a pack operation. Think of it like a binary log file. Therefore there are only two ways a simple, live backup of Data.fs can fail: 1) You do the copy in the middle of a pack operation. For most, packing is a rare operation. But if this happened, your backup would be worthless. 2) You get a partly truncated record at the very end. This is actually not a big deal, and AFAIK in most cases ZODB is capable of automatically rolling back a partially committed transaction. Shane ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
On Thu, 29 Jun 2000, Erik Enge wrote: > On Thu, 29 Jun 2000, Jeff K. Hoffman wrote: > > > It is my understanding that the ZODB on disk is always in a consistent > > state; Jim has said, in the past, that you can simply make a copy of > > Data.fs, even while Zope is running, without problems. > > But the Data.fs has to be updated sometimes. And if you read from a file > that is changing, the integrity of your data isn't. How can you make sure > that the file is not written to during a certain period of time? You > can't? Now that I'm in the spotlight I can't seem to find the message I was referring to in the archives; I sure hope I didn't misquote anyone (i.e. Jim). Unfortunately, I don't have an answer to your question. I still believe I am correct, but cannot tell you, technically, why. I have some theories, but I don't want to cloud the issue anymore with my hypotheses. Let's hope someone who does know chimes in. --Jeff --- Jeff K. Hoffman 704.849.0731 x108 Chief Technology Officer mailto:[EMAIL PROTECTED] Going Virtual, L.L.C. http://www.goingv.com/ ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
> Wilkinson Charlie E wrote: > > Just to start by pointing out the bloody obvious: > > - Restoring from backup means you lose all data between > backup date/time and system failure. Sucks, but it > beats losing *all* your data. (RAID5 anyone?) A) Please: no html! B) I just had one weird and wild thought. A RAID5 type of storage using ZODBs instead of blocl devices. Probably not doable, and might not gain anything if doable, but it was one of those ideas that shoots across your brain and sounds cool at first. Just thought I'd share. :) -- "Linux: the operating system with a CLUE... Command Line User Environment". seen in a posting on comp.software.testing ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
On Thu, 29 Jun 2000, Jeff K. Hoffman wrote: > It is my understanding that the ZODB on disk is always in a consistent > state; Jim has said, in the past, that you can simply make a copy of > Data.fs, even while Zope is running, without problems. But the Data.fs has to be updated sometimes. And if you read from a file that is changing, the integrity of your data isn't. How can you make sure that the file is not written to during a certain period of time? You can't? ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
On Thu, 29 Jun 2000, Wilkinson Charlie E wrote: > Another random thought is that if ZODB transactions and > writes are atomic, than none of this should be an issue. > Anyone know the answer to that one? It is my understanding that the ZODB on disk is always in a consistent state; Jim has said, in the past, that you can simply make a copy of Data.fs, even while Zope is running, without problems. Based on that, I think the answer to the above is yes. > -cw- --Jeff --- Jeff K. Hoffman 704.849.0731 x108 Chief Technology Officer mailto:[EMAIL PROTECTED] Going Virtual, L.L.C. http://www.goingv.com/ ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
Title: RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?) Just to start by pointing out the bloody obvious: - Restoring from backup means you lose all data between backup date/time and system failure. Sucks, but it beats losing *all* your data. (RAID5 anyone?) - With that in mind, *consistency* of the backup database takes the highest priority. All existing transactions should be fully completed - no half-done updates, etc. It seems to me that the easiest way to get there with ZODB might be to just make a simple file copy of the database and have some kind of integrity checking tool to run against the static backup copy to weed out the junk. Is this possible, does it already exist? Probably a dumb question, but would Zope itself do this if you fired it off against an inconsistent ZODB? Another random thought is that if ZODB transactions and writes are atomic, than none of this should be an issue. Anyone know the answer to that one? -cw- > -Original Message- > From: Chris Withers [mailto:[EMAIL PROTECTED]] > Sent: Thursday, June 29, 2000 7:49 AM > To: Erik Enge > Cc: Oleg Broytmann; Zope Mailing List > Subject: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?) > > > Erik Enge wrote: > > > > On Thu, 29 Jun 2000, Chris Withers wrote: > > > > > Hmm, about extending this so you have 'rotating data.fs > files' in the > > > same way you have rotating log files? > > > > In general, yes :-). Do you think it would be solving the > right problem > > the right way? > > I think so, but that's only my view. > > If anyone's got any better ideas, please pipe up :-) > > cheers, > > Chris > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) >
