Re: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-30 Thread Toby Dickenson

On Thu, 29 Jun 2000 17:19:32 -0400, Shane Hathaway
[EMAIL PROTECTED] wrote:

Quite simply, Data.fs is appended, not modified, except during a pack
operation.  Think of it like a binary log file.  Therefore there are
only two ways a simple, live backup of Data.fs can fail:

1) You do the copy in the middle of a pack operation.  For most,
packing is a rare operation.  But if this happened, your backup would
be worthless.

I think its better than that. FileStorage packs itself into a
temporary file, which is renamed to data.fs only when the pack is
complete.

2) You get a partly truncated record at the very end.  This is actually
not a big deal, and AFAIK in most cases ZODB is capable of
automatically rolling back a partially committed transaction.


I think the other exception is during an Undo, where Zope will rewrite
one byte in the middle of the file.


Toby Dickenson
[EMAIL PROTECTED]

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Wilkinson Charlie E
Title: RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)





Just to start by pointing out the bloody obvious:


- Restoring from backup means you lose all data between
 backup date/time and system failure. Sucks, but it
 beats losing *all* your data. (RAID5 anyone?)
- With that in mind, *consistency* of the backup database
 takes the highest priority. All existing transactions
 should be fully completed - no half-done updates, etc.


It seems to me that the easiest way to get there with ZODB
might be to just make a simple file copy of the database and
have some kind of integrity checking tool to run against the
static backup copy to weed out the junk. Is this possible,
does it already exist?


Probably a dumb question, but would Zope itself do this if
you fired it off against an inconsistent ZODB?


Another random thought is that if ZODB transactions and
writes are atomic, than none of this should be an issue.
Anyone know the answer to that one?


-cw-


 -Original Message-
 From: Chris Withers [mailto:[EMAIL PROTECTED]]
 Sent: Thursday, June 29, 2000 7:49 AM
 To: Erik Enge
 Cc: Oleg Broytmann; Zope Mailing List
 Subject: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)
 
 
 Erik Enge wrote:
  
  On Thu, 29 Jun 2000, Chris Withers wrote:
  
   Hmm, about extending this so you have 'rotating data.fs 
 files' in the
   same way you have rotating log files?
  
  In general, yes :-). Do you think it would be solving the 
 right problem
  the right way?
 
 I think so, but that's only my view.
 
 If anyone's got any better ideas, please pipe up :-)
 
 cheers,
 
 Chris
 
 ___
 Zope maillist - [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope
 ** No cross posts or HTML encoding! **
 (Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )
 





RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Jeff K. Hoffman

On Thu, 29 Jun 2000, Wilkinson Charlie E wrote:

 Another random thought is that if ZODB transactions and
 writes are atomic, than none of this should be an issue.
 Anyone know the answer to that one?

It is my understanding that the ZODB on disk is always in a consistent
state; Jim has said, in the past, that you can simply make a copy of
Data.fs, even while Zope is running, without problems.

Based on that, I think the answer to the above is yes.

 -cw-

--Jeff

---
Jeff K. Hoffman   704.849.0731 x108
Chief Technology Officer  mailto:[EMAIL PROTECTED]
Going Virtual, L.L.C. http://www.goingv.com/


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Erik Enge

On Thu, 29 Jun 2000, Jeff K. Hoffman wrote:

 It is my understanding that the ZODB on disk is always in a consistent
 state; Jim has said, in the past, that you can simply make a copy of
 Data.fs, even while Zope is running, without problems.

But the Data.fs has to be updated sometimes.  And if you read from a file
that is changing, the integrity of your data isn't.  How can you make sure
that the file is not written to during a certain period of time?  You
can't?


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Bill Anderson

 Wilkinson Charlie E wrote:
 
 Just to start by pointing out the bloody obvious:
 
 - Restoring from backup means you lose all data between
   backup date/time and system failure.  Sucks, but it
   beats losing *all* your data.  (RAID5 anyone?)


A) Please: no html!
B) I just had one weird and wild thought. A RAID5 type of storage using
ZODBs instead of blocl devices. Probably not doable, and might not gain
anything if doable, but it was one of those ideas that shoots across
your brain and sounds cool at first. Just thought I'd share.

:)

-- 
"Linux: the operating system with a CLUE...
Command Line User Environment".

seen in a posting on comp.software.testing

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Jeff K. Hoffman

On Thu, 29 Jun 2000, Erik Enge wrote:

 On Thu, 29 Jun 2000, Jeff K. Hoffman wrote:
 
  It is my understanding that the ZODB on disk is always in a consistent
  state; Jim has said, in the past, that you can simply make a copy of
  Data.fs, even while Zope is running, without problems.
 
 But the Data.fs has to be updated sometimes.  And if you read from a file
 that is changing, the integrity of your data isn't.  How can you make sure
 that the file is not written to during a certain period of time?  You
 can't?

Now that I'm in the spotlight I can't seem to find the message I was
referring to in the archives; I sure hope I didn't misquote anyone (i.e.
Jim).

Unfortunately, I don't have an answer to your question. I still believe I
am correct, but cannot tell you, technically, why. I have some theories,
but I don't want to cloud the issue anymore with my hypotheses.

Let's hope someone who does know chimes in.

--Jeff

---
Jeff K. Hoffman   704.849.0731 x108
Chief Technology Officer  mailto:[EMAIL PROTECTED]
Going Virtual, L.L.C. http://www.goingv.com/


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Wilkinson Charlie E
Title: RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)





 -Original Message-
 From: Erik Enge [mailto:[EMAIL PROTECTED]]
 
 On Thu, 29 Jun 2000, Jeff K. Hoffman wrote:
 
  It is my understanding that the ZODB on disk is always in a 
 consistent
  state; Jim has said, in the past, that you can simply make a copy of
  Data.fs, even while Zope is running, without problems.
 
 But the Data.fs has to be updated sometimes. And if you read 
 from a file
 that is changing, the integrity of your data isn't. How can 
 you make sure
 that the file is not written to during a certain period of time? You
 can't?


You can.


This would be consistent with atomic updates. What it means is
that a transaction is not written to the Data.fs file until it has
completed, and when it is finally written out, it's done as an
atomic operation, i.e. at the system level it is a single write
operation and nothing else happens on the system until it completes.


From that you can infer that if you were reading the Data.fs file
(making your backup copy) at the same time, it would be impossible
to read the file in the middle of an update. If both read and
write requests hit Data.fs at essentially the same time, one goes
first and the other waits until the first completes. So if the
write gets first shot, the read will block until the write (meaning
a completed transaction/update) finishes, and your read will include
that last update. If your read gets first shot, then it will not
include the update. You will under no circumstances get *half* an
update.


One of the other things that makes this possible is ZODB being of
a journaling nature. i.e., all updates are *appended* to Data.fs.
This is why you must periodically pack the database, which
conglomerates all the updates back into the base datastore (or
whatever those wacky Zopistas call it. ;) that comprises the first
part of the Data.fs file.


I think that's all correct. Someone please throttle me senseless
if not. :-)


-cw-





RE: [Zope] Backing Up Zope (was: Re: [Zope] Data.fs.lock?)

2000-06-29 Thread Erik Enge

On Thu, 29 Jun 2000, Wilkinson Charlie E wrote:

 This would be consistent with "atomic updates."  What it means is
 that a transaction is not written to the Data.fs file until it has
 completed, and when it is finally written out, it's done as an
 atomic operation, i.e. at the system level it is a single write
 operation and nothing else happens on the system until it completes.

Brilliant!  No worries, then :-)

(by the way, loose the HTML)


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )