(cc zodb-dev, who may also be interested)

On Wednesday 13 November 2002 4:18 pm, Barry A. Warsaw wrote:
> >>>>> "TD" == Toby Dickenson <[EMAIL PROTECTED]> writes:
>     >> worse" part.  If you've enable autopacking and you don't
>     >> cleanly close the storage, you won't exit the process because
>     >> the autopack thread won't get stopped and joined.
>     TD> Yes, I had the same problem with DirectoryStorage, which uses
>     TD> a seperate thread to move writes from its journal directory
>     TD> into the database directory.
> Ah.  Is this transactional?  What would happen if the thread got
> killed in the middle of the move?

There are lots of ways to look at that question....

I am relying on the filesystem guarantee that the files are all either in 
their original directory, or their new directory. DirectoryStorage has its 
equivalent of BerkeleyStorage's autorecovery which, at startup, 
asynchronously resumes the file moving wherever it left off.

A more interesting question is what happens if the thread is killed when it 
has moved some, but not all, of the files which relate to one transaction. In 
DirectoryStorage terminology this leaves the data directory in a state which 
"is not a snapshot". The only disadvantage is that you can not use the tools 
which assume the data directory is a snapshot: the checking tool, the 
incremental back tool, and the replication tool.

>     >> We could make the
>     >> autopack thread a daemon process
>     TD> Thats how DirectoryStorage now works.
> Hmm, maybe we make it a daemon thread and put a timeout on the join,
> so we'll try to exit cleanly and won't hang if we can't. 

Because some of the operations performed in the daemon thread take a long time 
to complete? Would it be possible to break those operations into 
transactionally-coherent chunks which complete in a reasonable time? close() 
could wait for the current chunk to finish.

> Autopacking
> should be safe transactionally, but we'd have to do a recovery if it
> got killed uncleanly.

Doesnt having a good checkpointing strategy mean that this should never take 

>     TD> Last time I looked at your BerkeleyDB storages, the
>     TD> administrator needed to implement his own checkpointing
>     TD> policy. I always thought that was a disadvantage. Would it now
>     TD> be a good idea for the storages to trigger their own
>     TD> checkpointing inside the autopacker thread?
> Actually, now you can configure the storages to automatically
> checkpoint every nth ZODB transaction.  Checkpointing in a thread may
> or may not provide additional benefit.

Interesting. BerkeleyStorage's automatic checkpointing is currently triggered 
inside a commit or abort. This means the checkpoint overhead is incurred at 
the one time you dont want it - while a user is waiting for his transaction 
to commit. DirectoryStorage's main use for the thread is to perform its 
equivalent asynchronously. Assuming your storage is not saturated with writes 
(which only happens in benchmarks ;-) then checkpointing happens for free.

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope )

Reply via email to