[Tim Peters]
The ZEO client cache is stored in a fixed-size disk file.  When a ZEO
client needs to create this file for the first time, it's trying to
ensure there's enough space on disk for it at the start, and reserve
that disk space then, rather than risk dying with a "no space left on
device" error umpteen hours later.  Alas, there is no portable way to
do so (the C standard says nothing about physical devices such as
disks).


[Paul Winkler]
So it might work as intended on some systems but inspire false
confidence on others. That's sad, but I guess we can't do much about
it.

The full intent is almost met on Windows.  On other systems it at
least /informs/ the OS about the intended size of the file.  In any
case it does no harm.

[Paul]
...
"saves" means. Does the behavior vary on different filesystems?

Yes.

Sounds like it optimizes for sparse files.  Not all filesystems do.

OK... I'm still wondering on which filesystems this code actually does
guarantee sufficient space. Our experiments suggest that ext2, ext3,
and reiserfs optimize for sparse files so there is no such guarantee.
AFAICT from some quick googling and wikipediaing, the same is true for
NTFS, XFS, JFS, ZFS. I suspect we've accounted for the majority of the
production Zope installations in the world.  The only fs I found that
has no sparse file support is HFS+.  Not sure about UFS (I found
people claiming both no and yes).

NTFS has "optional" sparse-file support, meaning there is a way to
create sparse files under NTFS, but it's not the default.  It isn't
exposed at the C stdio level (creating a sparse file under NTFS
requires additional Windows-specific API calls), and Python builds on
C stdio.

In any case there's no benefit to sparse files in this context:  while
the ZEO cache file starts out "almost empty", every byte is eventually
used, so it seems good to make whatever cheap efforts can be made to
minimize the chance that ZEO will die after umpteen hours if there's
not enough space for the client cache file it needs at the start.

If you care, the intent could probably be better served by changing
the code to write a junk byte at every (say) thousandth byte offset
throughout the file (at least one non-NUL byte per disk block).  Alas,
there would still be no guarantee that there's actually enough space
on the physical disk to store it.

OK, then I don't see how we could get a real guarantee without
actually writing junk to every byte, which might be just a little
slower :)

Still no guarantee (e.g., there might be enough RAM to hold all the
bytes in I/O buffers, but not enough disk space remaining to
materialize them), but writing a non-NUL byte per disk block would be
as effective in practice as writing to every byte position.

Short of that, since crystal ball technology is still not widely
deployed outside the Python Secret Underground, we can't know if
currently free space will still be available when we need it.

That's right.  On all platforms the current dance suffices to inform
the I/O system of how large a file ZEO needs (note that just seeking
to the max size does not suffice -- a byte needs to be written at the
end to set the EOF pointer).  On NTFS that's almost reliable (see
above); on other systems it may or may not be.
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to