On Dec 17, 2007, at 9:39 AM, Alan Runyan wrote:
Makes sense.
Is zc.FileStorage an alternative packing strategy?
Yes. I expect to roll this work into 3.9. It is a separate package
now so I can deploy it with 3.8. It is very much a work in progress.
Does it use less memory?
No, it uses quite a bit more. :)
Memory is pretty cheap. The current packing approach beats the [EMAIL PROTECTED]
out of the disk. This new approach tries to be much kinder by storing
cross-reference data in memory rather than massively seeking to read
cross-reference data from disk. Also, on linus systems, it uses
posix_fadvise to avoid trashing the disk cache.
It 2-3 times as fast as the old pack algorithm and I think it will be
much kinder to the server and other processes on the system.
It does most of the work in a sub-process so it can take better
advantage of multiple CPUs and, although it uses more memory, it will
to a much better job of returning the memory to the OS.
I'm not aware that reducing memory is a goal. If it is a goal, we can
look at having the packing code actually use a database for the data
it needs. This would slow down packing quite a bit, but could limit
the amount of memory used.
Another thing I looked at was adding an option to skip GC. This seems
to speed packing and reduce memory usage quite a bit. I plan to add
this as an database configuration option. I'd like to be able to
decide whether to do GC at pack time. The pack API doesn't support
this. I think I'll eventually resort to using the extension mechanism
to add additional pack methods. :)
Jim
--
Jim Fulton
Zope Corporation
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
