On Dec 17, 2007, at 9:39 AM, Alan Runyan wrote:

Makes sense.

Is zc.FileStorage an alternative packing strategy?

Yes. I expect to roll this work into 3.9. It is a separate package now so I can deploy it with 3.8. It is very much a work in progress.

 Does it use less memory?

No, it uses quite a bit more. :)

Memory is pretty cheap. The current packing approach beats the [EMAIL PROTECTED] out of the disk. This new approach tries to be much kinder by storing cross-reference data in memory rather than massively seeking to read cross-reference data from disk. Also, on linus systems, it uses posix_fadvise to avoid trashing the disk cache.

It 2-3 times as fast as the old pack algorithm and I think it will be much kinder to the server and other processes on the system.

It does most of the work in a sub-process so it can take better advantage of multiple CPUs and, although it uses more memory, it will to a much better job of returning the memory to the OS.

I'm not aware that reducing memory is a goal. If it is a goal, we can look at having the packing code actually use a database for the data it needs. This would slow down packing quite a bit, but could limit the amount of memory used.

Another thing I looked at was adding an option to skip GC. This seems to speed packing and reduce memory usage quite a bit. I plan to add this as an database configuration option. I'd like to be able to decide whether to do GC at pack time. The pack API doesn't support this. I think I'll eventually resort to using the extension mechanism to add additional pack methods. :)


Jim Fulton
Zope Corporation

For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to