Re: [ZODB-Dev] minimizing the need for database packing
On Dec 29, 2007, at 3:22 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-12-28 10:20 -0500: ... There Berkely Database Storage supported automatic incremental packing without garbage collection. If someone were to revitalize that effort and if one was willing to do without cyclic garbage collection, then that storage would remove the need for the sort of disruptive pack we have with FileStorage now. Why do you consider pack disruptive? Disruptive isn't exactly the word I was looking for, although the current pack approach is disruptive. It causes a concentrated I/O and CPU load. For our databases, database performance drops significantly, especially during the later stages of packing. Note that I'm working on a new FileStorage packer that is 2-3 times faster and, I believe, much less disruptive than the current packing algorithm. If you are at it: I think the lock which protects the finish test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the finish test has failed. I don't know what you are referring to. Could you be more specific? Note that the packing code in zc.FileStorage releases the commit lock far more frequently than the old pack code and doesn't begin acquiring it until after copying data written up to the time packing started, rather than after writing data up to the pack time. I expect that the packing code in 3.9 will acquire the commit lock much later. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
On Jan 2, 2008, at 1:30 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-12-29 16:06 -0500: ... If you are at it: I think the lock which protects the finish test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the finish test has failed. I don't know what you are referring to. Could you be more specific? The old pack code acquires the commit lock at the start of the copyRest (copy transactions after pack time) phase and releases it only every 20 copyOne calls for the duration of one copyOne. OK. I don't know what you mean by the finish test. As I mentioned, the new packing algorithm holds the commit lock much less than the old one does. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
Jim Fulton wrote at 2007-12-28 10:20 -0500: ... There Berkely Database Storage supported automatic incremental packing without garbage collection. If someone were to revitalize that effort and if one was willing to do without cyclic garbage collection, then that storage would remove the need for the sort of disruptive pack we have with FileStorage now. Why do you consider pack disruptive? Note that I'm working on a new FileStorage packer that is 2-3 times faster and, I believe, much less disruptive than the current packing algorithm. If you are at it: I think the lock which protects the finish test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the finish test has failed. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] minimizing the need for database packing
Hi, is there any way to minimize the need for database packing? I wouldn't mind loosing the ability of undoing transactions. thanks, -- Flávio Codeço Coelho My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition. Indira Gandhi registered Linux user # 386432 get counted at http://counter.li.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
On Dec 28, 2007, at 5:26 AM, Flavio Coelho wrote: Hi, is there any way to minimize the need for database packing? I wouldn't mind loosing the ability of undoing transactions. There's a significant performance benefit to keeping some non-current data to support multi-version concurrency control. There Berkely Database Storage supported automatic incremental packing without garbage collection. If someone were to revitalize that effort and if one was willing to do without cyclic garbage collection, then that storage would remove the need for the sort of disruptive pack we have with FileStorage now. Note that I'm working on a new FileStorage packer that is 2-3 times faster and, I believe, much less disruptive than the current packing algorithm. It also provides an option to skip garbage collection, making it twice again as fast. It is available now as a separate monkey-patch in the zc.FileStorage package. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
