Re: [ZODB-Dev] minimizing the need for database packing
On Jan 3, 2008, at 1:11 PM, Dieter Maurer wrote: ... When we use it, I will have a look whether I still see optimization potential :-) I'm sure you will. In fact, I'm pretty sure that it could be sped up by optimizing some of the Python code. For our largish databases, the old pack typically used a small percentage of the CPU most of the time, whereas the new packing code is mostly CPU bound, because it does much saner I/O. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
Jim Fulton wrote at 2008-1-2 13:32 -0500: > ... >> The old "pack" code acquires the "commit" lock at the start >> of the "copyRest" (copy transactions after pack time) phase >> and releases it only every 20 "copyOne" calls for the duration >> of one "copyOne". > > >OK. I don't know what you mean by the "finish test". A lock needs to be held at the end of "copyRest" until the state is changed (i.e. "Data.fs.pack" is renamed to "Data.fs"). This is the "finish test". >As I mentioned, the new packing algorithm holds the commit lock much >less than the old one does. Fine. When we use it, I will have a look whether I still see optimization potential :-) -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
On Jan 2, 2008, at 1:30 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-12-29 16:06 -0500: ... If you are at it: I think the lock which protects the "finish" test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the "finish" test has failed. I don't know what you are referring to. Could you be more specific? The old "pack" code acquires the "commit" lock at the start of the "copyRest" (copy transactions after pack time) phase and releases it only every 20 "copyOne" calls for the duration of one "copyOne". OK. I don't know what you mean by the "finish test". As I mentioned, the new packing algorithm holds the commit lock much less than the old one does. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
Jim Fulton wrote at 2007-12-29 16:06 -0500: > ... >> If you are at it: I think the lock which protects the "finish" test >> is hold too long. Currently, it is just release for a very short time >> and then immeadiately reacquired. It should be safe to release it >> immediately after the "finish" test has failed. > > >I don't know what you are referring to. Could you be more specific? The old "pack" code acquires the "commit" lock at the start of the "copyRest" (copy transactions after pack time) phase and releases it only every 20 "copyOne" calls for the duration of one "copyOne". -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
On Dec 29, 2007, at 3:22 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-12-28 10:20 -0500: ... There Berkely Database Storage supported automatic incremental packing without garbage collection. If someone were to revitalize that effort and if one was willing to do without cyclic garbage collection, then that storage would remove the need for the sort of disruptive pack we have with FileStorage now. Why do you consider "pack" disruptive? Disruptive isn't exactly the word I was looking for, although the current pack approach is disruptive. It causes a concentrated I/O and CPU load. For our databases, database performance drops significantly, especially during the later stages of packing. Note that I'm working on a new FileStorage packer that is 2-3 times faster and, I believe, much less disruptive than the current packing algorithm. If you are at it: I think the lock which protects the "finish" test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the "finish" test has failed. I don't know what you are referring to. Could you be more specific? Note that the packing code in zc.FileStorage releases the commit lock far more frequently than the old pack code and doesn't begin acquiring it until after copying data written up to the time packing started, rather than after writing data up to the pack time. I expect that the packing code in 3.9 will acquire the commit lock much later. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
Jim Fulton wrote at 2007-12-28 10:20 -0500: > ... >There Berkely Database Storage supported automatic incremental packing >without garbage collection. If someone were to revitalize that effort >and if one was willing to do without cyclic garbage collection, then >that storage would remove the need for the sort of disruptive pack we >have with FileStorage now. Why do you consider "pack" disruptive? >Note that I'm working on a new FileStorage packer that is 2-3 times >faster and, I believe, much less disruptive than the current packing >algorithm. If you are at it: I think the lock which protects the "finish" test is hold too long. Currently, it is just release for a very short time and then immeadiately reacquired. It should be safe to release it immediately after the "finish" test has failed. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] minimizing the need for database packing
On Dec 28, 2007, at 5:26 AM, Flavio Coelho wrote: Hi, is there any way to minimize the need for database packing? I wouldn't mind loosing the ability of undoing transactions. There's a significant performance benefit to keeping some non-current data to support multi-version concurrency control. There Berkely Database Storage supported automatic incremental packing without garbage collection. If someone were to revitalize that effort and if one was willing to do without cyclic garbage collection, then that storage would remove the need for the sort of disruptive pack we have with FileStorage now. Note that I'm working on a new FileStorage packer that is 2-3 times faster and, I believe, much less disruptive than the current packing algorithm. It also provides an option to skip garbage collection, making it twice again as fast. It is available now as a separate monkey-patch in the zc.FileStorage package. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
