Re: [ZODB-Dev] Re: ZODB Benchmarks
Hi, Am Freitag, den 02.11.2007, 09:56 -0400 schrieb David Binger: On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote: On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote: That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. But wouldn't then all other threads get a conflict? If they are trying to do insertions at the same time as the consolidation, yes. This data structure won't stop insertion conflicts, the intent is to make them less frequent. Hmm. Wouldn't a queue be a good data structure to do that? IIRC ZC already wrote a queue that doesn't conflict: http://svn.zope.de/zope.org/zc.queue/trunk/src/zc/queue/queue.txt If you store key/value pairs in the queue, you can do a step-by-step migration from the queue to the btree. Probably this should be encapsulated into a new data structure that looks btree-like and has an additional `consolidate` method. Calling the `consolidate` method would have to happen from the application that uses this data structure. Two issues I can think of immediately: - General: We need an efficient way to find all data structures that need reconciliation, maybe a ZODB-wide index of all objects that require reconciliation would be nice. - With Zope and ZEO: which Zope server is responsible for actually performing the reconciliation? One of the Zope servers that is marked in the zope.conf? Or maybe the ZEO server? Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote: On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote: That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. But wouldn't then all other threads get a conflict? If they are trying to do insertions at the same time as the consolidation, yes. This data structure won't stop insertion conflicts, the intent is to make them less frequent. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZODB Benchmarks
Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay, perhaps the trigger should be the depth of the small-bucket tree. That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. -Matt ClockServer sections can now be specified in zope.conf. If you specify them with a period of say 10 mins (or even 2) then the queue should never get too large, and the linear search time is not a problem as n is small. Essentially you end up with a solution very similar to QueueCatalog but with the queue being searchable. The pain is then in modifying all of the indexes to search the queue in addition to their standard data structures. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Fri, 2007-11-02 at 16:00 +, Laurence Rowe wrote: Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay, perhaps the trigger should be the depth of the small-bucket tree. That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. -Matt ClockServer sections can now be specified in zope.conf. If you specify them with a period of say 10 mins (or even 2) then the queue should never get too large, and the linear search time is not a problem as n is small. Essentially you end up with a solution very similar to QueueCatalog but with the queue being searchable. The pain is then in modifying all of the indexes to search the queue in addition to their standard data structures. I don't think that you need to modify the indexes at all. You simply pass the query arguments to the queue in the exact same way that you apply the arguments to individual indexes. I think with a little enhancement to QueueCatalog one should be able to pull this off. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay, perhaps the trigger should be the depth of the small-bucket tree. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 5:48 AM, Lennart Regebro wrote: On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote: An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. Would it be possible to not occasionally consolidate, but actually do it ongoing, but just one process, thereby always inserting just one transaction into the normal BTree at a time? Or does that cause troubles? I think that option would work. I think it would suffice to do a Big.update(Small); Small.clear() operation before a nightly pack. It might invalidate every bucket in every cache, but BTrees are designed to perform reasonably without a cache. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote: An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/branch factor. Would it be possible to not occasionally consolidate, but actually do it ongoing, but just one process, thereby always inserting just one transaction into the normal BTree at a time? Or does that cause troubles? //In way over my head. -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZODB Benchmarks
David Binger dbinger at mems-exchange.org writes: On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay, perhaps the trigger should be the depth of the small-bucket tree. That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. -Matt ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
This is the 'batch' or 'distribute' pattern that crops up in many fields. The best path is normally to understand what the conflicts are, and where the time is spent. If in, this case, much time is spent in the preamble, and the actual inserts are quick, then diving down one time through the security layers and stuffing in 10 items is clearly better than 10 preambles, one for each insert. The other truism is that all optimisation is for a single case. There may be different answers for different cases. Ideally a single parameter would be enough to tune the system for different cases. Good luck Roche, with the outcome, I'm excited to see some figures. --r. On 2 Nov 2007, at 15:24, David Binger wrote: On Nov 2, 2007, at 10:58 AM, Lennart Regebro wrote: It seems to me having one thread doing a background consolidation one transaction at a time seems a better way to go, Maybe, but maybe that just causes big buckets to get invalidated in all of the clients over and over again, when we could accomplish the same objective in one invalidation by waiting longer and executing a bigger consolidation. although certainly the best thing would be to test all kinds of solutions and see. No doubt about that. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday [EMAIL PROTECTED] office: +44 118 3217026 mobile: +44 7789 338868 skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
