[ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Hello, Somehow relevant to the subject I just found an article on Wickert's site: http://www.wiggy.net/ , Using a seperate Data.fs for the catalog -- Best regards, Groszer Adam -- Quote of the day: Keep your fears for yourself, but share your courage with others. - Robert Louis Stevenson ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob test failure on FreeBSD
Ok, it was just a 64-bit int/longint issue. I've fixed the test, the buildbot is passing again. Christian Am Mittwoch, den 28.03.2007, 01:06 +0200 schrieb Christian Theune: Ok, I'll try to figure out whether this happens on 64bit linux tomorrow. After that I can try 32bit Freebsd, otherwise I have to guess. Christian Am Dienstag, den 27.03.2007, 14:16 -0400 schrieb Benji York: Christian Theune wrote: Am Dienstag, den 27.03.2007, 12:44 -0400 schrieb Benji York: Christian Theune wrote: Who's the owner of the machine? You might know him. A guy by the name of Thomas Lotze. :) Nope. That's the buildbot that has been offline for ages. ;) The failing one is 'tmiddleton'. :) Indeed! I read the wrong column. I guess I need to ping Thomas about restarting his slave, or just remove it. To answer your original question: that machine is 64-bit. tmiddleton is Tim Middleton ([EMAIL PROTECTED]). I've copied him on this message. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] two level cache
Hi, I certainly do not know much about ZODB internals, therefore I do not know if this idea make much sense. But it has been bothering me for some time. Problem: Current ZODB design creates a separate cache for each ZODB connection (ie. a thread in zope). It means that the same object could be replicated in each connection cache. We cannot do much about it since we do not know in advance that a particular object will not be modified. But it is a kind of waste when a number of modified objects is relatively low to a number of read-only objects. Two level cache: A current ZODB cache will become a first level cache (L1), each ZODB connection maintains its own copy. When an object is needed it reads it from L2 cache instead from a storage. L2 cache is a common for all ZODB connections. Objects are stored there in the state where references are not yet resolved from their persistent form. We can still have the same object replicated in respective L1 caches, now even one more copy is in L2. But a large L2 and a smaller L1 could possibly lead to a more efficient caching strategy than it is now. Regards, Atmasamarpan Novy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: two level cache
Jens Vagelpohl wrote: Use ZEO and you get your single L2 cache, file-based, for free... jens I was referring to a memory cache. Secondly, there is no possible performance gain (only loss) one can get by using zeo with its file cache comparing to a direct file storage. ZEO is a great tool, but it will not help us in this sense. Regards, Atmasamarpan Novy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] two level cache
Atmasamarpan Novy wrote: A current ZODB cache will become a first level cache (L1), each ZODB connection maintains its own copy. When an object is needed it reads it from L2 cache instead from a storage. L2 cache is a common for all ZODB connections. Objects are stored there in the state where references are not yet resolved from their persistent form. If you're using FileStorage without ZEO, then your operating system's disk cache will act in a way similar to what you're describing as your L2 cache. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Adam Groszer wrote: Somehow relevant to the subject I just found an article on Wickert's site: http://www.wiggy.net/ , Using a seperate Data.fs for the catalog The win here is actually partitioning the object cache... Similar wins could be achieved without making backup/pack/etc more complicated by making the object cache more intelligent: - based on size of objects, not just number - memory limiting (including intra-transaction) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO and time.sleep
On Mar 28, 2007, at 6:28 PM, Benji York wrote: Last week I spent a very enjoyable day (no kidding) debugging a very, very slow cold-start situation (more than 15 minutes to return from the first request). When making the first request to the app (Zope 3 based), the app server and storage server would show virtually no CPU utilization, and there would be about a megabit of network traffic (on a gigabit link). There was no obvious bottleneck. After liberal application of strace, tcpdump, wireshark (aka ethereal), and the Python profiler we discovered that while waiting for an outstanding request for an object to load, ZEO calls a threading.Connection threading.Condition :) instance's wait method with a timeout. When given a timeout that method enters a wait loop with a time.sleep to sleep for a while and then see if the condition has been met. We found that time.sleep on that box had a minimum granularity of 10ms (when passed a non-zero value), thus causing each object load to take approximately that long. As you can imagine, that somewhat slowed down the retrieval of the several thousand objects required to satisfy the initial request(s) (until the ZEO cache was sufficiently warm). The fix? Short-term: bump the operating system's timer interrupt on that box to 1000Hz from 100Hz, increasing time.sleep's granularity to 1ms (this was on Linux, Window's time.sleep appears have a much higher resolution). Long-term: Jim has found that the timeout call in the wait-for- result code can be avoided, side-stepping the call to time.sleep altogether. What was the time to return the first request after the fix? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO and time.sleep
On Mar 28, 2007, at 6:37 PM, Benji York wrote: Jim Fulton wrote: On Mar 28, 2007, at 6:28 PM, Benji York wrote: threading.Connection threading.Condition :) Yeah, that. :) What was the time to return the first request after the fix? I didn't stick around for the post-Linux-kernel-recompile testing, but when we hacked the Python standard library to use sleep(0) all the time (emulating an infinite granularity timer), it went down to about 10 seconds. A pretty nice improvement. ;) And the original time was 15 minutes! :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
