[Zope-dev] Zope Tests: 7 OK
Summary of messages to the zope-tests list. Period Sat Mar 3 12:00:00 2007 UTC to Sun Mar 4 12:00:00 2007 UTC. There were 7 messages: 7 from Zope Unit Tests. Tests passed OK --- Subject: OK : Zope-2.6 Python-2.1.3 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:03:46 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007377.html Subject: OK : Zope-2.6 Python-2.3.6 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:05:16 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007378.html Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:06:46 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007379.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:08:16 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007380.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:09:46 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007381.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:11:16 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007382.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Sat Mar 3 21:12:47 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-March/007383.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] TypesTool speedup
I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. Since Zope startup is as far as I know the only time the result of _product_packages is change perhaps it would be useful to generate that result on speedup. Or perhaps to make _product_packages cache its output. I certainly do want any monkey patches in Plone for this since it should be fixed in either Zope or CMF. Wichert. - Forwarded message from Alexander Limi [EMAIL PROTECTED] - From: Alexander Limi [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: [Plone-developers] Plone 3 - now 3x faster for logged-in users Date: Sat, 03 Mar 2007 19:37:54 -0800 Message-ID: [EMAIL PROTECTED] Hello from the optimization sprint :) Just thought I'd keep you updated on some of the stuff we've been up to here. The main news for the day is that we found a combination of slowness in Zope 2.10 and CMF using that method in a less-than-optimal way. The result was that rendering the content menu (the one that decides what types you can add where etc) was eating an amazing amount of time. Explained in my usual UI designer hand-wavy way, every time we load a page in logged in mode, it checks the list of Products to figure out what you can add. So, by caching this locally in TypesTool so it doesn't get looked up every time (the list of Products doesn't change with every request, after all), we get a significant speedup. Let's see some numbers of loading the front page of a fresh plone : Before types tool optimization (10 requests): Products/CMFPlone/skins/plone_content/document_view.pt 8.54s lib/python/plone/app/contentmenu/contentmenu.pt6.26s After types tool optimization: Products/CMFPlone/skins/plone_content/document_view.pt 2.55s lib/python/plone/app/contentmenu/contentmenu.pt0.29s That makes every logged in page over 3 times faster. I like it. :) Patch for CMFCore TypesTool included. -- Alexander Limi · http://limi.net ___ Plone-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/plone-developers - End forwarded message - -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
--On 4. März 2007 13:57:23 +0100 Wichert Akkerman [EMAIL PROTECTED] wrote: I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. Any idea why the code is different between 2.9 and 2.10? They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. I would not be opposed against such a speedup patch as long as it does not raise other problems. However I don't know App.* enough to be evaluate the patch. Andreas pgp4nEbEdRjxB.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
Previously Andreas Jung wrote: --On 4. März 2007 13:57:23 +0100 Wichert Akkerman [EMAIL PROTECTED] wrote: I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. Any idea why the code is different between 2.9 and 2.10? Rocky modified it in changeset 67869 to be able to support external methods outside of products. The problem with the new code is that it imports all products to figure out some of their details. So for every factory dispatcher access we now have an import for each product, which means lots of filesystem accesses and python interpreter work being done. They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. I would not be opposed against such a speedup patch as long as it does not raise other problems. However I don't know App.* enough to be evaluate the patch. http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
Previously Wichert Akkerman wrote: http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher. Obvious and stupid mistake in that patch, http://paste.plone.org/13218 should be better. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
Mutable default values are evil. I would get rid of that. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
Previously Sidnei da Silva wrote: Mutable default values are evil. I would get rid of that. I modeled that on how get_module_info worked, so at least the pattern is not new in the Zope2 codebase. It could just as easily be done in a global value as well. I'm more interested in seeing if people agree with the basic approach though. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypesTool speedup
Looking at the changes rocky made, I don't see why allowing external methods from non-Products should require making the whole process completely dynamic on every call. The old mechanism of storing the product list should be put back in place, along with the addition of non-Product information to these results. Alec ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: TypesTool speedup
On Mar 4, 1:49 pm, Alec Mitchell [EMAIL PROTECTED] wrote:
Looking at the changes rocky made, I don't see why allowing external
methods from non-Products should require making the whole process
completely dynamic on every call. The old mechanism of storing the
product list should be put back in place, along with the addition of
non-Product information to these results.
Following this message are two patches I'm ready to apply. One for
Five and one for Zope2. The lookup is still dynamic, but at least it
doesn't open a zodb connection everytime anymore. It more closely
resembles what was trying to be done originally, and that is to look
up factory info from the Products.* package.
I'm prepared to commit this but posting it here for review first.
Plus ... what is the policy for updating the Five svn:external for
Zope ? (ie right now it points at Five 1.5.2 tag)
Regards,
Rocky
Five Patch:
Index: fiveconfigure.py
===
--- fiveconfigure.py(revision 72973)
+++ fiveconfigure.py(working copy)
@@ -218,6 +218,11 @@
if init_func is not None:
newContext = ProductContext(product, app, module_)
init_func(newContext)
+
+registered_packages = getattr(Products,
'_registered_packages', None)
+if registered_packages is None:
+registered_packages = Products._registered_packages = []
+registered_packages.append(module_)
finally:
try:
import transaction
Index: tests/test_registerpackage.py
===
--- tests/test_registerpackage.py (revision 72973)
+++ tests/test_registerpackage.py (working copy)
@@ -65,7 +65,11 @@
'pythonproduct2' in product_listing
True
+Make sure it also shows up in ``Products._registered_packages``.
+ [x.__name__ for x in getattr(Products,
'_registered_packages', [])]
+ ['pythonproduct2']
+
Clean up:
tearDown()
Index: CHANGES.txt
===
--- CHANGES.txt (revision 72973)
+++ CHANGES.txt (working copy)
@@ -2,6 +2,14 @@
Five Changes
+Five 1.5.x (svn/unreleased)
+===
+
+* Added change to registerPackage directive so that it stores the
newly
+ registered packages on the Products package object for faster
reference.
+ This means code that looks this up (ie Zope2's FactoryDispatcher)
no longer
+ has to open a zodb connection each time.
+
Five 1.5.2 (2007-01-10)
===
Zope2 Patch:
Index: lib/python/App/FactoryDispatcher.py
===
--- lib/python/App/FactoryDispatcher.py (revision 72972)
+++ lib/python/App/FactoryDispatcher.py (working copy)
@@ -26,32 +26,15 @@
zope2 packages and those without the Products namespace package.
-old_product_packages = {}
+packages = {}
for x in dir(Products):
m = getattr(Products, x)
if isinstance(m, types.ModuleType):
-old_product_packages[x] = m
+packages[x] = m
+
+for m in getattr(Products, '_registered_packages', []):
+packages[m.__name__] = m
-packages = {}
-app = Zope2.app()
-try:
-products = app.Control_Panel.Products
-
-for product_id in products.objectIds():
-product = products[product_id]
-if hasattr(product, 'package_name'):
-pos = product.package_name.rfind('.')
-if pos -1:
-packages[product_id] =
__import__(product.package_name,
- globals(), {},
-
product.package_name[pos+1:])
-else:
-packages[product_id] =
__import__(product.package_name)
-elif old_product_packages.has_key(product_id):
-packages[product_id] =
old_product_packages[product_id]
-finally:
-app._p_jar.close()
-
return packages
class ProductDispatcher(Acquisition.Implicit):
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
