[Zope-CMF] CMF Collector: Open Issues
The following supporters have open issues assigned to them in this collector (http://www.zope.org/Collectors/CMF). Assigned and Open mhammond - Windows DevelopmentMode penalty in CMFCore.DirectoryView, [Accepted] http://www.zope.org/Collectors/CMF/366 Pending / Deferred Issues - FSPropertiesObject.py cannot handle multiline input for lines, text attributes, [Deferred] http://www.zope.org/Collectors/CMF/271 - Can't invalidate skin items in a RAMCacheManager, [Pending] http://www.zope.org/Collectors/CMF/343 - workflow notify success should be after reindex, [Deferred] http://www.zope.org/Collectors/CMF/389 - Possible bug when using a BTreeFolder Member folder, [Pending] http://www.zope.org/Collectors/CMF/441 - Proxy Roles not Working/Applied to Worflow Transition Scripts, [Pending] http://www.zope.org/Collectors/CMF/449 - safe_html filters some tags which should probably not be filtered, [Pending] http://www.zope.org/Collectors/CMF/452 - purge_old in runAllImportSteps not working, [Pending] http://www.zope.org/Collectors/CMF/455 - Danger from Caching Policy Manager, [Pending] http://www.zope.org/Collectors/CMF/460 - properties setup handler: support for non-ascii strings, [Pending] http://www.zope.org/Collectors/CMF/468 - CMFUid reindexes all fields unnecessarily, [Pending] http://www.zope.org/Collectors/CMF/469 - GenericSetup snapshot redirect does not work, [Pending] http://www.zope.org/Collectors/CMF/470 Pending / Deferred Features - Favorite.py: queries and anchors in remote_url, [Pending] http://www.zope.org/Collectors/CMF/26 - DefaultDublinCore should have Creator property, [Pending] http://www.zope.org/Collectors/CMF/61 - Document.py: universal newlines, [Pending] http://www.zope.org/Collectors/CMF/174 - portal_type is undefined in initialization code, [Pending] http://www.zope.org/Collectors/CMF/248 - CMFTopic Does Not Cache, [Deferred] http://www.zope.org/Collectors/CMF/295 - Wishlist: a flag that tags the selected action., [Pending] http://www.zope.org/Collectors/CMF/301 - CMFDefault should make use of allowCreate(), [Pending] http://www.zope.org/Collectors/CMF/340 - Nested Skins, [Deferred] http://www.zope.org/Collectors/CMF/377 - CatalogVariableProvider code + tests, [Pending] http://www.zope.org/Collectors/CMF/378 - manage_doCustomize() : minor additions, [Pending] http://www.zope.org/Collectors/CMF/382 - CMF needs View-based TypeInformation, [Pending] http://www.zope.org/Collectors/CMF/437 - Marker attributes should be deprecated, [Pending] http://www.zope.org/Collectors/CMF/440 - New getNextEvent Method, [Pending] http://www.zope.org/Collectors/CMF/462 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] CMF Tests: 9 OK
Summary of messages to the cmf-tests list. Period Sat Mar 3 12:00:00 2007 UTC to Sun Mar 4 12:00:00 2007 UTC. There were 9 messages: 9 from CMF Unit Tests. Tests passed OK --- Subject: OK : CMF-1.5 Zope-2.7 Python-2.3.6 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:37:07 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004235.html Subject: OK : CMF-1.5 Zope-2.8 Python-2.3.6 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:38:38 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004236.html Subject: OK : CMF-1.5 Zope-2.9 Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:40:08 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004237.html Subject: OK : CMF-1.6 Zope-2.8 Python-2.3.6 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:41:38 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004238.html Subject: OK : CMF-1.6 Zope-2.9 Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:43:08 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004239.html Subject: OK : CMF-2.0 Zope-2.9 Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:44:38 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004240.html Subject: OK : CMF-2.0 Zope-2.10 Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:46:08 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004241.html Subject: OK : CMF-trunk Zope-2.10 Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:47:38 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004242.html Subject: OK : CMF-trunk Zope-trunk Python-2.4.4 : Linux From: CMF Unit Tests Date: Sat Mar 3 21:49:08 EST 2007 URL: http://mail.zope.org/pipermail/cmf-tests/2007-March/004243.html ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: tools-as-utilities, merging, releasing, etc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1 Mar 2007, at 17:36, yuppie wrote: Jens Vagelpohl wrote: On 28 Feb 2007, at 15:14, Rocky wrote: Now that I have completed the primary functionality for five.localsitemanager it seems to me that the CMF jens_tools_as_utilities branch should be ready for merging into trunk in anticipation of the of the next cmf 2.1 alpha/beta release. It's ready for merging when we have a story for existing sites, meaning a clear migration path. I was going to do some simple testing closer to this weekend (busy until then) and do a simple script that can be run via zopectl run and document it if no one else has a better idea or steps up. My assumption here is that the script needs to do two things for each CMF Site encountered in the ZODB: - - create the new magic component registry - - duplicate tool registration as done by the GS componentregistry step Please correct me if I am wrong. I resolved the first part some days ago: http://svn.zope.org/?view=revrev=72782 But I'm not sure if notify(BeforeTraverseEvent(self, REQUEST)) is in the right place. Maye it should just be called by PortalObjectBase, not by all DynamicType subclasses. BTW: While fixing the tests a month ago I made two quick and dirty changes, adding some XXX comments: - in SkinnableObjectManager http://svn.zope.org/?view=revrev=72251 - in PropertiesTool http://svn.zope.org/?view=revrev=72252 Maybe someone can work on a more sophisticated solution and tests? I have now removed all the additional manual tool wrapping in the code and rely on the new component registry wrapping only. Tests are passing 100%. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFF6rr8RAx5nvEhZLIRAiuWAKCFImduWyycEYRBB9dP7YYLtO+K5gCfW/wP /tG+7QMzP9Vld+1sq0YM4Nk= =G8N9 -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] TypesTool speedup
I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. Since Zope startup is as far as I know the only time the result of _product_packages is change perhaps it would be useful to generate that result on speedup. Or perhaps to make _product_packages cache its output. I certainly do want any monkey patches in Plone for this since it should be fixed in either Zope or CMF. Wichert. - Forwarded message from Alexander Limi [EMAIL PROTECTED] - From: Alexander Limi [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: [Plone-developers] Plone 3 - now 3x faster for logged-in users Date: Sat, 03 Mar 2007 19:37:54 -0800 Message-ID: [EMAIL PROTECTED] Hello from the optimization sprint :) Just thought I'd keep you updated on some of the stuff we've been up to here. The main news for the day is that we found a combination of slowness in Zope 2.10 and CMF using that method in a less-than-optimal way. The result was that rendering the content menu (the one that decides what types you can add where etc) was eating an amazing amount of time. Explained in my usual UI designer hand-wavy way, every time we load a page in logged in mode, it checks the list of Products to figure out what you can add. So, by caching this locally in TypesTool so it doesn't get looked up every time (the list of Products doesn't change with every request, after all), we get a significant speedup. Let's see some numbers of loading the front page of a fresh plone : Before types tool optimization (10 requests): Products/CMFPlone/skins/plone_content/document_view.pt 8.54s lib/python/plone/app/contentmenu/contentmenu.pt6.26s After types tool optimization: Products/CMFPlone/skins/plone_content/document_view.pt 2.55s lib/python/plone/app/contentmenu/contentmenu.pt0.29s That makes every logged in page over 3 times faster. I like it. :) Patch for CMFCore TypesTool included. -- Alexander Limi · http://limi.net ___ Plone-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/plone-developers - End forwarded message - -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: tools-as-utilities, merging, releasing, etc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1 Mar 2007, at 17:36, yuppie wrote: BTW: While fixing the tests a month ago I made two quick and dirty changes, adding some XXX comments: - in SkinnableObjectManager http://svn.zope.org/?view=revrev=72251 - in PropertiesTool http://svn.zope.org/?view=revrev=72252 Maybe someone can work on a more sophisticated solution and tests? My checkins today addressed at least the second change set you list by making the site itself a utility and looking it up that way. The additional wrapping is gone as well. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFF6svQRAx5nvEhZLIRAoWMAJ9H6mELtpy4MnMEF06pMbDB76Yk1ACghibn 26TOJ/zYktPWnmym05eYD7o= =HMKr -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Five's local sitemanager, CMF, etc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26 Feb 2007, at 17:03, Martin Aspeli wrote: To get to the portal root / CMF site, I suggest a pattern that is sometimes used in Zope3: We register the CMF site object as a utility providing ICMFSite (or whatever). Then whichever code that's executed below the portal (and that includes CMF tools) can do getUtility(ICMFSite) to get to the site. +1 - in fact, we already have Products.CMFCore.interfaces.ISiteRoot. I use it all the time. :) This is now completed on the branch. I did not try to locate every single place in the code where the site is looked up, though. I patched a couple specific places pointed out by Yuppie, and the main URLTool.getPortalObject method to use the utility. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFF6szDRAx5nvEhZLIRAihRAJ9HejiRPThS2Tck/nbFnPv3s1jVUQCgmnck L4NMFaWpqLE0zozhuc5oo/g= =6FX+ -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [Zope-dev] TypesTool speedup
Previously Andreas Jung wrote: --On 4. März 2007 13:57:23 +0100 Wichert Akkerman [EMAIL PROTECTED] wrote: I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. Any idea why the code is different between 2.9 and 2.10? Rocky modified it in changeset 67869 to be able to support external methods outside of products. The problem with the new code is that it imports all products to figure out some of their details. So for every factory dispatcher access we now have an import for each product, which means lots of filesystem accesses and python interpreter work being done. They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. I would not be opposed against such a speedup patch as long as it does not raise other problems. However I don't know App.* enough to be evaluate the patch. http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [Zope-dev] TypesTool speedup
Previously Wichert Akkerman wrote: http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher. Obvious and stupid mistake in that patch, http://paste.plone.org/13218 should be better. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [Zope-dev] TypesTool speedup
Mutable default values are evil. I would get rid of that. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [Zope-dev] TypesTool speedup
Looking at the changes rocky made, I don't see why allowing external methods from non-Products should require making the whole process completely dynamic on every call. The old mechanism of storing the product list should be put back in place, along with the addition of non-Product information to these results. Alec ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Five's local sitemanager, CMF, etc
Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26 Feb 2007, at 17:03, Martin Aspeli wrote: To get to the portal root / CMF site, I suggest a pattern that is sometimes used in Zope3: We register the CMF site object as a utility providing ICMFSite (or whatever). Then whichever code that's executed below the portal (and that includes CMF tools) can do getUtility(ICMFSite) to get to the site. +1 - in fact, we already have Products.CMFCore.interfaces.ISiteRoot. I use it all the time. :) This is now completed on the branch. I did not try to locate every single place in the code where the site is looked up, though. I patched a couple specific places pointed out by Yuppie, and the main URLTool.getPortalObject method to use the utility. Fantastic :) Thanks! Martin ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: TypesTool speedup
On Mar 4, 1:49 pm, Alec Mitchell [EMAIL PROTECTED] wrote:
Looking at the changes rocky made, I don't see why allowing external
methods from non-Products should require making the whole process
completely dynamic on every call. The old mechanism of storing the
product list should be put back in place, along with the addition of
non-Product information to these results.
Following this message are two patches I'm ready to apply. One for
Five and one for Zope2. The lookup is still dynamic, but at least it
doesn't open a zodb connection everytime anymore. It more closely
resembles what was trying to be done originally, and that is to look
up factory info from the Products.* package.
I'm prepared to commit this but posting it here for review first.
Plus ... what is the policy for updating the Five svn:external for
Zope ? (ie right now it points at Five 1.5.2 tag)
Regards,
Rocky
Five Patch:
Index: fiveconfigure.py
===
--- fiveconfigure.py(revision 72973)
+++ fiveconfigure.py(working copy)
@@ -218,6 +218,11 @@
if init_func is not None:
newContext = ProductContext(product, app, module_)
init_func(newContext)
+
+registered_packages = getattr(Products,
'_registered_packages', None)
+if registered_packages is None:
+registered_packages = Products._registered_packages = []
+registered_packages.append(module_)
finally:
try:
import transaction
Index: tests/test_registerpackage.py
===
--- tests/test_registerpackage.py (revision 72973)
+++ tests/test_registerpackage.py (working copy)
@@ -65,7 +65,11 @@
'pythonproduct2' in product_listing
True
+Make sure it also shows up in ``Products._registered_packages``.
+ [x.__name__ for x in getattr(Products,
'_registered_packages', [])]
+ ['pythonproduct2']
+
Clean up:
tearDown()
Index: CHANGES.txt
===
--- CHANGES.txt (revision 72973)
+++ CHANGES.txt (working copy)
@@ -2,6 +2,14 @@
Five Changes
+Five 1.5.x (svn/unreleased)
+===
+
+* Added change to registerPackage directive so that it stores the
newly
+ registered packages on the Products package object for faster
reference.
+ This means code that looks this up (ie Zope2's FactoryDispatcher)
no longer
+ has to open a zodb connection each time.
+
Five 1.5.2 (2007-01-10)
===
Zope2 Patch:
Index: lib/python/App/FactoryDispatcher.py
===
--- lib/python/App/FactoryDispatcher.py (revision 72972)
+++ lib/python/App/FactoryDispatcher.py (working copy)
@@ -26,32 +26,15 @@
zope2 packages and those without the Products namespace package.
-old_product_packages = {}
+packages = {}
for x in dir(Products):
m = getattr(Products, x)
if isinstance(m, types.ModuleType):
-old_product_packages[x] = m
+packages[x] = m
+
+for m in getattr(Products, '_registered_packages', []):
+packages[m.__name__] = m
-packages = {}
-app = Zope2.app()
-try:
-products = app.Control_Panel.Products
-
-for product_id in products.objectIds():
-product = products[product_id]
-if hasattr(product, 'package_name'):
-pos = product.package_name.rfind('.')
-if pos -1:
-packages[product_id] =
__import__(product.package_name,
- globals(), {},
-
product.package_name[pos+1:])
-else:
-packages[product_id] =
__import__(product.package_name)
-elif old_product_packages.has_key(product_id):
-packages[product_id] =
old_product_packages[product_id]
-finally:
-app._p_jar.close()
-
return packages
class ProductDispatcher(Acquisition.Implicit):
___
Zope-CMF maillist - Zope-CMF@lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf
See http://collector.zope.org/CMF for bug reports and feature requests
