[Zope-CMF] Re: Making TypesTool faster
On Fri, 04 May 2007 05:17:36 -0700, Tres Seaver [EMAIL PROTECTED] wrote: I think I misunderstood: you guys have actually measured that the 'manage_addProducts[product]' dance is taking measurable time when building the add list? I thought we were trying to cache the result of the whole lookup, especially the meta_type-is-allowed + user-has-permission checks, which I would imagine are much more expensive than the factory dispatcher lookup. Actually, it seems to be — but I'm not a programmer — I just know that when we cached the result of that lookup locally, the whole thing is about twice as fast overall. That's including our metatype-is-allowed and has-permission checks. That said, I didn't really understand the patch as earlier proposed; I'm sorry for not reading more carefully. Did you have time to look at it now? If the two are orthogonal, and we already have a sane patch, then why not (at least if we can get some tests in place) fix the one at the CMF level and deal with the other Archetypes issue separately. If factory dispatcher lookup is truly the hotspot, then we should be fixing it even lower in the stack (at the Zope level). It's just my experience that it's harder to get small optimizations into Zope for these kinds of things, as it's more likely that people rely on side effects like this (the fact that the list of products can change without restarting) there. But if you think it can be done on that level, it would of course be the optimal location to do it. -- Alexander Limi · http://limi.net ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: Making TypesTool faster
Am 03.05.2007 um 00:50 schrieb Alexander Limi: At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. I'm sorry but it's -1 from me. The change is unnecessary and the problem seems to be with Archetypes and not the CMF. In my view it is incorrect to change the infrastucture to compensate for the deficiencies of a component. If you need it in Plone you can add it as a patch. Charlie -- Charlie Clark Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-938-5360 GSM: +49-178-782-6226 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: Making TypesTool faster
Charlie Clark-3 wrote: Am 03.05.2007 um 00:50 schrieb Alexander Limi: At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. I'm sorry but it's -1 from me. The change is unnecessary and the problem seems to be with Archetypes and not the CMF. In my view it is incorrect to change the infrastucture to compensate for the deficiencies of a component. If you need it in Plone you can add it as a patch. I'm not entirely convinced the problem is in AT, and I think we are conflating two different issues here. One is an observation of Tres' that AT has a function that could perform more cacheing. That's a good point, and may mitigate some of the performance issues by cacheing at a higher level, but it doesn't mean there aren't further opportunities for improvements to the underlying code. The other issue is some code in TypesTool which is making a lot of calls to a procedure in Zope which is unlikely (impossible?) to ever return a different result until Zope is restarted and new products are detected. If the two are orthogonal, and we already have a sane patch, then why not (at least if we can get some tests in place) fix the one at the CMF level and deal with the other Archetypes issue separately. Martin -- View this message in context: http://www.nabble.com/Making-TypesTool-faster-tf3678114.html#a10321411 Sent from the Zope - CMF list2 mailing list archive at Nabble.com. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin Aspeli wrote: Charlie Clark-3 wrote: Am 03.05.2007 um 00:50 schrieb Alexander Limi: At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. I'm sorry but it's -1 from me. The change is unnecessary and the problem seems to be with Archetypes and not the CMF. In my view it is incorrect to change the infrastucture to compensate for the deficiencies of a component. If you need it in Plone you can add it as a patch. I'm not entirely convinced the problem is in AT, and I think we are conflating two different issues here. One is an observation of Tres' that AT has a function that could perform more cacheing. That's a good point, and may mitigate some of the performance issues by cacheing at a higher level, but it doesn't mean there aren't further opportunities for improvements to the underlying code. The other issue is some code in TypesTool which is making a lot of calls to a procedure in Zope which is unlikely (impossible?) to ever return a different result until Zope is restarted and new products are detected. I think I misunderstood: you guys have actually measured that the 'manage_addProducts[product]' dance is taking measurable time when building the add list? I thought we were trying to cache the result of the whole lookup, especially the meta_type-is-allowed + user-has-permission checks, which I would imagine are much more expensive than the factory dispatcher lookup. That said, I didn't really understand the patch as earlier proposed; I'm sorry for not reading more carefully. If the two are orthogonal, and we already have a sane patch, then why not (at least if we can get some tests in place) fix the one at the CMF level and deal with the other Archetypes issue separately. If factory dispatcher lookup is truly the hotspot, then we should be fixing it even lower in the stack (at the Zope level). Trse. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGOyRg+gerLs4ltQ4RAi2cAKCq13gmBFz9MNfJNlmZTjiGgudnzgCgnWqC NTjFLPoZw0xL2OSbIBrpztQ= =1zFZ -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: Making TypesTool faster
Am 04.05.2007 um 14:00 schrieb Martin Aspeli: I'm not entirely convinced the problem is in AT, and I think we are conflating two different issues here. One is an observation of Tres' that AT has a function that could perform more cacheing. That's a good point, and may mitigate some of the performance issues by cacheing at a higher level, but it doesn't mean there aren't further opportunities for improvements to the underlying code. Indeed it doesn't but the argument is that there is a performance problem in Plone rather than CMF. Tres' most telling argument for me is: The cache is actually based only on the container (which could be a path) and the user ID. It would be a perfect use of a RAM Cache Manager (assuming that we cached mappings rather than persistent references). Using a volatile here (which is what I assume you mean) would dilute the locality of the cache by the number of database connections in use. The other issue is some code in TypesTool which is making a lot of calls to a procedure in Zope which is unlikely (impossible?) to ever return a different result until Zope is restarted and new products are detected. No, the issue is how often the TypesTool is being called (from within Plone). Surely the cache should be higher upstream in this case? Or maybe no, see below. If the two are orthogonal, and we already have a sane patch, then why not (at least if we can get some tests in place) fix the one at the CMF level and deal with the other Archetypes issue separately. The Archetypes issue should indeed be dealt with separately. I am, however, not convinced that this patch is sane. If the calls to Zope are indeed the issue as you suggest then I would suggest looking at their mechanism and whether Zope rather than the CMF can be improved in this respect - going on what Alec is quoted as saying it still makes sense to me to put the cache in the Zope making it available to all components. I've only had a brief look at the Product registration, etc. in Zope but from what I saw it looked like it could indeed be improved. Charlie -- Charlie Clark Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-938-5360 GSM: +49-178-782-6226 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexander Limi wrote: On Wed, 02 May 2007 10:03:39 -0700, Martin Aspeli [EMAIL PROTECTED] wrote: +1, but I don't think the two need to be mutually exclusive. Amen. At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. I do have a notion which would make this better, faster, stronger for Plone, which is much more afflicted by this problem than other CMF-based sites (because it presents an add menu on each page): I think the bottleneck is actually inside 'ATContentTypes.lib.contraintypes', in the 'getDefaultAddableTypes' method of the 'ConstrainTypesMixin' class: it could cache the typeinfo list (or a list of mappings derived from it), with the container and the user_id as keys. Most of the win would come immediatelely, and at the place most affected by the clurrent slowness, leaving us time to figure out why Z3-style factories don't work with AT. I know why... or rather, I did know, and I'm sure if I looked it again I could remember. It has to do with the extra work that's being done in the auto-generated factory (see ClassGen.py) which calls initializeArchetype(). I'd love to get rid of this (for various reasons), but I tried to rewrite it as a factory and failed (this is where I don't remember why). Given that Alec co essentially had a patch which worked, I suggest that we have a look at it, and include it if possible, and then encourage the use factories in general. I'd hope we could avoid deprecating it outright until we can fix up AT to use them, though. Also note that according to Alec, it would work better as a instance-level thing than a thread-level one. The cache is actually based only on the container (which could be a path) and the user ID. It would be a perfect use of a RAM Cache Manager (assuming that we cached mappings rather than persistent references). Using a volatile here (which is what I assume you mean) would dilute the locality of the cache by the number of database connections in use. I've looksd, and all the callers of 'getDefaultAddableTypes' need only the ID and / or the title of the FTI, which simplifies things a bunch. Isn't this something like 2 lines of code in TypesTool? :) I don't think so. Getting the tests right is likely to be *lots* of lines of code in 'tests/test_TypesTool.py', if we make the change there. I feel like we're having a very abstract discussions here. Have you looked at the patch Alec/Ben/Mike/Rob did? If not, Limi - can you find it and paste it here so that we can discuss its merits? I don't recall whether it came with tests, but even so it didn't look terribly complicated. Martin ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
On Thu, 03 May 2007 00:33:05 -0700, Martin Aspeli [EMAIL PROTECTED] wrote: I feel like we're having a very abstract discussions here. Have you looked at the patch Alec/Ben/Mike/Rob did? If not, Limi - can you find it and paste it here so that we can discuss its merits? I don't recall whether it came with tests, but even so it didn't look terribly complicated. I included it in my initial post, but here it is again: http://paste.plone.org/13211 -- Alexander Limi · http://limi.net ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
Chris Withers schrieb: Alexander Limi wrote: This issue goes back to the original discovery that Zope had a bug that made its Product lookup mechanism What do you mean by Product lookup mechanism? The way in which '_get/_queryFactoryMethod' (called by 'isContructionAllowed' for each and every type on every page) are implemented - I think. Raphael cheers, Chris ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexander Limi wrote: Hi all, I would like some input on making CMF (and by extension Plone) more efficient. This issue goes back to the original discovery that Zope had a bug that made its Product lookup mechanism be very slow in Zope 2.10.0/1, and affected TypesTool lookups in particular. When we experimented with this during the Performance Sprint at Google, we lowered the call time from 6.26s to 0.29s by caching it in a thread local cache. (All calls and timings are 10 calls to the standard Plone front page on a 2GHz MacBook) The contentmenu.pt template in Plone (which pretty much just does the TypesTool lookup for what types are addable), currently takes up roughly 30% (!) of the total cost of rendering a standard page in Plone. After the bug fix was applied to Zope 2.10.3, the call went down to 0.80s with the same hardware/#calls — there's still a significant potential for improvement here: 0.80s - 0.29s. Here's what Alec Mitchell wrote about it at the time: The fundamental issue is not that product lookup is slow in Zope, but that CMF uses the Product lookup mechanism as a registry without any sort of caching. This is quite easy to fix either in Plone or in the CMF depending on the timeframe we need it merged in for. I initially used a thread local for safety because I wasn't entirely sure what sorts of things would end up in this cache. However, by now it's pretty clear to me that an instance level cache should be perfectly safe, which should be a little faster yet. So what I'm wondering is whether it's possible to get this instance-level caching of the lookup in the types tool before CMF 2.1 final? I'd obviously prefer it if Plone didn't have to subclass and override parts of TypesTool to fix this. :) (Oh, and for reference — the original thread local cache that we used for testing is here: http://paste.plone.org/13211 ) First, thanks for investigating. I know Plone developers have suspected that the add list (which is rendered on every page view in a typical Plone site) was a major source of sluggishness; I'm glad to get it confirmed with specifics. Before we look at optimizing the use of the product dispatch mechanism, I'd like to propose deprecating it in favor of the factory-utility mechanism: leaving the 'product' field blank in an FTI, and having the 'factory' field be the name of a utility registered for 'zope.component.interfaces.IFactory'. Typically, the code to enable this looks like the following (the CMF already does this):: # Products/MyProduct/MyContent.py from zope.component.factory import Factory ... MyContentFactory = Factory(MyContent) # right after InitializeClass # Products/MyProduct/configure.zcml ... utility component=.MyContent.MyContentFactory name=myproduct.mycontent / ... # Products/MyProduct/profiles/myproduct/types/MyContent.xml ... property name=product/property property name=factorymyproduct.mycontent/property ... If y'all still have your benchmarks available, could you publish the setup code you used? I'd really like to see the results on a site set up using the new-style factories. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFGOL1N+gerLs4ltQ4RAmkhAJdpZCdNGSH8K/LKl9R3eRQEES58AJ0XH/wF FUwezLhNvQdLkmjeXtdo9A== =PXSC -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
On Wed, 02 May 2007 10:03:39 -0700, Martin Aspeli [EMAIL PROTECTED] wrote: +1, but I don't think the two need to be mutually exclusive. Amen. At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. Given that Alec co essentially had a patch which worked, I suggest that we have a look at it, and include it if possible, and then encourage the use factories in general. I'd hope we could avoid deprecating it outright until we can fix up AT to use them, though. Also note that according to Alec, it would work better as a instance-level thing than a thread-level one. Isn't this something like 2 lines of code in TypesTool? :) -- Alexander Limi · http://limi.net ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: Making TypesTool faster
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexander Limi wrote: On Wed, 02 May 2007 10:03:39 -0700, Martin Aspeli [EMAIL PROTECTED] wrote: +1, but I don't think the two need to be mutually exclusive. Amen. At present, Archetypes-based content types cannot be used with a factory (I tried hard, but there are some acquisition-related/factory related reasons); I'd like to refactor this, but we can't for Plone 3.0 at least. Right. Obviously, we'd like to do this the proper way, but we can't do that for a while yet. I do have a notion which would make this better, faster, stronger for Plone, which is much more afflicted by this problem than other CMF-based sites (because it presents an add menu on each page): I think the bottleneck is actually inside 'ATContentTypes.lib.contraintypes', in the 'getDefaultAddableTypes' method of the 'ConstrainTypesMixin' class: it could cache the typeinfo list (or a list of mappings derived from it), with the container and the user_id as keys. Most of the win would come immediatelely, and at the place most affected by the clurrent slowness, leaving us time to figure out why Z3-style factories don't work with AT. Given that Alec co essentially had a patch which worked, I suggest that we have a look at it, and include it if possible, and then encourage the use factories in general. I'd hope we could avoid deprecating it outright until we can fix up AT to use them, though. Also note that according to Alec, it would work better as a instance-level thing than a thread-level one. The cache is actually based only on the container (which could be a path) and the user ID. It would be a perfect use of a RAM Cache Manager (assuming that we cached mappings rather than persistent references). Using a volatile here (which is what I assume you mean) would dilute the locality of the cache by the number of database connections in use. I've looksd, and all the callers of 'getDefaultAddableTypes' need only the ID and / or the title of the FTI, which simplifies things a bunch. Isn't this something like 2 lines of code in TypesTool? :) I don't think so. Getting the tests right is likely to be *lots* of lines of code in 'tests/test_TypesTool.py', if we make the change there. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGOUKa+gerLs4ltQ4RAoVvAJ9KeQJ6yBYVfvZnG9QMh5hdJ/Q6pACfZLbd +yihEEAEXfTMV4pYyA8ixbI= =kVBV -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
