Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
Phillip J. Eby wrote: > Congratulations, you're the first person (that I know of, anyway) to hit a > wall here. However, I think you may be barking up the wrong tree on your > profiling. See below. What a dubious honor. > SkinScript expressions execute using Zope security checking. If you're > using a "WITH QUERY SQLmethod() COMPUTE attributelist" statement, security > checks are applied to both the query expression and to every name > referenced in 'attributelist'. So the longer the attribute list, the > slower the execution. Although, now that I think of it, eval() is not used > if you use a pure attribute name list like "name1,name2,name3", so the > problem probably isn't on the attribute list side. This makes it much less > likely that the security machinery is to blame (unless you have a very slow > user folder which also doesn't cache well, and you haven't given your > SkinScript proxy roles to shortcut the security checks.) Yeah... I just added proxy roles to all the skinscripts and it didn't make that much of a difference (maybe a small one though). > Now, for your SQL query... is it by any chance dynamically generated in a > complex way? I mean, more complex than an sqlvar or two? Keep in mind > that when Zope caches SQL queries it does it by generating the text of the > query and using it as a cache key. That means any DTML in your query will > be executed each time. It'd have to be pretty complex, or else there'd > need to be lots of slow security lookups, to get the kind of poor > performance you're seeing, though. In the sql method in question has 1 dtml-sqlvar in it. Everything else is pre-rendered (there's a method that generates the sql methods). > 1. If you're not using LoginManager as your user folder, and maybe even if > you are, your SkinScripts should have proxy roles. Proxy roles can slash > the overhead of performing security checks, and are usually appropriate for > SkinScripts because SkinScript is "internal" to the object and Zope > security will apply to the results of the SkinScripts as well. Most of my tests have been with the builtin user folder. But as I mentioned above, proxy roles don't help much. > 2. Why are you retrieving 200 dataskins? Is this all in one transaction? > If so, you may have a design issue. Multi-object operations should > generally be implemented as domain-specific methods on the Specialist, so > they can be implementation-specific (and thereby take advantage of speedups > like a ZCatalog index or specialized SQL queries). In other words, you > probably want to iterate directly over a 200-row query result from an > SQLMethod, rather than retrieiving 200 DataSkins. (You're accessing over > 10 times as many DataSkins in one transaction as our most complex ZPatterns > application touches in its most complex type of transaction, which involves > about 4 or 5 Specialists at once!) Pretty much, if you're displaying a > list or report or doing some kind of summary analysis or mass update, it > belongs in an implementation-specific method in the Specialist. Yeah... this is mainly a design flaw. I and others at my company were working on various parts of this project in parallel. I was working on the base level ZPatterns model. They were working on client apps. We missed the communication bus at some point and they got to thinking of Specialists as DB tables. Something that needs to be addressed in the future, but it doesn't help me now. :( > 3. If you *really* need 200 dataskins in one transaction, and the overhead > is *really* in SkinScript, then you can bypass the overhead by writing an > AttributeProvider of your own in Python. I'm guessing, however, that > SkinScript per se is not the source of the slowdown, and you need to look > more closely at the things the SkinScript is calling. I think I'll have the developer just use a SQL method to get the data. This is how it should be done anyways... Thanks for the reply. -- John Eikenberry [[EMAIL PROTECTED]] __ "A society that will trade a little liberty for a little order will deserve neither and lose both." --B. Franklin ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
At 01:57 PM 3/29/01 -0800, John Eikenberry wrote: > >We have a fairly large and complex app framework built on ZPatterns. It >uses MySQL for storage and the standard Specialist/Rack/DataSkin setup with >skinscripts for attributes and triggers. > >We've found that the speed of getItem is a bit slower than we need. For >instance retrieving 200 dataskins takes about 8 seconds on a P2-300. After >profiling and digging around I think I've found the primary bottleneck. Its the >running of eval() on the skinscript's python expression (stored in the Compute >class as _fromex and Triggers as callexpr). Congratulations, you're the first person (that I know of, anyway) to hit a wall here. However, I think you may be barking up the wrong tree on your profiling. See below. >The optimization I've been looking at is changing the code from storing a >string and eval()ing the string to using compile() at save time and exec() when >evaluated. > >Profiling these 2 ways in little test programs seems to indicate about a 2.5x >speedup. Not huge, but combined with better hardware should be enough. > >But I'm curious why this avenue wasn't taken to begin with. Seems like the way >to do it to me. Am I missing something? Yes, as your later message mentions, ZPatterns already does that caching. I think what's more likely to be taking up time is either: 1) Zope security checks on the expressions, or 2) DTML execution of your SQL queries Or perhaps something else altogether. First a few comments on the above, then I'll ask some more probing questions regarding your setup. SkinScript expressions execute using Zope security checking. If you're using a "WITH QUERY SQLmethod() COMPUTE attributelist" statement, security checks are applied to both the query expression and to every name referenced in 'attributelist'. So the longer the attribute list, the slower the execution. Although, now that I think of it, eval() is not used if you use a pure attribute name list like "name1,name2,name3", so the problem probably isn't on the attribute list side. This makes it much less likely that the security machinery is to blame (unless you have a very slow user folder which also doesn't cache well, and you haven't given your SkinScript proxy roles to shortcut the security checks.) Now, for your SQL query... is it by any chance dynamically generated in a complex way? I mean, more complex than an sqlvar or two? Keep in mind that when Zope caches SQL queries it does it by generating the text of the query and using it as a cache key. That means any DTML in your query will be executed each time. It'd have to be pretty complex, or else there'd need to be lots of slow security lookups, to get the kind of poor performance you're seeing, though. Okay. Here are a few things to look at in your app: 1. If you're not using LoginManager as your user folder, and maybe even if you are, your SkinScripts should have proxy roles. Proxy roles can slash the overhead of performing security checks, and are usually appropriate for SkinScripts because SkinScript is "internal" to the object and Zope security will apply to the results of the SkinScripts as well. 2. Why are you retrieving 200 dataskins? Is this all in one transaction? If so, you may have a design issue. Multi-object operations should generally be implemented as domain-specific methods on the Specialist, so they can be implementation-specific (and thereby take advantage of speedups like a ZCatalog index or specialized SQL queries). In other words, you probably want to iterate directly over a 200-row query result from an SQLMethod, rather than retrieiving 200 DataSkins. (You're accessing over 10 times as many DataSkins in one transaction as our most complex ZPatterns application touches in its most complex type of transaction, which involves about 4 or 5 Specialists at once!) Pretty much, if you're displaying a list or report or doing some kind of summary analysis or mass update, it belongs in an implementation-specific method in the Specialist. 3. If you *really* need 200 dataskins in one transaction, and the overhead is *really* in SkinScript, then you can bypass the overhead by writing an AttributeProvider of your own in Python. I'm guessing, however, that SkinScript per se is not the source of the slowdown, and you need to look more closely at the things the SkinScript is calling. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
At 05:58 PM 3/29/01 -0500, Steve Spicklemire wrote:
>
>bring up a point I've been wondering about anyway. Now that Ty and
>Phillip have moved on to TransWarp, who will be maintaining all the
>changes to ZPatterns? SteveA has done a great job of keeping a
>modified version available for folks running 2.3.X, but I've seen no
>motion to move those changes into the "real" ZPatterns. Now if you
>find a great optimization, will it get movedinto ZPatterns too?
The reason I haven't moved Steve Alexander's changes in, is that there's
not an easy way to make them backward compatible with 2.2.x. All of the
production applications Ty and I have are on 2.2.x Zopes, and we'd like to
not break them. Zope 2.3.x is still too fresh for us to use for production
apps - we'd like to see the double-dot releases and hotfixes to slow to
more of a trickle before we upgrade. Once we're ready to upgrade, I'll put
SteveA's changes in place. Meanwhile, any patches submitted which are not
2.3.x-dependent can still be accepted. Unfortunately, SteveA gave me his
patches all in one big gulp and I really don't want to take the time to
split it out.
I really expected 2.3.x to stabilize faster for some reason, although in
retrospect the 2.2.x series took this long to stabilize also. Ah well.
Anyway, when I do that stuff, I'll be putting in the "kickTriggers"
feature, too, but I expect to modify it slightly in name and function, to
something like "forceChangedStatus(attributeName)" or
"forceStatus('CHANGED')".
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
Steve Spicklemire wrote: >Makes sense to me! I'm guessing eval was used since it's a little > simpler not to have to keep track of both the string expression and > the compiled expression.. but that's just a guess. However it does Crap... just noticed that the eval()'d version does compile() the expression and caches it. That shoots that optimization... time for more profiling. > bring up a point I've been wondering about anyway. Now that Ty and > Phillip have moved on to TransWarp, who will be maintaining all the > changes to ZPatterns? SteveA has done a great job of keeping a > modified version available for folks running 2.3.X, but I've seen no > motion to move those changes into the "real" ZPatterns. Now if you > find a great optimization, will it get movedinto ZPatterns too? There > are a number of folks now whove contributed to the 'ZPatterns' project > and have invested significant effort in projects that are based on > ZPatterns, and would like to see it maintained. (me!) I wonder if > there is some way that stewardship for ZPatterns could be either > 'handed off' or 'shared' so that these kinds of things can be kept > up-to-date without delaying the promised TransWarp goodies. > > What do you think? I've been thinking something similar since hearing about TW. If Phillip and Ty no longer want to maintain ZPatterns perhaps we could get a little group to keep it up-to-date and tweak it. I'd be interested and I'm sure my company would pay me to help do this (we use ZPatterns extensively). -- John Eikenberry [[EMAIL PROTECTED]] __ "A society that will trade a little liberty for a little order will deserve neither and lose both." --B. Franklin ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
Hi John, Makes sense to me! I'm guessing eval was used since it's a little simpler not to have to keep track of both the string expression and the compiled expression.. but that's just a guess. However it does bring up a point I've been wondering about anyway. Now that Ty and Phillip have moved on to TransWarp, who will be maintaining all the changes to ZPatterns? SteveA has done a great job of keeping a modified version available for folks running 2.3.X, but I've seen no motion to move those changes into the "real" ZPatterns. Now if you find a great optimization, will it get movedinto ZPatterns too? There are a number of folks now whove contributed to the 'ZPatterns' project and have invested significant effort in projects that are based on ZPatterns, and would like to see it maintained. (me!) I wonder if there is some way that stewardship for ZPatterns could be either 'handed off' or 'shared' so that these kinds of things can be kept up-to-date without delaying the promised TransWarp goodies. What do you think? take care, -steve > "JAE" == John Eikenberry <[EMAIL PROTECTED]> writes: JAE> We have a fairly large and complex app framework built on JAE> ZPatterns. It uses MySQL for storage and the standard JAE> Specialist/Rack/DataSkin setup with skinscripts for JAE> attributes and triggers. JAE> We've found that the speed of getItem is a bit slower than we JAE> need. For instance retrieving 200 dataskins takes about 8 JAE> seconds on a P2-300. After profiling and digging around I JAE> think I've found the primary bottleneck. Its the running of JAE> eval() on the skinscript's python expression (stored in the JAE> Compute class as _fromex and Triggers as callexpr). JAE> Note that this becomes the bottleneck after the SQL Method JAE> gets cached. The query to the DB takes the most time on the JAE> first hit, but after its been cached it takes very little JAE> time. JAE> The optimization I've been looking at is changing the code JAE> from storing a string and eval()ing the string to using JAE> compile() at save time and exec() when evaluated. JAE> Profiling these 2 ways in little test programs seems to JAE> indicate about a 2.5x speedup. Not huge, but combined with JAE> better hardware should be enough. JAE> But I'm curious why this avenue wasn't taken to begin JAE> with. Seems like the way to do it to me. Am I missing JAE> something? JAE> -- JAE> John Eikenberry [[EMAIL PROTECTED]] JAE> __ JAE> "A society that will trade a little liberty for a little JAE> order will deserve neither and lose both." --B. Franklin JAE> ___ Zope-Dev JAE> maillist - [EMAIL PROTECTED] JAE> http://lists.zope.org/mailman/listinfo/zope-dev ** No cross JAE> posts or HTML encoding! ** (Related lists - JAE> http://lists.zope.org/mailman/listinfo/zope-announce JAE> http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
