Re: [Zope] conflicts, sessions, and programming best practices guidelines
On 11/18/05, Chris McDonough [EMAIL PROTECTED] wrote: On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote: Thank Chris, good pointsB. The session variable timeout is currently several hours as requird by our application. The timeout resolution remains at its default value. Making the resolution higher has been observed to reduce conflicts (see dunny's chart). If you sessions live for several hours then perhaps you want a increase session-resolution-seconds to something like 1800 (30 minutes) or more. Also worth seeing if increasing WRITEGRANULARITY of Products.Transience.TransientObject helps you. I have not dorked with disabling the inband housekeeping. I don't see how it would impact on our problem. Am I missing something? Turning off inband housekeeping has the potential to reduce the numnber of conflicts because less work is done during normal sessioning operations. Instead of doing the work to figure out whether it needs to clean up after itself during normal operations it relies on an external process. And it is not that difficult to do. Michael. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Dennis Allison wrote at 2005-11-15 14:54 -0800: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? * Localize out into separate persistent objects attributes with high write frequency. E.g. when you have a counter, put into its own persistent object (you can use a BTrees.Length.Length object for a counter). * Implement conflict resolution for your high frequently written persistent objects. Formerly, TemporaryStorage had only very limited history information to support conflict resolution (which limited the wholesome effect of conflict resolution). Rumours say that this improved with Zope 2.8. * Write only when you really change something. E.g. instead of session[XXX] = sss use if session[XXX] != sss: session[XXX] = sss (at least, if there is a high chance that session already contains the correct value). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Thanks Dieter. On Fri, 18 Nov 2005, Dieter Maurer wrote: Dennis Allison wrote at 2005-11-15 14:54 -0800: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? * Localize out into separate persistent objects attributes with high write frequency. E.g. when you have a counter, put into its own persistent object (you can use a BTrees.Length.Length object for a counter). * Implement conflict resolution for your high frequently written persistent objects. Formerly, TemporaryStorage had only very limited history information to support conflict resolution (which limited the wholesome effect of conflict resolution). Rumours say that this improved with Zope 2.8. * Write only when you really change something. E.g. instead of session[XXX] = sss use if session[XXX] != sss: session[XXX] = sss (at least, if there is a high chance that session already contains the correct value). -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Thank Chris, good pointsB. The session variable timeout is currently several hours as requird by our application. The timeout resolution remains at its default value. I have not dorked with disabling the inband housekeeping. I don't see how it would impact on our problem. Am I missing something? I suppose I could do a custom p_resolveConflict since the vast majority of conflicts are are non-conflicting and actual conflicts are fairly easy to identify. Still, on the whole, I think we have to refactor our programs to minimize the opportunities for conflicts as well as tuning the mechanism. As a temporary workaround, I'm backing out of the shared temporary storage in favor of a local temp storage. That should make things a bit better. On Wed, 16 Nov 2005, Chris McDonough wrote: Changing the architecture will likely get you the most bang but note also that there are a few knobs that you can turn on the transient object container UI that may limit the number of conflicts (in the temp_folder/session_data_container): - Data object timeout value -- the timeout value in minutes.. make this higher. - Timeout resolution -- the timeout resolution in seconds.. make this higher. You can also turn off inband housekeeping of session data by calling the method disableInbandHousekeeping on a transient object container. At that point, sessions will continue to work properly but old session data won't be garbage collected and you will need to do this every so often by calling the housekeep method on TOC. See this written by dunny about conflicts and sessions (although this was written before MVCC): http://www.plope.com/Members/dunny/conflicts/view HTH, - C On Nov 16, 2005, at 6:18 PM, Dennis Allison wrote: Chris, I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly. Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture. On Tue, 15 Nov 2005, Chris McDonough wrote: On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. Nothing in general except the (probably too general) attempt not to allow one process/thread to write to the same object at the same time as another process/thread. One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a normal Zope storage. This is because: - the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist. - you have n processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete with itself (between all of its threads). I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound). - C -- -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote: Thank Chris, good pointsB. The session variable timeout is currently several hours as requird by our application. The timeout resolution remains at its default value. Making the resolution higher has been observed to reduce conflicts (see dunny's chart). I have not dorked with disabling the inband housekeeping. I don't see how it would impact on our problem. Am I missing something? Turning off inband housekeeping has the potential to reduce the numnber of conflicts because less work is done during normal sessioning operations. Instead of doing the work to figure out whether it needs to clean up after itself during normal operations it relies on an external process. I suppose I could do a custom p_resolveConflict since the vast majority of conflicts are are non-conflicting and actual conflicts are fairly easy to identify. I couldn't know one way or another, but if you believe so... Still, on the whole, I think we have to refactor our programs to minimize the opportunities for conflicts as well as tuning the mechanism. As a temporary workaround, I'm backing out of the shared temporary storage in favor of a local temp storage. That should make things a bit better. Good idea... - C On Wed, 16 Nov 2005, Chris McDonough wrote: Changing the architecture will likely get you the most bang but note also that there are a few knobs that you can turn on the transient object container UI that may limit the number of conflicts (in the temp_folder/session_data_container): - Data object timeout value -- the timeout value in minutes.. make this higher. - Timeout resolution -- the timeout resolution in seconds.. make this higher. You can also turn off inband housekeeping of session data by calling the method disableInbandHousekeeping on a transient object container. At that point, sessions will continue to work properly but old session data won't be garbage collected and you will need to do this every so often by calling the housekeep method on TOC. See this written by dunny about conflicts and sessions (although this was written before MVCC): http://www.plope.com/Members/dunny/conflicts/view HTH, - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Try to partition your collision space. Do you mount a shared temp_folder? Don't! Do as Chris said and keep a temp_folder per ZEO client. Let the load balancer send people to the same ZEO client every time (according to the cookie). Note that there is no solution for bad coding practice other than to fix it ;-). Keep session access to an absolute minimum (read *and* write). You may also want to look at SQLSessions, which seem to be able to take a higher load (haven't tried for a while). But only up to a point until you get concurrency issues in the RDBMS. Psycopg in particular defers conflicts to Zope by again raising ConflictErrors... In the end, the key to scaling is to keep state out of the app server. http://www.loudthinking.com/arc/000479.html http://www.danga.com/words/2004_oscon/oscon2004.pdf Stefan On Tue, 15 Nov 2005, Dennis Allison wrote: Peter, That's pretty much what we've done, but it is not really enough in our case. Our sites are interactive with lots of per user state. We make fairly heavy use of session variables to track the state. Our number of simultaneous users is also fairly high. In addition, during development, we structured things for ease of development rather than for mimimum liklihood of conflict errors. At this point we are seeing many conflicts and interactions with both the sessioning mechanism and the persistence mechanism--or so it appears. Even when we resolve the current database interaction problem, a refactoring seems in order to ensure the number of conflicts is kept to a minimum. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Chris, I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly. Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture. On Tue, 15 Nov 2005, Chris McDonough wrote: On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. Nothing in general except the (probably too general) attempt not to allow one process/thread to write to the same object at the same time as another process/thread. One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a normal Zope storage. This is because: - the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist. - you have n processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete with itself (between all of its threads). I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound). - C -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
On Nov 16, 2005, at 6:18 PM, Dennis Allison wrote: Chris, I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly. Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture. Sure. The alternative solutions are: - session affinity load balancing solutions which always send the user to the same appserver instance across some number of requests based on a cookie value or the remote IP address. - session data manager implementations that don't use ZODB. Tres Seaver coded up a drop-in SDM implementation that used MySQL although I'm not sure that it was able to be released OSS at the time. A quick Google search doesn't turn up anything, so it's hard to tell. - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Changing the architecture will likely get you the most bang but note also that there are a few knobs that you can turn on the transient object container UI that may limit the number of conflicts (in the temp_folder/session_data_container): - Data object timeout value -- the timeout value in minutes.. make this higher. - Timeout resolution -- the timeout resolution in seconds.. make this higher. You can also turn off inband housekeeping of session data by calling the method disableInbandHousekeeping on a transient object container. At that point, sessions will continue to work properly but old session data won't be garbage collected and you will need to do this every so often by calling the housekeep method on TOC. See this written by dunny about conflicts and sessions (although this was written before MVCC): http://www.plope.com/Members/dunny/conflicts/view HTH, - C On Nov 16, 2005, at 6:18 PM, Dennis Allison wrote: Chris, I am aware that using ZEO to back session database is likely to increase the opportunity for conflicts, but using a single session database seems to be reaquired if you want, as we do, to distribute out interactive application acrosss a cluster of processors cleanly. Under our current approach, we have multiple Zopes running on multiple machines, all referencing a shared database of session variables. This allows a HTTP request to be blindly distributed to any of the machines. Given the conflict problems, we may need to rethink this simplistic architecture. On Tue, 15 Nov 2005, Chris McDonough wrote: On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. Nothing in general except the (probably too general) attempt not to allow one process/thread to write to the same object at the same time as another process/thread. One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a normal Zope storage. This is because: - the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist. - you have n processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete with itself (between all of its threads). I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound). - C -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
I don't know of any documentation but (correct me if I'm wrong) but - install zope 2.8.0 or something higher that uses zodb 3.4 - use zeo's and load balancers so that certain requests don't clog your pipes On 11/15/05, Dennis Allison [EMAIL PROTECTED] wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
On Nov 15, 2005, at 5:54 PM, Dennis Allison wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. Nothing in general except the (probably too general) attempt not to allow one process/thread to write to the same object at the same time as another process/thread. One tip, since you've told us you use sessions heavily: using ZEO to back your session database provides more opportunity for conflicts than if you used a normal Zope storage. This is because: - the opportunity for conflict naturally grows as transaction time increases and using a ZEO storage rather than a local storage adds latency to the total transaction time that would not otherwise exist. - you have n processes attempting to write to essentially the same objects at the same time when you use a shared ZEO sessioning database. If these were instead local databases, each process would only compete with itself (between all of its threads). I tend to try to use local sessioning databases backed by tempstorage plus a frontend load balancer that allows for cookie-based session affinity (Pound). - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
Peter, That's pretty much what we've done, but it is not really enough in our case. Our sites are interactive with lots of per user state. We make fairly heavy use of session variables to track the state. Our number of simultaneous users is also fairly high. In addition, during development, we structured things for ease of development rather than for mimimum liklihood of conflict errors. At this point we are seeing many conflicts and interactions with both the sessioning mechanism and the persistence mechanism--or so it appears. Even when we resolve the current database interaction problem, a refactoring seems in order to ensure the number of conflicts is kept to a minimum. On Tue, 15 Nov 2005, Peter Bengtsson wrote: I don't know of any documentation but (correct me if I'm wrong) but - install zope 2.8.0 or something higher that uses zodb 3.4 - use zeo's and load balancers so that certain requests don't clog your pipes On 11/15/05, Dennis Allison [EMAIL PROTECTED] wrote: Has anyone prepared a set of best practice guidelines on the techniques to use to minimize conflicts? It is becoming clear that we need to do to refactor some of our code to get us out of our current conflict pickle. A quick google produced lots of commentary but nothing I could hand to our programmers as guidelines. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
