Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer ...snip... What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. ...snip... I actually need this whithin the next couple of weeks - only one caveat: I'd have to do a serious read-up on sessioning, which I don't understand at all and the PluggableAuthService product, which I never heard of until I read about it. We have something similar to SQL-Sessions. However, our security machinery is built on top of Zopes own (in parallell not replacing it). It is loosely modelled around a mix of Zope's way of doing things with a hint of XUF (we couldn't use XUF at the time, because it wasn't stable enough back then). One of the goals wiht the current rewrite of one of our apps is to go back doing things the Zope-way, by using Zope's own security mechanisms. We have working solutions for authentication, but the sessioning part remains - especically since we, like in Steve's setup expire sessions on the server rather than on the browser end. So... any hints on where to begin reading for the completely-lost newbie? /dario -- -- --- Dario Lopez-Ksten, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Conflict errors on BDBMinimal storage
Dario Lopez-Ksten wrote: Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer ...snip... What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. ...snip... I actually need this whithin the next couple of weeks - only one caveat: I'd have to do a serious read-up on sessioning, which I don't understand at all and the PluggableAuthService product, which I never heard of until I read about it. It wouldn't necessarily require PAS, merely the PluginRegistry which was released with it. The reason for this requirement is that a SQLSessionNG product needs to have a set of plugins, which will typically be customized on a per-site basis to fit whatever backend RDBMS schema is in use. Perhaps we should call the product PluggableSessions, as there isn't any particular requirement that the backend be SQL based. We have something similar to SQL-Sessions. However, our security machinery is built on top of Zopes own (in parallell not replacing it). It is loosely modelled around a mix of Zope's way of doing things with a hint of XUF (we couldn't use XUF at the time, because it wasn't stable enough back then). One of the goals wiht the current rewrite of one of our apps is to go back doing things the Zope-way, by using Zope's own security mechanisms. We have working solutions for authentication, but the sessioning part remains - especically since we, like in Steve's setup expire sessions on the server rather than on the browser end. PAS does have a SessionAuthHelper which stashes credentials in the session; you could probably wrap up your current authentication solution as an IAuthenticationPlugin plugin for it. So... any hints on where to begin reading for the completely-lost newbie? Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Conflict errors on BDBMinimal storage
Chris McDonough wrote: Hi Dr. Wagner, thanks for the hint with minimal storage. however the same problem occurs with filestorage and is even worse. we know very well, that we cannot avoid the appearance of conflict errors in particular with zeo, and we understand what causes them. what wonders us is, that we find some contributions about this errors in the mailing lists, but no one seems to have a real problem with it. but we have! we are not worried about some errors appearing in a log, but we are using plone2 witch currently 30 concurrent authenticated users, so that this error occurs permanently, thus decreasing system performance due to several retries and moreover leading to very frequent nasty 'POS.Exception Errors' in the browser, whenever four successive tries fail (almost every 100. request). this is not acceptable, and i wonder wheter there are any plone sites with more than a handful of authenticated users that experience the same problem. we found that one (there are more) product responsible for it is the placeless translation service, which heavily writes to the session. Are you using ZEO for your session storage or are you using a local storage? (You hint at it above but you don't actually say one way or the other.) so we desperately need a solution for this problem, maybe uncouple the session of the locking mechanism or whatever, and this should be a problem that should be addressed by the core team. because this is a strong reason (and so far the only one) to avoid using zope/plone in a large user environment. I also need reliable sessioning for my own customers, so I am working towards fixing the issues I know about. But the issues I know about may not be the same issues as those that you want fixed. You can ensure your issues are addressed by devoting resources towards the problem, be that time+expertise or money or both. I recently reworked Anthony Baxter's SQLSessions product for a consulting customer, essentially modernizing the code and making it compliant with the new Sessions interface (I dropped the bits which do the work currently done by BrowserIDManager). It took me about six hours or so, including writing unit tests. Unfortunately, I don't have access to that code, due to the nature of the consulting arrangement. What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
On Tue, 2004-05-18 at 14:15, Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer, essentially modernizing the code and making it compliant with the new Sessions interface (I dropped the bits which do the work currently done by BrowserIDManager). It took me about six hours or so, including writing unit tests. Unfortunately, I don't have access to that code, due to the nature of the consulting arrangement. What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. I would also like to see this happen. I don't have the time to help implement this right now, but I may be able to work it in to an ongoing engagement at some point in the next few months. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
Chris McDonough wrote: On Tue, 2004-05-18 at 14:15, Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer, essentially modernizing the code and making it compliant with the new Sessions interface (I dropped the bits which do the work currently done by BrowserIDManager). It took me about six hours or so, including writing unit tests. Unfortunately, I don't have access to that code, due to the nature of the consulting arrangement. What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. I would also like to see this happen. I don't have the time to help implement this right now, but I may be able to work it in to an ongoing engagement at some point in the next few months. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) I do not know whether I understand this email correctly. So mabe I am off topic. We would like to use thre PluggableAuthService to replace a userfolder that authenticates against a an email server (calling a web service to do so) and then caches authentification and other user info in a session. So we would like to dig into what both of you seem not to have time to do. How can we get at PluggableAuthService? I did not find it anywhere. And we would of course gladly accept Tres offer for advise. Robert ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
On Tue, 2004-05-18 at 14:58, Chris McDonough wrote: On Tue, 2004-05-18 at 14:15, Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer, essentially modernizing the code and making it compliant with the new Sessions interface (I dropped the bits which do the work currently done by BrowserIDManager). It took me about six hours or so, including writing unit tests. Unfortunately, I don't have access to that code, due to the nature of the consulting arrangement. What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. I would also like to see this happen. I don't have the time to help implement this right now, but I may be able to work it in to an ongoing engagement at some point in the next few months. In my testing, I'm able to see that naive sessioning applications (ones that hit the session on every request, whether they really need to or not) seem to fail down due to the abort after 3 conflicts in a row hardcoded policy under extremely high load (~ 40-60 req/sec or so). This will happen regardless of what code I put in the sessioning machinery; attempting to prevent it there is a lost cause: it gets a bit like trying to prove the Heisenberg Uncertainty Principal or trying to predict the future. As a result, I am coming to believe that along with the errors as part of main transaction patch for 2.7.1, I should also make the retry-on-conflict-error policy pluggable for those who really desperately need to slow their Zope systems to a conflict-induced crawl. They can bump it up to 100 as far as I care and if it works for them, great. The more selective people can wait around for a different (non-ZODB-based) sessioning implementation or change their code to not pound the snot out of the sessioning machinery unnecessarily. As far as I'm concerned, the new Transience stuff is looking really good for the common case. I haven't been able to reproduce any of the corruption issues that happened with the older implementation. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
robert rottermann wrote: On Tue, 2004-05-18 at 14:15, Tres Seaver wrote: I recently reworked Anthony Baxter's SQLSessions product for a consulting customer, essentially modernizing the code and making it compliant with the new Sessions interface (I dropped the bits which do the work currently done by BrowserIDManager). It took me about six hours or so, including writing unit tests. Unfortunately, I don't have access to that code, due to the nature of the consulting arrangement. What I would like to do is revisit the effort, using the new PluginRegistry code released as part of the PluggableAuthService release; I am, however, pretty well booked up for the next few weeks. I would be glad to advise anybody who felt called to take on such an effort, however. I do not know whether I understand this email correctly. So mabe I am off topic. We would like to use thre PluggableAuthService to replace a userfolder that authenticates against a an email server (calling a web service to do so) and then caches authentification and other user info in a session. So we would like to dig into what both of you seem not to have time to do. How can we get at PluggableAuthService? I did not find it anywhere. The PluggableAuthService product is available in CVS: http://cvs.zope.org/Products/PluggableAuthService It depends on the PluginRegistry product: http://cvs.zope.org/Products/PluginRegistry Both are releasable, but not yet released (I just reminded Zac to cut the release tarballs). PAS has a SessionAuthHelper plugin which should serve as a model for the part of your problem which revolves around the session; you would need to write an authenticator plugin which validated credentials against your web service. And we would of course gladly accept Tres offer for advise. I was referring to using the PluginRegistry, which is a lower-level utility, to (re)write the SQLSessions product, which needs to be pluggable in order to accomodate different backends. The PIR is a Zope 2 analog to the Zope 3 local utility service: it stores a perstent map of the plugins registered / activated for a set of application-defined plugin interfaces. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
On 19/05/2004, at 8:36 AM, Chris McDonough wrote: As a result, I am coming to believe that along with the errors as part of main transaction patch for 2.7.1, I should also make the retry-on-conflict-error policy pluggable for those who really desperately need to slow their Zope systems to a conflict-induced crawl. They can bump it up to 100 as far as I care and if it works for them, great. Or this could be used to avoid retries altogether. And at the very least it would advertise the fact that zope does retry requests. Michael. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
On Tue, 2004-05-18 at 16:36, Chris McDonough wrote: In my testing, I'm able to see that naive sessioning applications (ones that hit the session on every request, whether they really need to or not) seem to fail down due to the abort after 3 conflicts in a row hardcoded policy under extremely high load (~ 40-60 req/sec or so). This will happen regardless of what code I put in the sessioning machinery; attempting to prevent it there is a lost cause: it gets a bit like trying to prove the Heisenberg Uncertainty Principal or trying to predict the future. Just as a followup: when I observed this, it was because I had all of my extra debugging code turned on. True to the Uncertainty Principle, the execution of the debugging code caused a raft of extra conflicts. Turning off the debugging code leads to a much rosier scenario. In cases where it was raising conflicts to the end user, it now operates without doing so. I still need to do some really hardcore testing of the thing to see exactly where it buckles, but I can no longer provoke it trivially. As a result, I am coming to believe that along with the errors as part of main transaction patch for 2.7.1, I should also make the retry-on-conflict-error policy pluggable for those who really desperately need to slow their Zope systems to a conflict-induced crawl. They can bump it up to 100 as far as I care and if it works for them, great. The more selective people can wait around for a different (non-ZODB-based) sessioning implementation or change their code to not pound the snot out of the sessioning machinery unnecessarily. Michael Dunstan also had another good idea. At least one knob exists in the source code to increase/decrease the resolution of a part of the sessioning/transience code. Basically this knob allows you to choose a tradeoff between the frequency of a class of writes done by the session code and the accuracy of the session finalization. Setting the knob higher will decrease the aggregate number of database writes that would need to be done over any given period of time in order to keep sessions current, which might lead to fewer conflicts. On the flip side, it would mean that the session might not be finalized on time (sessions are finalized when they are garbage collected; sessions are only allowed to be garbage collected every n seconds where n is the knob's setting). This would typically be meaningful only to people who use the Script to call at session deletion time feature of sessions; that script might be called much later than you think it might be called if you set this value high enough. There's currently no guarantee that it will be called exactly when a session expires even with the default value, but setting the resolution higher will decrease that probability even further. Since that feature isn't used very much (at least I don't think it is), I think this is an ok tradeoff for a lot of people. I'd need to make sure that it actually helps something before doing so, but assuming it did, it would be put into the config file for 2.7.1 as something like session-resolution-seconds or somesuch. For those of you with conflict problems now, this knob is the PERIOD variable in Transience.py... you might be able to get away with setting it higher now to see if that helps anything, especially those people trying to use sessions+ZEO. It's preset to 20, which means 20 seconds. Try 120 instead, or something around there. Just don't set it to a number higher than the timeout value for the TOC (which is in minutes, remember to multiply it by 60) or all sorts of insanity will happen. You'll need to restart Zope and recreate your transient object container after doing this, as doing this will effectively invalidate all data previously in the container and using old data left over from running it under the default PERIOD value might make it go a little insane too. As far as I'm concerned, the new Transience stuff is looking really good for the common case. I haven't been able to reproduce any of the corruption issues that happened with the older implementation. This is still true (it's been almost an hour since I posted it, I think that's a new record!) - c ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
