Re: [Zope-dev] Conflict errors on BDBMinimal storage
Hi Chris, hi folks. thanks for the hint with minimal storage. however the same problem occurs with filestorage and is even worse. we know very well, that we cannot avoid the appearance of conflict errors in particular with zeo, and we understand what causes them. what wonders us is, that we find some contributions about this errors in the mailing lists, but no one seems to have a real problem with it. but we have! we are not worried about some errors appearing in a log, but we are using plone2 witch currently 30 concurrent authenticated users, so that this error occurs permanently, thus decreasing system performance due to several retries and moreover leading to very frequent nasty 'POS.Exception Errors' in the browser, whenever four successive tries fail (almost every 100. request). this is not acceptable, and i wonder wheter there are any plone sites with more than a handful of authenticated users that experience the same problem. we found that one (there are more) product responsible for it is the placeless translation service, which heavily writes to the session. so we desperately need a solution for this problem, maybe uncouple the session of the locking mechanism or whatever, and this should be a problem that should be addressed by the core team. because this is a strong reason (and so far the only one) to avoid using zope/plone in a large user environment. i hope some of the community sees the importance of this. Regards Thomas On Thu, 2004-05-13 at 05:23, Gerhard Schmidt wrote: Hi, whe are running portal unter zope2.7 with one zeo server and six frontend application servers. I have set up zeo to server two storages filestorage main path $INSTANCE_HOME/var/Data.fs /filestorage minimalstorage temp envdir $INSTANCE_HOME/var/session /minimalstorage temp should provide an shared session. But when is set up the frontends to use temp as /temp_folder, we see many ZODB conflict errors. The performance decreases and PosErrors appear quite often. BDBMininalStorage is no longer maintained and doesn't do any conflict resolution. Even though you'll need to pack frequently, you might want to try a FileStorage instead. Regardless, using ZEO will presumably always generate more conflict errors than using a local ZODB because the commit time is longer, and thus there's a better chance that two threads will be committing at the same time. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On 18/05/2004, at 6:57 AM, Gerhard Schmidt wrote: On Mon, May 17, 2004 at 01:38:15PM -0400, Chris McDonough wrote: AFAIK Zope is the only appserver that even makes an attempt at allowing session data to be shared between servers like this. Most other appservers require you to set up session affinity in your load balancer and have a session database on each appserver. and nealy every system failed to do so. How do you implement this session affinity. Based on source IP. At the moment every large ISP uses multiple Proxyserver so every request comes from a different IP so the affinity doesn´t work for them. The Proxyserver provided by our Computing Center is also split on at least 8 Hosts, so everyone using this proxy would lose session infomation, getting very strange effekts. Perhaps you can get some relief by using pound as your load balancer. It is quite capable at sniffing out user sessions including watching the Cookie header. See http://www.apsis.ch/pound/ Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
Hi Dr. Wagner, thanks for the hint with minimal storage. however the same problem occurs with filestorage and is even worse. we know very well, that we cannot avoid the appearance of conflict errors in particular with zeo, and we understand what causes them. what wonders us is, that we find some contributions about this errors in the mailing lists, but no one seems to have a real problem with it. but we have! we are not worried about some errors appearing in a log, but we are using plone2 witch currently 30 concurrent authenticated users, so that this error occurs permanently, thus decreasing system performance due to several retries and moreover leading to very frequent nasty 'POS.Exception Errors' in the browser, whenever four successive tries fail (almost every 100. request). this is not acceptable, and i wonder wheter there are any plone sites with more than a handful of authenticated users that experience the same problem. we found that one (there are more) product responsible for it is the placeless translation service, which heavily writes to the session. Are you using ZEO for your session storage or are you using a local storage? (You hint at it above but you don't actually say one way or the other.) so we desperately need a solution for this problem, maybe uncouple the session of the locking mechanism or whatever, and this should be a problem that should be addressed by the core team. because this is a strong reason (and so far the only one) to avoid using zope/plone in a large user environment. I also need reliable sessioning for my own customers, so I am working towards fixing the issues I know about. But the issues I know about may not be the same issues as those that you want fixed. You can ensure your issues are addressed by devoting resources towards the problem, be that time+expertise or money or both. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
Gerhard Schmidt wrote at 2004-5-17 21:01 +0200: ... However, due to a ZEO weakness, its conflict resolution currently does not work (ZEO no longer supports INSTANCE_HOME). Is that a bug or intetional. I expect it to be a bug. I plan to fix it soon -- but currently, I have lots of things to do. May take some months... -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On Mon, May 17, 2004 at 12:45:16PM -0400, Chris McDonough wrote: On Thu, 2004-05-13 at 05:23, Gerhard Schmidt wrote: Hi, whe are running portal unter zope2.7 with one zeo server and six frontend application servers. I have set up zeo to server two storages filestorage main path $INSTANCE_HOME/var/Data.fs /filestorage minimalstorage temp envdir $INSTANCE_HOME/var/session /minimalstorage temp should provide an shared session. But when is set up the frontends to use temp as /temp_folder, we see many ZODB conflict errors. The performance decreases and PosErrors appear quite often. BDBMininalStorage is no longer maintained and doesn't do any conflict resolution. Why this. I was told some time ago we should use BDBMinimalStorage to fix the conflict problems. Even though you'll need to pack frequently, you might want to try a FileStorage instead. There is no difference between BDB Storage and Filestorage regarding the conflicts. Regardless, using ZEO will presumably always generate more conflict errors than using a local ZODB because the commit time is longer, and thus there's a better chance that two threads will be committing at the same time. The bottom line is that there is no way of getting an shared session running. Why isn´t there an implementation of temporary storage for ZEO I dont care about conflit resolution. Its only session data. I simply need a shared session storage. But as Far as i can see the is no way of doing this. Are we the only site out there running Plone2 (Placeless transaltion Service) at that scale. How do others handle there Session Bye Estartu -- Gerhard Schmidt| Nick : estartu IRC : Estartu | Fischbachweg 3 || PGP Public Key 86856 Hiltenfingen | [EMAIL PROTECTED] | auf Anfrage/ Tel: 08232 77 36 4 ||on request Fax: 08232 77 36 3 || pgpqXUCWUVwl0.pgp Description: PGP signature ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On Mon, 2004-05-17 at 13:06, Gerhard Schmidt wrote: BDBMininalStorage is no longer maintained and doesn't do any conflict resolution. Why this. I was told some time ago we should use BDBMinimalStorage to fix the conflict problems. When was that and who told you so? Even though you'll need to pack frequently, you might want to try a FileStorage instead. There is no difference between BDB Storage and Filestorage regarding the conflicts. OK. Regardless, using ZEO will presumably always generate more conflict errors than using a local ZODB because the commit time is longer, and thus there's a better chance that two threads will be committing at the same time. The bottom line is that there is no way of getting an shared session running. It works doesn't it? You just get conflict errors, right? Conflict errors are normal at a certain level. Why isnt there an implementation of temporary storage for ZEO There is, just use the tempstorage package from Zope. I dont care about conflit resolution. Its only session data. For better or worse, because it is based on ZODB, the implementation makes you need to care. I simply need a shared session storage. But as Far as i can see the is no way of doing this. I don't understand. You seem to have it working. Is performance the issue? Are we the only site out there running Plone2 (Placeless transaltion Service) at that scale. How do others handle there Session AFAIK Zope is the only appserver that even makes an attempt at allowing session data to be shared between servers like this. Most other appservers require you to set up session affinity in your load balancer and have a session database on each appserver. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
Gerhard Schmidt wrote at 2004-5-13 11:23 +0200: ... Is there any why to get a shared session working. You know that the ZODB is not designed for high frequency concurrent writes? When you want to reduce conflicts you will need to try hard to minimize concurrent writes. I have a (non-open) Transience implementation that writes much less then Zope's standard implementation. However, due to a ZEO weakness, its conflict resolution currently does not work (ZEO no longer supports INSTANCE_HOME). -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On Mon, May 17, 2004 at 01:38:15PM -0400, Chris McDonough wrote: On Mon, 2004-05-17 at 13:06, Gerhard Schmidt wrote: BDBMininalStorage is no longer maintained and doesn't do any conflict resolution. Why this. I was told some time ago we should use BDBMinimalStorage to fix the conflict problems. When was that and who told you so? Even though you'll need to pack frequently, you might want to try a FileStorage instead. There is no difference between BDB Storage and Filestorage regarding the conflicts. OK. Regardless, using ZEO will presumably always generate more conflict errors than using a local ZODB because the commit time is longer, and thus there's a better chance that two threads will be committing at the same time. The bottom line is that there is no way of getting an shared session running. It works doesn't it? You just get conflict errors, right? Conflict errors are normal at a certain level. No It doesn`t. At the Moment we run with single sessions on each frontend we have rewitten most ob the code using sessions to store ther information in cockies. But that´s not realy a good solution. Why isn´t there an implementation of temporary storage for ZEO There is, just use the tempstorage package from Zope. I didn´t find a way to use Tempstorage in zeo. temporarystorage name temporary storage for sessioning /temporarystorage didn´t work. How do i Set up a temporarystorage in zeo. I dont care about conflit resolution. Its only session data. For better or worse, because it is based on ZODB, the implementation makes you need to care. Than maybe the sessioncode should be redesigned not to use ZODB as storage but store ther information in some different system maybe a RDMB or some other system able to handle the kind of usage. I simply need a shared session storage. But as Far as i can see the is no way of doing this. I don't understand. You seem to have it working. Is performance the issue? The Performance decrase is there but the Problem is far deeper. If an read conflict isn´t resolved after three retries an Exception is raised an the request is aborted. And thats the real problem. Are we the only site out there running Plone2 (Placeless transaltion Service) at that scale. How do others handle there Session AFAIK Zope is the only appserver that even makes an attempt at allowing session data to be shared between servers like this. Most other appservers require you to set up session affinity in your load balancer and have a session database on each appserver. and nealy every system failed to do so. How do you implement this session affinity. Based on source IP. At the moment every large ISP uses multiple Proxyserver so every request comes from a different IP so the affinity doesn´t work for them. The Proxyserver provided by our Computing Center is also split on at least 8 Hosts, so everyone using this proxy would lose session infomation, getting very strange effekts. Bye Estartu -- Gerhard Schmidt| Nick : estartu IRC : Estartu | Fischbachweg 3 || PGP Public Key 86856 Hiltenfingen | [EMAIL PROTECTED] | auf Anfrage/ Tel: 08232 77 36 4 ||on request Fax: 08232 77 36 3 || pgpsBoo3UooYb.pgp Description: PGP signature ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On Mon, 2004-05-17 at 14:57, Gerhard Schmidt wrote: Why isnt there an implementation of temporary storage for ZEO There is, just use the tempstorage package from Zope. I didnt find a way to use Tempstorage in zeo. temporarystorage name temporary storage for sessioning /temporarystorage didnt work. How do i Set up a temporarystorage in zeo. Good question. ;-) It involves changing the ZEO configuration machinery to recognize the tempstorage package. The steps to do this I couldn't tell you without actually doing it, however. I dont care about conflit resolution. Its only session data. For better or worse, because it is based on ZODB, the implementation makes you need to care. Than maybe the sessioncode should be redesigned not to use ZODB as storage but store ther information in some different system maybe a RDMB or some other system able to handle the kind of usage. Zope doesn't ship with a useful and maintained relational database. You can implement your own SQL-based TransientObjectContainer using the interfaces defined in Transience/Interfaces.py if you wish, or use Anthony Baxter's SQLSession product. It would be nice if there was an alternate implementation that used a relational database like this. I simply need a shared session storage. But as Far as i can see the is no way of doing this. Of course there is, it's just not built in. ;-) I don't understand. You seem to have it working. Is performance the issue? The Performance decrase is there but the Problem is far deeper. If an read conflict isnt resolved after three retries an Exception is raised an the request is aborted. And thats the real problem. I see. I'm not sure switching to TemporaryStorage will help much. I would first try using a FileStorage and make sure to put the Zope lib/python directory on the ZEO server's Python's PYTHONPATH (to allow for conflict resolution under ZEO, ZEO must be able to import the Products package and subpackages therein). I don't know if this will help very much, but it's worth a shot. Are we the only site out there running Plone2 (Placeless transaltion Service) at that scale. How do others handle there Session AFAIK Zope is the only appserver that even makes an attempt at allowing session data to be shared between servers like this. Most other appservers require you to set up session affinity in your load balancer and have a session database on each appserver. and nealy every system failed to do so. How do you implement this session affinity. Based on source IP. At the moment every large ISP uses multiple Proxyserver so every request comes from a different IP so the affinity doesnt work for them. The Proxyserver provided by our Computing Center is also split on at least 8 Hosts, so everyone using this proxy would lose session infomation, getting very strange effekts. Yes. It's a hard problem. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
