Re: [Zope-dev] Session concurrency problem
On Wednesday 30 January 2008, Dieter Maurer wrote: 2. Sometimes, the session package data does not have the data in it and fails with a key error. This happens maybe 1 in 50 times and I have not reliably reproduced this problem. What is session package data? A very simple object: class SessionPkgData(persistent.Persistent, IterableUserDict): See zope.session.interfaces.ISessionData session = SessionPkgData() ISessionPkgData.providedBy(session) True zope.interface.implements(ISessionPkgData) def __init__(self): self.data = OOBTree() (zope.session.session, line 517) Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session concurrency problem
Stephan Richter wrote at 2008-1-30 08:45 -0500: ... 1. The data's __dict__ returns an empty dictionary. When you access an attribute directly, though, the data can be seen. This symptom can be reliably reproduced. This is normal for persistent objects. When a persistent object is in Ghost state, its __dict__ appears to be empty (it is empty). When you access a non special attribute, the object is unghosted and then __dict__ is filled in. 2. Sometimes, the session package data does not have the data in it and fails with a key error. This happens maybe 1 in 50 times and I have not reliably reproduced this problem. What is session package data? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
I have never seen disappearing sessions -- but I must admit that we use our own session implementation - deriving from Zope's session but getting rid of the over-complex session management code. I've not explained very well. Session object doesn't disappears, just some data in session object does it. (they occur non-deterministically, they are extremely difficult to reproduce and analyse). As said: The advertised behaviour when several requests modify the same session object is that all but the first committing request get a ConflictError and are retried. Everything else is a bug. I'm agree that, if it works fine, the retrieing request method is a good solution. My problem is quasi deterministic, it crashes most of times :-( I will try to provide a example.zexp when entering the bug so it could be easier to reproduce. Another note: it is known (though we do not yet fully understand why) that sessions behave non-deterministically (disappear, revert to old values) when they are read/written in standard_error_message (Zope's error handling hook for failing requests). I'm not using SESSION in standard_error_message, but symptoms are very similar. Santi Camps ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
It is unlikely that the the low conflict connection causes your problem. Low conflict affects the following situation: You try to load an object (because it is not in your ZODB cache) and get informed that is has been modified (by another commited transaction) since your transaction started. With a normal ZODB connection, you will get a ReadConflictError as adding the object to your ZODB cache would make it inconsistent: some objects would have the state from your transactions start and others from a later date. With a low conflict connection, the inconsistency is accepted and no ReadConflictError is raised. As you described, you loose the session object. This is not easily understood as the effect of a cache inconsistency. Sure. Disabling Low conflict hasn't solved anything. I'm not able to find where is the problem. I have not enought knowledge of Zope internals. I will try to store my data in persisten subobjects, as you said. One thing is true: Changing session object simultaniously in various frames doesn't work fine. Sometimes session data is not the espected becouse another frame has erased your changes in the session object. I think this is a serious inconvenient in front of other platforms. From my point of view, it would be better to have SESSION out of the transaction but with changes stored inmediatly. You don't? Perhaps some day I will be able to implement this for myself. Until that, do you think I should enter a bug, or this cann't be considered as a bug? Thanks for your help Santi Camps ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
I'm not able to find where is the problem. I have not enought knowledge of Zope internals. I will try to store my data in persisten subobjects, as you said. Finally I've found a better and easy partial workarround. I will explain it here so other users in similar situation can know. It consist in to have multiple Transitient Object Containers and multiple Session Data Managers. For instance, in my case: Folder1/some_scripts Folder1/session_data (a Transitient Object Container) Folder1/session_data_manager (a Session Data Manager using previous object and referenced in REQUEST as SESSION_FOLDER1) Folder2/some_scripts Folder2/session_data (a Transitient Object Container) Folder2/session_data_manager (a Session Data Manager using previous object and referenced in REQUEST as SESSION_FOLDER2) Then, scripts in Folder1 can use SESSION_FOLDER1 to store its session data, scripts in Folder2 can use SESSION_FOLDER2 to do the same. To use SESSSION_FOLDER1 from scripts in Folder2, a simple traversal will work fine. Using this solution, my simultanious frames are working using session with no problems. Cheers Santi Camps http://zetadb.sourceforge.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
Santi Camps wrote at 2003-11-27 17:41 +0100: ... One thing is true: Changing session object simultaniously in various frames doesn't work fine. Sometimes session data is not the espected becouse another frame has erased your changes in the session object. I think this is a serious inconvenient in front of other platforms. This is a general problem of concurrently updating the same object. Zope's Session mechanism is not that bad in this respect (it is in some way better than that for other plattforms). When it works as advertised (I see it working this way; you do not), then concurrent modification will result in a ConflictError with automatic request retrial. This provides for consistency despite concurrent updates. A *BIG* problem is the large granularity of a Zope session. It is essentially one big monolitic object (with respect to concurrent update). This makes ConflictError much more likely. I have never seen disappearing sessions -- but I must admit that we use our own session implementation - deriving from Zope's session but getting rid of the over-complex session management code. This code caused many inconsistency errors for the session management data structures; which is a bad sign. The complexity comes from the fact that it tries to scale well for large numbers of sessions. Our simplified code scales not that well but can easily cope with several thousand (locally maintained) sessions we sometimes see. From my point of view, it would be better to have SESSION out of the transaction but with changes stored inmediatly. You don't? No: you would face the general inconsistency problem caused by concurrent updates of an object. Analysing and fixing such problems is a nightmare (they occur non-deterministically, they are extremely difficult to reproduce and analyse). Perhaps some day I will be able to implement this for myself. No need. There is a product called ZSession. It maintains session data in RAM and (I think) allows threads to read/write sessions asynchronously. It is probably no longer maintained, now that Zope comes with Sessions in the core distribution. But that need not be a big problem. Should it break in some future Zope version (which may be possible), the fixes should be quite easy and straight forward. Until that, do you think I should enter a bug, or this cann't be considered as a bug? When you see session objects disappearing (and they have not expired), then this is a bug! As said: The advertised behaviour when several requests modify the same session object is that all but the first committing request get a ConflictError and are retried. Everything else is a bug. Another note: it is known (though we do not yet fully understand why) that sessions behave non-deterministically (disappear, revert to old values) when they are read/written in standard_error_message (Zope's error handling hook for failing requests). standard_error_message is called after the transaction has been aborted -- thus any (transactional) effect from the request has already been wiped out. Accessing Session there may read old values or create a new (empty) session. Apparently, and this is what we do not yet understand, such phony sessions seem to be able to spill over into later requests. One hypothesis was that it were caused by Zope forgetting to abort/commit the transaction implicitly created for standard_error_message execution (this definitely is a bug!). But the transaction *is* aborted later, before it is being used in another request -- maybe this happens then in the wrong context? I will not see problems caused by this failing to commit/abort error transaction bug because I fixed our Zope as soon as the bug became known. I am not sure whether the bug is fixed in Zope 2.6.2. That's a disadvantage of maintaining our own Zope version - I do not know and can not easily check the distributed code any longer. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
Santi Camps wrote at 2003-11-24 23:14 +0100: I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data It should *NOT* loose its session data but get a ConflictError and automatically retry the request. That's normal ZODB behaviour. Search for Application specific conflict resolution for a (complex, in your case) way to work around this. A partial workaround may be to put your various keys into persistent subobjects. I'm trying to understand how Conflict Resolution works. The first step will be change my DirectoryStorage for a FileStorage and try again. As you say, I seems that Zope retries the request 3 times when a ConflictError is raised, so it should work in my case. If not, I will try to debug and see how not, or perhaps disabling low conflict flag. If, at the end, it countinues crashing, I will store my data in persistent subobjects. Thanks a lot Santi Camps ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
[EMAIL PROTECTED] wrote at 2003-11-26 11:54 +0100: ... I'm trying to understand how Conflict Resolution works. The first step will be change my DirectoryStorage for a FileStorage and try again. This will have no influence to your session problems. Sessions are (by default) put into a RAM based storage independent from your main storage (DirectoryStorage or FileStorage). As you say, I seems that Zope retries the request 3 times when a ConflictError is raised, so it should work in my case. Indeed. If not, I will try to debug and see how not, or perhaps disabling low conflict flag. It is unlikely that the the low conflict connection causes your problem. Low conflict affects the following situation: You try to load an object (because it is not in your ZODB cache) and get informed that is has been modified (by another commited transaction) since your transaction started. With a normal ZODB connection, you will get a ReadConflictError as adding the object to your ZODB cache would make it inconsistent: some objects would have the state from your transactions start and others from a later date. With a low conflict connection, the inconsistency is accepted and no ReadConflictError is raised. As you described, you loose the session object. This is not easily understood as the effect of a cache inconsistency. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session concurrency
Santi Camps wrote at 2003-11-24 23:14 +0100: I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data It should *NOT* loose its session data but get a ConflictError and automatically retry the request. That's normal ZODB behaviour. Search for Application specific conflict resolution for a (complex, in your case) way to work around this. A partial workaround may be to put your various keys into persistent subobjects. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
