[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Michael Dunstan wrote: On 18/05/2004, at 10:03 PM, Andrea Patuzzo wrote: Hi, dear developers: We are now randomly getting this kind of error: [...] File /usr/local/Zope270CVS/lib/python/Products/Transience/Transience.py, line 341, in __setitem__ current_bucket = self._data[current_ts] KeyError: 1084872640 This one was with the latest publish.py / startup.py patch and the new Transience.py implementation, on a yesterday's Zope-2_7-branch checkout. New Zope instance with FileStorage does not solve the problem. Same happens with the patches applied to the official Zope-2.7.0 release. Note: once the error happens, zope has to be restarted to function again. Before applying patches we were getting get errors (instead of __setitem__) and we just had to close browser to get back to work. Here we have to restart Zope (with TemporaryStorage) or delete Sessions.fs and restart (with FileStorage). My guess at the moment is that somewhere a ConflictError is being swallowed. There are a few things that can be responsible for this: - Use of hasattr on Persistent objects. - try/except blocks that don't raise ConflictErrors. From your trace that looks like a Plone site. A very quick glance through Plone and its friends indicates that there may be some chance of either of the above causing some problems. Although Plone 2.0.1 includes the following fix: - Fixed bare 'except:' statements in Plone so we don't swallow ZODB ConflictErrors. Mostly by adding 'except ConflictError: raise'. [stefan] Someone please shoot me down and claim that Plone and friends all do the right thing. ;-) We should have a 'hasattr-geddon' and remove every trace of that monstrosity from Zope and the CMF; likewise a 'bareexcept-geddon' (there might be a few places which are smart enough to do 'except:', but I doubt it). Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 23:08, Chris McDonough wrote: There indeed is a minor off-by-one error: it manifests itself as sessions timing out at most 20 seconds early. But there is also a deeper issue which involves the fact that a session data object is not properly removed from an older bucket when it moves due to being accessed in a later timeslice; the symptom only appears when a browser id is reused to start a session after it was used to start an older one that had timed out normally. I've got almost no clue why this happens at this point, but I'm working on it. Ugh. This is almost certainly what Steve is experiencing. I take that back. Actually, I think I was just reading the test results and debug output wrong. It appears to be operating normally except for the off-by-one problem (which is minor). I need to jack up the tests to do some comparisons of data values; currently I'm just testing to ensure that *something* is in the session.. I need to test if the right thing is in the session over time. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi, dear developers:
We are now randomly getting this kind of error:
2004-05-18T11:30:41 ERROR(200) SiteError
http://XXX/prefs_error_log_form
Traceback (most recent call last):
File /usr/local/Zope270CVS/lib/python/ZPublisher/Publish.py, line 100,
in publish
request, bind=1)
File /usr/local/Zope270CVS/lib/python/ZPublisher/mapply.py, line 88, in
mapply
if debug is not None: return debug(object,args,context)
File /usr/local/Zope270CVS/lib/python/ZPublisher/Publish.py, line 40, in
call_object
result=apply(object,args) # Type scr to step into published object.
File /usr/local/Zope270CVS/lib/python/Shared/DC/Scripts/Bindings.py,
line 306, in __call__
return self._bindAndExec(args, kw, None)
File /usr/local/Zope270CVS/lib/python/Shared/DC/Scripts/Bindings.py,
line 343, in _bindAndExec
return self._exec(bound_data, args, kw)
File /zope_instances/patuzzo.ch/Products/CMFCore/FSPageTemplate.py, line
191, in _exec
result = self.pt_render(extra_context=bound_names)
File /zope_instances/patuzzo.ch/Products/CMFCore/FSPageTemplate.py, line
124, in pt_render
result = FSPageTemplate.inheritedAttribute('pt_render')(
File
/usr/local/Zope270CVS/lib/python/Products/PageTemplates/PageTemplate.py,
line 96, in pt_render
tal=not source, strictinsert=0)()
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 189,
in __call__
self.interpret(self.program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 663,
in do_useMacro
self.interpret(macro)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 408,
in do_optTag_tal
self.do_optTag(stuff)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 393,
in do_optTag
return self.no_tag(start, program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 388,
in no_tag
self.interpret(program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 663,
in do_useMacro
self.interpret(macro)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 408,
in do_optTag_tal
self.do_optTag(stuff)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 393,
in do_optTag
return self.no_tag(start, program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 388,
in no_tag
self.interpret(program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 663,
in do_useMacro
self.interpret(macro)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 629,
in do_condition
self.interpret(block)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 406,
in do_optTag_tal
self.no_tag(stuff[-2], stuff[-1])
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 388,
in no_tag
self.interpret(program)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 605,
in do_loop_tal
self.interpret(block)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 233,
in interpret
handlers[opcode](self, args)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 290,
in do_startTag
ok, name, s = attrAction(self, item)
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 369,
in attrAction_tal
translated = self.translate(msgid or value, value, {})
File /usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py, line 615,
in translate
msgid, i18ndict, default=default)
File /usr/local/Zope270CVS/lib/python/Products/PageTemplates/TALES.py,
line 264, in translate
target_language=target_language)
File
/zope_instances/patuzzo.ch/Products/PlacelessTranslationService/PlacelessTr
anslationService.py, line 109, in translate
return service.translate(domain, msgid, mapping, context,
target_language, default)
File
/zope_instances/patuzzo.ch/Products/PlacelessTranslationService/PlacelessTr
anslationService.py, line 423, in translate
catalogs = self.getCatalogsForTranslation(context, domain,
target_language)
File
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Good morning. I just got in and checked on my customer's system. In the past 22 1/2 hours they've had 15000 page hits and last night at about 9:30, ONE person got a KeyError. Actually, this same person got twenty KeyErrors over a period of about 45 seconds. I'm downloading their log files now and plan to spend some time this morning going through them. Anyway, it appears that I was wrong when I said that the problem doesn't show up when I use FileStorage (although it does seem to happen less frequently -- but who can be sure of anything at this point?). In answer to your questions earlier, Chris, we set up the user session at login time because we make the user answer some questions at login time that determine which portions of the interface to present to him/her. For example, using the same login id and password, a user may choose to login as an administrator or as a normal user. We store this choice and other info based on this choice in the session. Also, we don't rely on the browser to time out the authentication cookie. Once a user authenticates with ExUserFolder, ExUserFolder keeps their credentials in a cache until they have been inactive for 10 minutes (the timer resets with each cache hit). If their credentials are not in the cache, rather than looking them up again, the user is logged out and must re-authenticate. It seems like a reasonable way to handle logins and sessions. In addition to going through log files, I will spend some more time today making sure we're not doing something stupid in our app. Thanks again (to Chris, Michael, Alex and everyone else who has lost sleep over this session stuff). I'll keep you posted on any new information I find. Steve Chris McDonough wrote: On Mon, 2004-05-17 at 23:08, Chris McDonough wrote: There indeed is a minor off-by-one error: it manifests itself as sessions timing out at most 20 seconds early. But there is also a deeper issue which involves the fact that a session data object is not properly removed from an older bucket when it moves due to being accessed in a later timeslice; the symptom only appears when a browser id is reused to start a session after it was used to start an older one that had timed out normally. I've got almost no clue why this happens at this point, but I'm working on it. Ugh. This is almost certainly what Steve is experiencing. I take that back. Actually, I think I was just reading the test results and debug output wrong. It appears to be operating normally except for the off-by-one problem (which is minor). I need to jack up the tests to do some comparisons of data values; currently I'm just testing to ensure that *something* is in the session.. I need to test if the right thing is in the session over time. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Andrea Patuzzo wrote: We are now randomly getting this kind of error: Note: once the error happens, zope has to be restarted to function again. Before applying patches we were getting get errors (instead of __setitem__) and we just had to close browser to get back to work. That's right. Under Plone 2.0 and Zope 2.7 I had the same kind of problem. But when I restarted the browser it would be solved. That could point in the direction of a session id being somehow involved. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Well, after much log reading, I have found that the KeyError we got last night was OUR fault. I will fix the problem with our app, then I want to change back to TemporaryStorage and watch the system some more. I'll keep you posted. Here's the brief explanation of our problem (you can skip it if you like): A user logged in and did some stuff then left his browser for almost an hour. When he returned and tried to do more stuff, he was no longer in the ExUserFolder's credential cache and his session had expired. He was forced to log in again. Upon supplying his ID and password, he was sent to the loginSuccess page. This is the one that calls our method to set up his user session. The Z2.log shows a 302 result code on this page. His browser had the loginSuccess page in cache, so it did not request it again and his session was never re-created. Score one for Chris's suggestion on how we should be setting up the user's session. For now, however, I think I'll just add the please-don't-cache-me header stuff to the RESPONSE. Steve Jibson wrote: Good morning. I just got in and checked on my customer's system. In the past 22 1/2 hours they've had 15000 page hits and last night at about 9:30, ONE person got a KeyError. Actually, this same person got twenty KeyErrors over a period of about 45 seconds. I'm downloading their log files now and plan to spend some time this morning going through them. Anyway, it appears that I was wrong when I said that the problem doesn't show up when I use FileStorage (although it does seem to happen less frequently -- but who can be sure of anything at this point?). In answer to your questions earlier, Chris, we set up the user session at login time because we make the user answer some questions at login time that determine which portions of the interface to present to him/her. For example, using the same login id and password, a user may choose to login as an administrator or as a normal user. We store this choice and other info based on this choice in the session. Also, we don't rely on the browser to time out the authentication cookie. Once a user authenticates with ExUserFolder, ExUserFolder keeps their credentials in a cache until they have been inactive for 10 minutes (the timer resets with each cache hit). If their credentials are not in the cache, rather than looking them up again, the user is logged out and must re-authenticate. It seems like a reasonable way to handle logins and sessions. In addition to going through log files, I will spend some more time today making sure we're not doing something stupid in our app. Thanks again (to Chris, Michael, Alex and everyone else who has lost sleep over this session stuff). I'll keep you posted on any new information I find. Steve ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 18/05/2004, at 10:03 PM, Andrea Patuzzo wrote: Hi, dear developers: We are now randomly getting this kind of error: [...] File /usr/local/Zope270CVS/lib/python/Products/Transience/Transience.py, line 341, in __setitem__ current_bucket = self._data[current_ts] KeyError: 1084872640 This one was with the latest publish.py / startup.py patch and the new Transience.py implementation, on a yesterday's Zope-2_7-branch checkout. New Zope instance with FileStorage does not solve the problem. Same happens with the patches applied to the official Zope-2.7.0 release. Note: once the error happens, zope has to be restarted to function again. Before applying patches we were getting get errors (instead of __setitem__) and we just had to close browser to get back to work. Here we have to restart Zope (with TemporaryStorage) or delete Sessions.fs and restart (with FileStorage). My guess at the moment is that somewhere a ConflictError is being swallowed. There are a few things that can be responsible for this: - Use of hasattr on Persistent objects. - try/except blocks that don't raise ConflictErrors. From your trace that looks like a Plone site. A very quick glance through Plone and its friends indicates that there may be some chance of either of the above causing some problems. Although Plone 2.0.1 includes the following fix: - Fixed bare 'except:' statements in Plone so we don't swallow ZODB ConflictErrors. Mostly by adding 'except ConflictError: raise'. [stefan] Someone please shoot me down and claim that Plone and friends all do the right thing. ;-) Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
:-( I grabbed Transience.py and TemporaryStorage.py from the Zope-2_7-branch of CVS this morning and dropped them onto one of our customer's systems. About 20 minutes later I found the following in the error_log: Traceback (most recent call last): File c:\Zope-2.7\lib\python\ZPublisher\Publish.py, line 100, in publish request, bind=1) File c:\Zope-2.7\lib\python\ZPublisher\mapply.py, line 88, in mapply if debug is not None: return debug(object,args,context) File c:\Zope-2.7\lib\python\ZPublisher\Publish.py, line 40, in call_object result=apply(object,args) # Type scr to step into published object. File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 128, in __call__ r=apply(HTML.__call__, (self, (client, bself), REQUEST), kw) File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line 474, in __call__ try: result = render_blocks(self._v_blocks, md) File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 121, in __call__ r=apply(HTML.__call__, (self, bself, REQUEST), kw) File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line 474, in __call__ try: result = render_blocks(self._v_blocks, md) File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 121, in __call__ r=apply(HTML.__call__, (self, bself, REQUEST), kw) File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line 474, in __call__ try: result = render_blocks(self._v_blocks, md) File c:\Zope-2.7\lib\python\DocumentTemplate\DT_Util.py, line 201, in eval return eval(code, d) File string, line 1, in expression File c:\Zope-2.7\lib\python\AccessControl\ZopeGuards.py, line 67, in guarded_getitem v = object[index] File c:\Zope-2.7\lib\python\Products\Transience\TransientObject.py, line 170, in __getitem__ return self._container[k] KeyError: 'accountType' After getting the error, I changed them over to FileStorage. I haven't seen any errors since then. I will be more than happy to spend some time doing whatever I can to help track this down and/or test updates. Steve Chris McDonough wrote: On Sat, 2004-05-15 at 23:55, Steve Jibson wrote: quite yet been vetted (see earlier posts in this thread regarding Publish.py). This should work itself out over the next week or so I suspect. Relatively speaking, this is only a minor concern. It just means I shouldn't try to access session in the standard_error_message until a solution is found. We were doing that in our product and I've already pulled that out. After all, there are NEVER any errors, so why should we need to monkey with standard_error_message ;-) It would be nice, for completeness, if this were resolved for 2.7.1. Yes, it should work itself out over the next week or so, and definitely before 2.7.1. Am I safe just dropping those two files from CVS into an existing 2.7 site or do I need to grab everything else from the Zope-2_7-branch? I think you should be safe with just those two, but were I you I would just get the 2.7 branch out of CVS in its entirety (then you can update more easily if there are fixes). Note that if you have transient object containers in a permanent storage (like FileStorage), the operation is pretty much a one way one if you want to keep the data in the transient object container: you can go to the new code, then go back to the old code if necessarfy, but you will likely lose data in the TOC if you do. Obviously this doesn't effect stuff in a TemporaryStorage (because it goes away when Zope is shut down). - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 12:09, Steve Jibson wrote:
:-(
I grabbed Transience.py and TemporaryStorage.py from the Zope-2_7-branch
of CVS this morning and dropped them onto one of our customer's systems.
About 20 minutes later I found the following in the error_log:
Traceback (most recent call last):
File c:\Zope-2.7\lib\python\ZPublisher\Publish.py, line 100, in
publish request, bind=1)
File c:\Zope-2.7\lib\python\ZPublisher\mapply.py, line 88, in mapply
if debug is not None: return debug(object,args,context)
File c:\Zope-2.7\lib\python\ZPublisher\Publish.py, line 40, in
call_object
result=apply(object,args) # Type scr to step into published object.
File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 128, in __call__
r=apply(HTML.__call__, (self, (client, bself), REQUEST), kw)
File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line
474, in __call__
try: result = render_blocks(self._v_blocks, md)
File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 121, in __call__
r=apply(HTML.__call__, (self, bself, REQUEST), kw)
File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line
474, in __call__
try: result = render_blocks(self._v_blocks, md)
File c:\Zope-2.7\lib\python\OFS\DTMLDocument.py, line 121, in __call__
r=apply(HTML.__call__, (self, bself, REQUEST), kw)
File c:\Zope-2.7\lib\python\DocumentTemplate\DT_String.py, line
474, in __call__
try: result = render_blocks(self._v_blocks, md)
File c:\Zope-2.7\lib\python\DocumentTemplate\DT_Util.py, line 201,
in eval
return eval(code, d)
File string, line 1, in expression
File c:\Zope-2.7\lib\python\AccessControl\ZopeGuards.py, line 67,
in guarded_getitem
v = object[index]
File c:\Zope-2.7\lib\python\Products\Transience\TransientObject.py,
line 170, in __getitem__
return self._container[k]
KeyError: 'accountType'
This looks to me like an application error, not a Transience error.
After getting the error, I changed them over to FileStorage. I haven't
seen any errors since then. I will be more than happy to spend some
time doing whatever I can to help track this down and/or test updates.
It looks like the code assumes 'accountType' is available in the
session; change it to not make that assumption (perhaps via
SESSION.get('accountType', 'default')).
- C
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 17 May 2004 19:00:16 +0200 Dieter Maurer [EMAIL PROTECTED] wrote: Chris McDonough wrote at 2004-5-15 13:04 -0400: ... Dieter, do you think you can read this patch and give a thumbs up or down on it? The patch looks good. On a different subject, the publisher probably shouldn't pass around traceback objects (e.g. when it calls into err_hook) as Tres believes that may be a memory leak waiting to happen. The traceback is vital for error analysis. It may not be necessary that ZPublisher touches the traceback but we will definitely need access to it during error handling. Perhaps the traceback can be passed as a string to avoid leaks? Furthermore why can't the traceback be retrieved later from sys.exc_info()? -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Casey Duncan wrote: On Mon, 17 May 2004 19:00:16 +0200 Dieter Maurer [EMAIL PROTECTED] wrote: Chris McDonough wrote at 2004-5-15 13:04 -0400: ... Dieter, do you think you can read this patch and give a thumbs up or down on it? The patch looks good. On a different subject, the publisher probably shouldn't pass around traceback objects (e.g. when it calls into err_hook) as Tres believes that may be a memory leak waiting to happen. The traceback is vital for error analysis. It may not be necessary that ZPublisher touches the traceback but we will definitely need access to it during error handling. Because the traceback contains stack frames, passing it through another stack frame (via a function call) is inherently tricky: the called function must *not* raise another exception. Perhaps the traceback can be passed as a string to avoid leaks? Furthermore why can't the traceback be retrieved later from sys.exc_info()? +1; I don't want untrusted code handling tracebacks anyway. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 13:06, Steve Jibson wrote: I'm sure 'accountType' should have been in the session. Immediately after a user logs in we populate his session with a bunch of stuff (including accountType). What happens when the session expires and he's still logged in? What I don't want to do is hide problems with session/tempstorage by using get and a default. In this particular case, there is no good default and having the application assume an incorrect accountType will cause me all kids of trouble. I still suspect this is an application error. Also of note: 1 - After having the system run for 45 minutes, I had 8 similar errors. Some were on different web pages and some had different keys that were causing the error. 2 - I have also traced through Z2.log and followed the same path through the web site that produced one of the errors and I did not get an error. 3 - Since changing this server to use FileStorage (1hr 39min ago), there has not been a single error. Is there anything I can do to help here? A small reproducible test case would help if you still believe this error is not in your own application. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: On Mon, 2004-05-17 at 13:06, Steve Jibson wrote: I'm sure 'accountType' should have been in the session. Immediately after a user logs in we populate his session with a bunch of stuff (including accountType). What happens when the session expires and he's still logged in? We're using ExUserFolder for authentication. We have it set to log users out after 10 minutes. We have sessions set to expire after 20 minutes. So, in theory, it should never happen. Also, since my last Zope restart (3hr 30min), I've had 57 users let their authentication timeout and then access the site (forcing another login), and still no errors with session. (It's still using FileStorage). 1 - After having the system run for 45 minutes, I had 8 similar errors. Some were on different web pages and some had different keys that were causing the error. 2 - I have also traced through Z2.log and followed the same path through the web site that produced one of the errors and I did not get an error. 3 - Since changing this server to use FileStorage (1hr 39min ago), there has not been a single error. Is there anything I can do to help here? A small reproducible test case would help if you still believe this error is not in your own application. I'm sure you know by now that this is easier said than done. Over the past few weeks, I've probably put close to 30 hours into just trying to reproduce (on demand) these session errors. I'll probably want to look at the test rig you and Michael have been using. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 14:58, Steve Jibson wrote: We're using ExUserFolder for authentication. We have it set to log users out after 10 minutes. We have sessions set to expire after 20 minutes. So, in theory, it should never happen. Are you sure that their auth really does time out after exactly 10 minutes? 10 minutes of inactivity? Is it based on a cookie timeout? Does it reset the cookie on every request? This relies on assuming all browsers respect the cookie timeout adequately at that resolution, as well. I don't know how well various browsers do this. Also, since my last Zope restart (3hr 30min), I've had 57 users let their authentication timeout and then access the site (forcing another login), and still no errors with session. (It's still using FileStorage). It might be pedantic, but I still think relying on the auth timeout behavior is wrong. Instead, of populating initial session data at login time, a bit of code called from your main template should probably do something like if not context.session_data_manager.hasSessionData(): ... populate session data based on auth info Or maybe an access rule that does the same thing (might be tricky though, I haven't tested that well). The right thing to do here might be to use an auth product that stores auth info in a session; that would isolate the problem nicely because there wouldn't be competing timeouts. Apparently someone has coded one up as SessionCrumbler somwhere. The fact that it works under FileStorage is curious, but might be a coincidence too. Lots of machinery here. ;-) A small reproducible test case would help if you still believe this error is not in your own application. I'm sure you know by now that this is easier said than done. Sure. I've spent god knows how long doing it. ;-) Over the past few weeks, I've probably put close to 30 hours into just trying to reproduce (on demand) these session errors. I'll probably want to look at the test rig you and Michael have been using. Yes, although that doesn't do any data validity checks, AFAIK. It would be interesting to try to extend it to do so. You might also want to consider running Zope with the Z_TOC_DEBUG envvar set to true. This spews out a bunch of messages to the error log for each session access. Coupled with a log message noting when someone logs in *and their browser id*, this could provide clues. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 18/05/2004, at 5:42 AM, Chris McDonough wrote: On Mon, 2004-05-17 at 13:06, Steve Jibson wrote: Also of note: 1 - After having the system run for 45 minutes, I had 8 similar errors. Some were on different web pages and some had different keys that were causing the error. 2 - I have also traced through Z2.log and followed the same path through the web site that produced one of the errors and I did not get an error. 3 - Since changing this server to use FileStorage (1hr 39min ago), there has not been a single error. Is there anything I can do to help here? A small reproducible test case would help if you still believe this error is not in your own application. Looks like session data can expire prematurely. See attached files for small changes to the test rig that reports cases where context.session_data_manager.hasSessionData() is False. Michael SessionRigExtensions.py Description: Binary data TestSessionRig.py Description: Binary data ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 17:52, Michael Dunstan wrote: Looks like session data can expire prematurely. See attached files for small changes to the test rig that reports cases where context.session_data_manager.hasSessionData() is False. Michael I think you're right, good eye! The symptoms aren't consistent with what Steve is reporting, however (where the problem appears when using TemporaryStorage but not FileStorage), so I this may be a distinct issue. I am looking in to it now. It feels like an off-by-one error. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
There indeed is a minor off-by-one error: it manifests itself as sessions timing out at most 20 seconds early. But there is also a deeper issue which involves the fact that a session data object is not properly removed from an older bucket when it moves due to being accessed in a later timeslice; the symptom only appears when a browser id is reused to start a session after it was used to start an older one that had timed out normally. I've got almost no clue why this happens at this point, but I'm working on it. Ugh. This is almost certainly what Steve is experiencing. - C On Mon, 2004-05-17 at 20:26, Chris McDonough wrote: On Mon, 2004-05-17 at 17:52, Michael Dunstan wrote: Looks like session data can expire prematurely. See attached files for small changes to the test rig that reports cases where context.session_data_manager.hasSessionData() is False. Michael I think you're right, good eye! The symptoms aren't consistent with what Steve is reporting, however (where the problem appears when using TemporaryStorage but not FileStorage), so I this may be a distinct issue. I am looking in to it now. It feels like an off-by-one error. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thu, 2004-05-13 at 18:11, Tres Seaver wrote: Chris McDonough wrote: Largely due to Michael I believe I have isolated and fixed every reported sessioning error except this (still-difficult-to-reproduce-but-definitely-still-existing) KeyError bug in temporary storage. I can let the test rig run for several hours; it happens maybe once every hour or two, so I've not gotten the provocation of it down to a science yet. It doesn't occur when the transient object container is placed in a FileStorage. I have now figured out where the KeyError from TemporaryStorage was coming from and have checked in a fix on the 2.7 branch and HEAD. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris, Okay, correct me if I'm wrong. This means the big ugly SESSION problem his completely fixed. If so, you are my new best friend! (whether you like it or not). Anyway, I'll put your updates (TemporaryStorage.py and Transience.py) on a couple of my customer's systems (that have been experiencing regular KeyErrors in SESSION) on Monday and I'll let you know if I see any problems. This is pretty big. If all goes well with testing (I assume Michael will be testing your latest stuff as well), I think this is important enough that y'all should consider a 2.7.1 release with this fix. Thanks again! (I know you've probably lost some hair and a bit of sanity over this one.) Chris McDonough wrote: On Thu, 2004-05-13 at 18:11, Tres Seaver wrote: Chris McDonough wrote: Largely due to Michael I believe I have isolated and fixed every reported sessioning error except this (still-difficult-to-reproduce-but-definitely-still-existing) KeyError bug in temporary storage. I can let the test rig run for several hours; it happens maybe once every hour or two, so I've not gotten the provocation of it down to a science yet. It doesn't occur when the transient object container is placed in a FileStorage. I have now figured out where the KeyError from TemporaryStorage was coming from and have checked in a fix on the 2.7 branch and HEAD. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Sat, 2004-05-15 at 23:11, Steve Jibson wrote: Chris, Okay, correct me if I'm wrong. This means the big ugly SESSION problem his completely fixed. If so, you are my new best friend! (whether you like it or not). Well, I'm afraid I will need to correct you. ;-) There is one more corner case that hasn't been covered that exhibits itself when a session is accessed from within the standard_error_message. It requires a change in the main publishing code, for which exists a patch that hasn't quite yet been vetted (see earlier posts in this thread regarding Publish.py). This should work itself out over the next week or so I suspect. Anyway, I'll put your updates (TemporaryStorage.py and Transience.py) on a couple of my customer's systems (that have been experiencing regular KeyErrors in SESSION) on Monday and I'll let you know if I see any problems. Thanks... make sure you use the latest code from the Zope-2_7-branch in CVS. This is pretty big. If all goes well with testing (I assume Michael will be testing your latest stuff as well), I think this is important enough that y'all should consider a 2.7.1 release with this fix. 2.7.1b1 is scheduled to be released on May 24 (http://dev.zope.org/Wikis/DevSite/Projects/Zope2.7/FrontPage/ReleaseSchedule27) Thanks again! (I know you've probably lost some hair and a bit of sanity over this one.) Hope it goes smoothly. I'm bald at this point. ;-) - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: On Sat, 2004-05-15 at 23:11, Steve Jibson wrote: Chris, Okay, correct me if I'm wrong. This means the big ugly SESSION problem his completely fixed. If so, you are my new best friend! (whether you like it or not). Well, I'm afraid I will need to correct you. ;-) There is one more corner case that hasn't been covered that exhibits itself when a session is accessed from within the standard_error_message. It requires a change in the main publishing code, for which exists a patch that hasn't quite yet been vetted (see earlier posts in this thread regarding Publish.py). This should work itself out over the next week or so I suspect. Relatively speaking, this is only a minor concern. It just means I shouldn't try to access session in the standard_error_message until a solution is found. We were doing that in our product and I've already pulled that out. After all, there are NEVER any errors, so why should we need to monkey with standard_error_message ;-) It would be nice, for completeness, if this were resolved for 2.7.1. Anyway, I'll put your updates (TemporaryStorage.py and Transience.py) on a couple of my customer's systems (that have been experiencing regular KeyErrors in SESSION) on Monday and I'll let you know if I see any problems. Thanks... make sure you use the latest code from the Zope-2_7-branch in CVS. Am I safe just dropping those two files from CVS into an existing 2.7 site or do I need to grab everything else from the Zope-2_7-branch? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Sat, 2004-05-15 at 23:55, Steve Jibson wrote: quite yet been vetted (see earlier posts in this thread regarding Publish.py). This should work itself out over the next week or so I suspect. Relatively speaking, this is only a minor concern. It just means I shouldn't try to access session in the standard_error_message until a solution is found. We were doing that in our product and I've already pulled that out. After all, there are NEVER any errors, so why should we need to monkey with standard_error_message ;-) It would be nice, for completeness, if this were resolved for 2.7.1. Yes, it should work itself out over the next week or so, and definitely before 2.7.1. Am I safe just dropping those two files from CVS into an existing 2.7 site or do I need to grab everything else from the Zope-2_7-branch? I think you should be safe with just those two, but were I you I would just get the 2.7 branch out of CVS in its entirety (then you can update more easily if there are fixes). Note that if you have transient object containers in a permanent storage (like FileStorage), the operation is pretty much a one way one if you want to keep the data in the transient object container: you can go to the new code, then go back to the old code if necessarfy, but you will likely lose data in the TOC if you do. Obviously this doesn't effect stuff in a TemporaryStorage (because it goes away when Zope is shut down). - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: Largely due to Michael I believe I have isolated and fixed every reported sessioning error except this (still-difficult-to-reproduce-but-definitely-still-existing) KeyError bug in temporary storage. I can let the test rig run for several hours; it happens maybe once every hour or two, so I've not gotten the provocation of it down to a science yet. It doesn't occur when the transient object container is placed in a FileStorage. I am tempted to check the following into the 2.7 branch and HEAD: - error occurs in same transaction as main request patch to Publish.py. See http://www.plope.com/Members/chrism/publishpy_errorinmaintrainsaction.patch/file_view for the patch. - new Transience implementation (changed only slightly from chrism-sessiongeddon-branch) ... and worry about the TemporaryStorage bug as a separate issue. Thoughts? +1 from me. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: On Wed, 2004-03-10 at 19:30, Paul Winkler wrote: I suggested doing a bug day as part of the pycon z2 sprint, but we haven't really discussed the various sprint proposals in depth yet. Since you and I are going to be there on Saturday and Sunday, why don't we do this on those days and encourage people to participate via IRC? We can decide to do something else (or not) for Monday and Tuesday, if the mood strikes. What dates are those Saturday and Sunday? I'm not at PyCon but might see if I can pitch in from the UK... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Saturday Mar 20, Sun Mar 21... On Thu, 2004-03-11 at 06:54, Chris Withers wrote: Chris McDonough wrote: On Wed, 2004-03-10 at 19:30, Paul Winkler wrote: I suggested doing a bug day as part of the pycon z2 sprint, but we haven't really discussed the various sprint proposals in depth yet. Since you and I are going to be there on Saturday and Sunday, why don't we do this on those days and encourage people to participate via IRC? We can decide to do something else (or not) for Monday and Tuesday, if the mood strikes. What dates are those Saturday and Sunday? I'm not at PyCon but might see if I can pitch in from the UK... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 10 Mar 2004 17:50:41 -0500 Chris McDonough [EMAIL PROTECTED] wrote: Nevermind. http://zope.org/Collectors/Zope/789 and http://zope.org/Collectors/Zope/786 The bug neglector is really living up to its name lately (not pointing fingers, mea culpa). 'Bout time fer a bug day I reckon... ;^) -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, Mar 10, 2004 at 06:14:30PM -0500, Casey Duncan wrote: On Wed, 10 Mar 2004 17:50:41 -0500 Chris McDonough [EMAIL PROTECTED] wrote: Nevermind. http://zope.org/Collectors/Zope/789 and http://zope.org/Collectors/Zope/786 The bug neglector is really living up to its name lately (not pointing fingers, mea culpa). 'Bout time fer a bug day I reckon... ;^) I suggested doing a bug day as part of the pycon z2 sprint, but we haven't really discussed the various sprint proposals in depth yet. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's FORNICATOR BOY! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
