Re: [Zope-dev] Re: Sessions and long requests = conflicts? Mcdutils?
> Did you configure the faster SDC to disable the 'lazy' flag? That flag > disables modification of the SDC's linked list until the sesion is > actually modified, rather than being just accessed as you are doing here. I tried both, with and without 'lazy' setting. I used _BUCKET_TYPE = AppendOnlyDict, timeout set to 1200 and timeout resolution: 20 I did three concurrent requests (from three browsers) twice (first request was just to initially create session data objects). With second request, lazy or not, every time I get errors like these attached below. Function that is called sleeps for 50 seconds and just gets the access to SESSION object - doesn't put anything into session. I've added print statements to _p_resolveConflict too but seems this function is only called once in this situation. In general tuning up session resolution seconds helps here. Thanks for your answers I've read about Faster storage and I think I understand this much more now, although I'm not sure about conflicts below. (...) 2007-04-12 21:43:28 DEBUG txn.1090525536 new transaction testme started 2007-04-12 21:43:29 DEBUG txn.1098918240 new transaction testme started 2007-04-12 21:43:30 DEBUG txn.1082132832 new transaction testme started testme after sleep 2007-04-12 21:44:18 DEBUG txn.1090525536 commit 2007-04-12 21:44:18 DEBUG txn.1090525536 commit testme after sleep 2007-04-12 21:44:19 INFO ZPublisher.Conflict ConflictError at /testme: database conflict error (oid 0x2b, class Products.faster.sessiondata.SessionDataContainer, serial this txn started with 0x036ce71f217604aa 2007-04-12 19:43:07.842424, serial currently committed 0x036ce7204e9b7a44 2007-04-12 19:44:18.423594) (21 conflicts (0 unresolved) since startup at Thu Apr 12 19:55:58 2007) 2007-04-12 21:44:20 DEBUG txn.1098918240 abort 2007-04-12 21:44:20 DEBUG txn.1098918240 new transaction testme started testme after sleep 2007-04-12 21:44:20 INFO ZPublisher.Conflict ConflictError at /testme: database conflict error (oid 0x2b, class Products.faster.sessiondata.SessionDataContainer, serial this txn started with 0x036ce71f217604aa 2007-04-12 19:43:07.842424, serial currently committed 0x036ce7204e9b7a44 2007-04-12 19:44:18.423594) (22 conflicts (0 unresolved) since startup at Thu Apr 12 19:55:58 2007) 2007-04-12 21:44:21 DEBUG txn.1082132832 abort 2007-04-12 21:44:21 DEBUG txn.1082132832 new transaction testme started testme after sleep 2007-04-12 21:45:10 DEBUG txn.1098918240 commit 2007-04-12 21:45:10 DEBUG txn.1098918240 commit testme after sleep inside of _p_resolveConflict for SessionDataContainer 2007-04-12 21:45:11 DEBUG txn.1082132832 commit 2007-04-12 21:45:11 DEBUG txn.1082132832 commit (...) -- Maciej Wisniowski ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sessions and long requests = conflicts? Mcdutils?
Maciej Wisniowski wrote at 2007-4-11 20:28 +0200: >Currently I have some problems with our application (Zope2.8.4) >and with Conflict Errors in sessions. >In general if we have few concurrent requests that are running >sometimes for 3-4 minutes (and they're touching session inside) >I get a lot of conflict errors with Inceraser, OOBTree, Length2 etc. Session conflict errors are very common -- for the following reason: While individual sessions are usually used only by a single request at a time, they share common administrative data structures (e.g. the OOBTree that holds the session objects). The administrative data structures usually have conflict resolution *BUT* conflict resolution requires sufficient history -- and the temporary storage usually holding the sessions only has minimal history. Therefore, conflict resolution is usually ineffective (impossible) and you get lots of conflicts. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: unit tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jens Vagelpohl wrote: > > On 12 Apr 2007, at 13:35, Miles wrote: >>> However, having logged this, the testrunner proceeded to run all my >>> tests correctly, and so the buildbot reported a pass. I expected >>> the testrunner to bomb out on encountering this error, but it >>> didn't - however if I start zope with zopectl fg, the bug I >>> introduced causes it to fail to start. >>> >>> I'm a bit concerned that there was a false test pass. Is there a >>> way to stop the testrunner in this case, in the same way as running >>> in fg mode? > > If I remember correctly the startup success/failure with a defective > product present is governed by debug mode. If you turn it on in > zope.conf and attempt to start Zope via "zopectl start" it should > fail as desired. "zopectl fg" forces debug mode to be switched on. Correct. The reason the OP's tests are passing is likely that they are real unit tests, and therefore don't depend on having the product's 'initialize' method run (if they did, they would be *integration* or *functional* tests). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGHkR9+gerLs4ltQ4RAng/AJ41pLpp8Ro89oxpkdCcSmtAC/9CvQCgnEzL 6jt0KI9fah8uPiJum901HD8= =ZpeV -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Sessions and long requests = conflicts? Mcdutils?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Maciej Wisniowski wrote: >> If you believe that 'faster' is raising conflicts due its own internal >> data structures (OOBTree bucket splits), rather than in the >> application-dveined session data, there is a conflict-free alternative >> available: we found that it was slower than the other, and therefore >> didn't scale as well, even given the possibility of conflicts. >> >> To enable the conflict-free storage, you need to patch the >> '_BUCKET_TYPE' class-level variable of the storage to use >> 'AppendOnlyDict' rather than 'OOBTree'. E.g.: >> >> from Products.faster.sessiondata import CBSessionDataContainer >> from Products.faster.appendict import AppendOnlyDict >> CBSessionDataContainer._BUCKET_TYPE = AppendOnlyDict >> > I've changed to: > _BUCKET_TYPE = AppendOnlyDict > > in sessiondata.py for SessionDataContainer and > CBSessionDataContainer. > > Unfortunatelly when I run external method like > below concurrently from 3 different browsers > (Opera, FF, Konqueror) I still get conflict errors. > > import time > def testme(self): > print 'testme started' > self.REQUEST.SESSION > time.sleep(50) > print 'testme after sleep' > return 'finished' Did you configure the faster SDC to disable the 'lazy' flag? That flag disables modification of the SDC's linked list until the sesion is actually modified, rather than being just accessed as you are doing here. > Faster 'Resolution secs' is set to 20. > Console output: > > 2007-04-12 10:31:50 DEBUG txn.-1260708960 new transaction > testme started > 2007-04-12 10:31:51 DEBUG txn.-1277494368 new transaction > testme started > 2007-04-12 10:31:54 DEBUG txn.-1252316256 new transaction > testme started > testme after sleep > 2007-04-12 10:32:40 DEBUG txn.-1260708960 commit > 2007-04-12 10:32:40 DEBUG txn.-1260708960 commit > testme after sleep > 2007-04-12 10:32:41 INFO ZPublisher.Conflict ConflictError at /testme: > database conflict error (oid 0x2b, class > Products.faster.sessiondata.SessionDataContainer, serial this txn > started with 0x036ce4767ce55799 2007-04-12 08:22:29.272469, serial > currently committed 0x036ce480ad6d3044 2007-04-12 08:32:40.646840) (3 > conflicts (0 unresolved) since startup at Thu Apr 12 10:17:09 2007) > 2007-04-12 10:32:41 DEBUG txn.-1277494368 abort > 2007-04-12 10:32:41 DEBUG txn.-1277494368 new transaction > testme started > testme after sleep > 2007-04-12 10:32:44 INFO ZPublisher.Conflict ConflictError at /testme: > database conflict error (oid 0x2b, class > Products.faster.sessiondata.SessionDataContainer, serial this txn > started with 0x036ce4767ce55799 2007-04-12 08:22:29.272469, serial > currently committed 0x036ce480ad6d3044 2007-04-12 08:32:40.646840) (4 > conflicts (0 unresolved) since startup at Thu Apr 12 10:17:09 2007) > 2007-04-12 10:32:45 DEBUG txn.-1252316256 abort > 2007-04-12 10:32:45 DEBUG txn.-1252316256 new transaction > testme started > testme after sleep > 2007-04-12 10:33:31 DEBUG txn.-1277494368 commit > 2007-04-12 10:33:31 DEBUG txn.-1277494368 commit > testme after sleep > 2007-04-12 10:33:35 DEBUG txn.-1252316256 commit > 2007-04-12 10:33:35 DEBUG txn.-1252316256 commit > > Also after few different tires, eg. with Resolution secs == 300 I > ecountered conflicts with > AppendOnlyDict like: > > 2007-04-12 10:53:07 INFO ZPublisher.Conflict ConflictError at /testme: > database conflict error (oid 0x4c, class > Products.faster.appendict.AppendOnlyDict, serial this txn started with > 0x036ce49409621366 2007-04-12 08:52:02.199166, serial currently > committed 0x036ce49515ed0477 2007-04-12 08:53:05.138871) (9 conflicts (0 > unresolved) since startup at Thu Apr 12 10:17:09 2007) That log message says you *are* modifying the AppendOnlyDict from multiple transactions, and it is resolving those conflicts; there are therefore no retried or failed transactions here. > Is there a point in faster where I can put debug message to see that > bucket splits (or something like that) happened? There are no bucket splits when using AOD. You could put a 'pdb.set_trace()' inside its '_p_resolveConflicts' method, to see when it was being called. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGHkPW+gerLs4ltQ4RAlfqAJ4zi+2IwUNV8AA9IT/K1revgMUwTQCfXxJ2 oJ9BfiRzIGcQZrUD1/sSwpQ= =T5oL -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sessions and long requests = conflicts? Mcdutils?
> You could keep experimenting with values to reduce the chances of > conflicts. Perhaps sessions that last for days. With resolution of > hours. Disabling inband housekeeping. > > Note that a session-timeout-minutes of 0 enables a slightly different > approach which has a little less "active" structure. Yes, setting high values for timeout and session resolution seconds or disabling session timeout by setting it to '0' reduces rate of conflict. I tried disabling inband housekeeping but this didn't helped in this case. > > I don't think session mechanics operates like that at the end of a > transaction. More generally what is happening is that the second > transaction is trying to commit data that was changed by an earlier > transaction after second transaction read that data. In this case the > data is various bits of the internals that make up sessions and > transience storage. Right but I would like to know how exactly this goes, eg. when I can expect conflicts. So far I'm still not sure when and why conflict will appear. Thanks for the answer -- Maciej Wisniowski ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] unit tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12 Apr 2007, at 13:35, Miles wrote: However, having logged this, the testrunner proceeded to run all my tests correctly, and so the buildbot reported a pass. I expected the testrunner to bomb out on encountering this error, but it didn't - however if I start zope with zopectl fg, the bug I introduced causes it to fail to start. I'm a bit concerned that there was a false test pass. Is there a way to stop the testrunner in this case, in the same way as running in fg mode? If I remember correctly the startup success/failure with a defective product present is governed by debug mode. If you turn it on in zope.conf and attempt to start Zope via "zopectl start" it should fail as desired. "zopectl fg" forces debug mode to be switched on. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGHiRPRAx5nvEhZLIRAg9+AJ9/Rc5iXyl3HNasxXcojJ0VrFi7AACbBaui LxCX4H/TRbg7G8NN8MZqcsE= =y1FY -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] unit tests
Hi, I am running a set of unit tests for a custom product nightly via a buildbot installation, via bin/zopectl test -s Products.MyProduct in zope 2.9. Sorry in advance that this looks complicated - my query is actually pretty straightforward! Recently, I introduced a bug in the __init__.py of the product, which caused the product installation to fail in zope. The following was written to stdout: 2007-04-12 10:05:34 ERROR Zope Couldn't install CMFMyProduct Traceback (most recent call last): File "/usr/local/Zope-2.9.5/lib/python/OFS/Application.py", line 783, in install_product initmethod(context) File "/home/buildslave/buildslaves/astrodon/build/Zope-2.9.5/Products/CMFMyProduct/__init__.py", line 227, in initialize context.registerClass(MyModule.MyClass, NameError: global name 'MyModule' is not defined The trivial error was that MyModule was not imported earlier in __init__.py However, having logged this, the testrunner proceeded to run all my tests correctly, and so the buildbot reported a pass. I expected the testrunner to bomb out on encountering this error, but it didn't - however if I start zope with zopectl fg, the bug I introduced causes it to fail to start. I'm a bit concerned that there was a false test pass. Is there a way to stop the testrunner in this case, in the same way as running in fg mode? Thanks, Miles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 5 OK
Summary of messages to the zope-tests list. Period Wed Apr 11 12:00:00 2007 UTC to Thu Apr 12 12:00:00 2007 UTC. There were 5 messages: 5 from Zope Unit Tests. Tests passed OK --- Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Wed Apr 11 20:50:22 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-April/007578.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Wed Apr 11 20:51:52 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-April/007579.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Wed Apr 11 20:53:22 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-April/007580.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Wed Apr 11 20:54:52 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-April/007581.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Wed Apr 11 20:56:22 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-April/007582.html ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Sessions and long requests = conflicts? Mcdutils?
> If you believe that 'faster' is raising conflicts due its own internal > data structures (OOBTree bucket splits), rather than in the > application-dveined session data, there is a conflict-free alternative > available: we found that it was slower than the other, and therefore > didn't scale as well, even given the possibility of conflicts. > > To enable the conflict-free storage, you need to patch the > '_BUCKET_TYPE' class-level variable of the storage to use > 'AppendOnlyDict' rather than 'OOBTree'. E.g.: > > from Products.faster.sessiondata import CBSessionDataContainer > from Products.faster.appendict import AppendOnlyDict > CBSessionDataContainer._BUCKET_TYPE = AppendOnlyDict > I've changed to: _BUCKET_TYPE = AppendOnlyDict in sessiondata.py for SessionDataContainer and CBSessionDataContainer. Unfortunatelly when I run external method like below concurrently from 3 different browsers (Opera, FF, Konqueror) I still get conflict errors. import time def testme(self): print 'testme started' self.REQUEST.SESSION time.sleep(50) print 'testme after sleep' return 'finished' Faster 'Resolution secs' is set to 20. Console output: 2007-04-12 10:31:50 DEBUG txn.-1260708960 new transaction testme started 2007-04-12 10:31:51 DEBUG txn.-1277494368 new transaction testme started 2007-04-12 10:31:54 DEBUG txn.-1252316256 new transaction testme started testme after sleep 2007-04-12 10:32:40 DEBUG txn.-1260708960 commit 2007-04-12 10:32:40 DEBUG txn.-1260708960 commit testme after sleep 2007-04-12 10:32:41 INFO ZPublisher.Conflict ConflictError at /testme: database conflict error (oid 0x2b, class Products.faster.sessiondata.SessionDataContainer, serial this txn started with 0x036ce4767ce55799 2007-04-12 08:22:29.272469, serial currently committed 0x036ce480ad6d3044 2007-04-12 08:32:40.646840) (3 conflicts (0 unresolved) since startup at Thu Apr 12 10:17:09 2007) 2007-04-12 10:32:41 DEBUG txn.-1277494368 abort 2007-04-12 10:32:41 DEBUG txn.-1277494368 new transaction testme started testme after sleep 2007-04-12 10:32:44 INFO ZPublisher.Conflict ConflictError at /testme: database conflict error (oid 0x2b, class Products.faster.sessiondata.SessionDataContainer, serial this txn started with 0x036ce4767ce55799 2007-04-12 08:22:29.272469, serial currently committed 0x036ce480ad6d3044 2007-04-12 08:32:40.646840) (4 conflicts (0 unresolved) since startup at Thu Apr 12 10:17:09 2007) 2007-04-12 10:32:45 DEBUG txn.-1252316256 abort 2007-04-12 10:32:45 DEBUG txn.-1252316256 new transaction testme started testme after sleep 2007-04-12 10:33:31 DEBUG txn.-1277494368 commit 2007-04-12 10:33:31 DEBUG txn.-1277494368 commit testme after sleep 2007-04-12 10:33:35 DEBUG txn.-1252316256 commit 2007-04-12 10:33:35 DEBUG txn.-1252316256 commit Also after few different tires, eg. with Resolution secs == 300 I ecountered conflicts with AppendOnlyDict like: 2007-04-12 10:53:07 INFO ZPublisher.Conflict ConflictError at /testme: database conflict error (oid 0x4c, class Products.faster.appendict.AppendOnlyDict, serial this txn started with 0x036ce49409621366 2007-04-12 08:52:02.199166, serial currently committed 0x036ce49515ed0477 2007-04-12 08:53:05.138871) (9 conflicts (0 unresolved) since startup at Thu Apr 12 10:17:09 2007) Is there a point in faster where I can put debug message to see that bucket splits (or something like that) happened? -- Maciej Wisniowski ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
