Re: [Webware-discuss] MiddleKit Threading Error?
--- On Tue, 11/4/08, Christoph Zwerschke [EMAIL PROTECTED] wrote: From: Christoph Zwerschke [EMAIL PROTECTED] Subject: Re: [Webware-discuss] MiddleKit Threading Error? To: Discussion of Webware for Python including feedback and proposals. webware-discuss@lists.sourceforge.net Date: Tuesday, November 4, 2008, 12:13 PM Roger Haase schrieb: I had to revise your new MiddleObject.py to put the _mk_cacheLock = threading.RLock() statement at the module level rather than in the MiddleObject __init__ method. A new MiddleObject instance is being created with each request so putting the lock inside the object instance does not force single-threading through the readStoreData method. Right. I've put the lock on the class level now in r7581, that should work as well. The MiddleKit test suite does not run for me either. I will look over the 3 other 3 MiddleKit changes in the queue and update them for 1.0Rc1. It will probably take me about 2 weeks to get around to the task. Thanks. Anybody else who wants to brush up MiddleKit a bit before 1.0? -- Christoph I have downloaded the latest version of MiddleObject.py and verified that putting the lock on the class level works for me. Per your previous messages, it looks like you have already processed the 3 other MiddleKit changes in the queue. Thanks for all your help! Roger Haase - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
Roger Haase schrieb: I have downloaded the latest version of MiddleObject.py and verified that putting the lock on the class level works for me. Per your previous messages, it looks like you have already processed the 3 other MiddleKit changes in the queue. Good to hear. Yes, I wanted to process all these ancient bugs and patches on the SF tracker so I can finally release 1.0 in good conscience ;-) I'll cut a 1.0rc2 later this week. Let me know how it works for you. The MiddleKit tests now do all pass with both MySQL and PostgreSQL. The main issue was that the tests assumed autocommit which is not the default in the newer DB drivers. All other MiddleKit changes are listed in the release notes. -- Christoph - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
Christoph Zwerschke schrieb: Btw, I was trying to run the MiddleKit test suite, but I get many errors. For instance when running python Test.py MKDelete I get a Lock wait timeout exceeded (this hasn't anything to do with the lock in the patch, I get it also without your patch). I had another look at the MiddleKit tests and found that the problems are all caused by the fact that newer MySQLdb versions have autocommit disabled by default. MiddleKit however (at least the whole test suite) assumes autocommit is switched on. I have now patched MySQLObjectStore so that autocommit is explicitly activated for all MySQLDB connections, including those coming from a connection pool. I have also added an autocommit flag to the storeArgs, so you can also still disable the autocommit if you like. All MiddleKit tests are now runing for me. -- Christoph - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
--- On Sun, 11/2/08, Christoph Zwerschke [EMAIL PROTECTED] wrote: From: Christoph Zwerschke [EMAIL PROTECTED] Subject: Re: [Webware-discuss] MiddleKit Threading Error? To: [EMAIL PROTECTED], Discussion of Webware for Python including feedback and proposals. webware-discuss@lists.sourceforge.net Date: Sunday, November 2, 2008, 3:14 PM Roger Haase schrieb: My revised solution is to put the lock at the beginning and end of the method. I never used MiddleKit, so I don't know how to test this, but since it looks reasonable I've checked this in anyway, and cleaned up MiddleObject.py a bit. Can you check whether this works for you? http://svn.w4py.org/Webware/trunk/MiddleKit/Run/MiddleObject.py Btw, I was trying to run the MiddleKit test suite, but I get many errors. For instance when running python Test.py MKDelete I get a Lock wait timeout exceeded (this hasn't anything to do with the lock in the patch, I get it also without your patch). Does the test suite run for you? If yes, on what OS, Py and database (driver) version? Also, while we're at it, can you or somebody else have a look at the following MiddleKit patches: http://sourceforge.net/tracker/?group_id=4866atid=304866 There are a handful of other very old open issues here, mostly MiddleKit or other stuff I don't use. If anybody wants to work on these: http://sourceforge.net/tracker/?group_id=4866atid=104866 Ideally I'd like to have a running MiddleKit test suite and most of the issues above solved before releasing 1.0. -- Christoph I had to revise your new MiddleObject.py to put the _mk_cacheLock = threading.RLock() statement at the module level rather than in the MiddleObject __init__ method. A new MiddleObject instance is being created with each request so putting the lock inside the object instance does not force single-threading through the readStoreData method. While doing some more testing, I found that the problem is largely dependent upon Firefox 3. I have a static web page with 12 images and graphs that are generated by individual calls to Webware/MiddleKit. To run the test that creates the conditions for failure, I must restart the AppServer, clear the browser cache, and reload the page. Firefox 3 almost always causes the failure, Safari for Windows and Google Chrome rarely cause the failure, and IE 7 and Opera 9 never cause the failure (so far). The MiddleKit test suite does not run for me either. I will look over the 3 other 3 MiddleKit changes in the queue and update them for 1.0Rc1. It will probably take me about 2 weeks to get around to the task. Roger Haase - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
Roger Haase schrieb: I had to revise your new MiddleObject.py to put the _mk_cacheLock = threading.RLock() statement at the module level rather than in the MiddleObject __init__ method. A new MiddleObject instance is being created with each request so putting the lock inside the object instance does not force single-threading through the readStoreData method. Right. I've put the lock on the class level now in r7581, that should work as well. The MiddleKit test suite does not run for me either. I will look over the 3 other 3 MiddleKit changes in the queue and update them for 1.0Rc1. It will probably take me about 2 weeks to get around to the task. Thanks. Anybody else who wants to brush up MiddleKit a bit before 1.0? -- Christoph - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
Roger Haase schrieb: My revised solution is to put the lock at the beginning and end of the method. I never used MiddleKit, so I don't know how to test this, but since it looks reasonable I've checked this in anyway, and cleaned up MiddleObject.py a bit. Can you check whether this works for you? http://svn.w4py.org/Webware/trunk/MiddleKit/Run/MiddleObject.py Btw, I was trying to run the MiddleKit test suite, but I get many errors. For instance when running python Test.py MKDelete I get a Lock wait timeout exceeded (this hasn't anything to do with the lock in the patch, I get it also without your patch). Does the test suite run for you? If yes, on what OS, Py and database (driver) version? Also, while we're at it, can you or somebody else have a look at the following MiddleKit patches: http://sourceforge.net/tracker/?group_id=4866atid=304866 There are a handful of other very old open issues here, mostly MiddleKit or other stuff I don't use. If anybody wants to work on these: http://sourceforge.net/tracker/?group_id=4866atid=104866 Ideally I'd like to have a running MiddleKit test suite and most of the issues above solved before releasing 1.0. -- Christoph - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
Re: [Webware-discuss] MiddleKit Threading Error?
--- On Sun, 10/19/08, Roger Haase [EMAIL PROTECTED] wrote: From: Roger Haase [EMAIL PROTECTED] Subject: MiddleKit Threading Error? To: webware-discuss@lists.sourceforge.net Date: Sunday, October 19, 2008, 3:46 PM I last reported this problem back on May 22, 2004. I am not sure what I have been doing different lately, but I have encountered the problem on both my test system and my production system this past month. I have never been able to reproduce this problem at will. The problem last went away with some 0.8x or 0.9x release but has now occurred on 1.0RC1. I think the problem is caused by: 1. a freshly started AppServer 2. a flurry of incoming transactions all trying to access a MySQL table that has not been accessed before. I think I am hitting the problem because my application generates web pages with multiple embedded images that are served by the Webware application. So accessing a web page may create a flurry of transactions, the first has to wait for a disk read, and the second transactions overruns it resulting in duplicate entries in the cache. The abend occurs in MiddleKit/Run/MiddleObject in readStoreData at line 92: assert len(cache) + 1 == len(row) Per the fancy traceback, cache is equal to: [unbound method Folder.setFileName, unbound method Folder.setSurveyDate, unbound method Folder.setLockUserId, unbound method Folder.setPhotographer, unbound method Folder.setComments, unbound method Folder.setRootId, unbound method Folder.setSiteId, unbound method Folder.setNumberFiles, unbound method Folder.setUpdatedBy, unbound method Folder.setUpdateTime, -- the problem is here, the fields repeat unbound method Folder.setFileName, unbound method Folder.setSurveyDate, unbound method Folder.setLockUserId, unbound method Folder.setPhotographer, unbound method Folder.setComments, unbound method Folder.setRootId, unbound method Folder.setSiteId, unbound method Folder.setNumberFiles, unbound method Folder.setUpdatedBy, unbound method Folder.setUpdateTime] And row is equal to: (96L, 'MonumentNashRuin_AR-03-09-02-08', datetime.date(2007, 2, 6), 2L, 'RH, GH', '', 8L, 94L, 42L, 2L, datetime.datetime(2007, 9, 18, 12, 44, 3)) I think the solution is to insert a lock around the code that deals with loading the cache the first time a table is accessed. This is my diff from MiddleKit/Run/MiddleObject.py (probably hard to read because of the line wraps). Well, the previous solution worked for the failing transaction, but then I hit another problem on a flurry of 5 transactions that were generating PNG images. The new problem occurred in the same area - MiddleKit/Run/MiddleObject.py method readStoreData on the line: assert 0, attempted to refresh changed object My revised solution is to put the lock at the beginning and end of the method. --- C:\...\MiddleObject.py-revBASE.svn005.tmp.pyTue Oct 21 08:41:35 2008 +++ C:...\MiddleObject.py Tue Oct 21 08:36:16 2008 @@ -5,6 +5,9 @@ from MiddleKit.Core.ObjRefAttr import ObjRefAttr from MiddleKit.Core.ListAttr import ListAttr +import thread # 2002-08-21 rdh +_cacheLock = thread.allocate_lock() # 2002-08-21 rdh + try: # for Python 2.2 object except NameError: @@ -67,6 +70,7 @@ for the same object in order to refresh the attributes from the persistent store. + _cacheLock.acquire() # 2008-10-20 rdh if self._mk_store: assert self._mk_store is store, 'Cannot refresh data from a different store.' if self._mk_changed and not self._mk_initing: @@ -114,6 +118,7 @@ self._mk_initing = 0 self._mk_inStore = 1 self._mk_changed = 0 # setting the values above will have caused this to be set; clear it now. + _cacheLock.release() # 2008-10-20 rdh return self Roger Haase - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Webware-discuss mailing list Webware-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/webware-discuss
