Re: [Zope-dev] SQLAlias Memory Leak?
The leaking SQLAlias I see is coming from: Shared.DC.ZRDB.Results I'm going to try the same type of fix in there and see what happens. -jon "Brian Lloyd" [EMAIL PROTECTED] writes: oops - I sent that before I had my coffee. I should have attached the checkin version, not the broken one :) Brian Lloyd[EMAIL PROTECTED] Software Engineer 540.371.6909 Digital Creations http://www.digicool.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] SQLAlias Memory Leak?
OK, I've put this into the Collector. I'm intrigued as to what the operating procedure is with the Collector though - there appear to be a large number of Pending jobs. Does that mean that Digital Creations haven't even looked at them, or that they just haven't done anything about them? The current Collector is very poor about providing visibility of what gets done. Quite a few bugs get closed out - check out the CHANGES.txt for each release. Each entry under "bugs fixed" generally represents a closed Collector item. I agree that it seems like there are a lot of things in the Collector, but realize that they are not all open and shut things. A fair number are feature requests rather than bugs, and some of the bug reports are in states that are not very visible from the public view (pending more info from the submitter, etc.) We hope to move to using Tracker in the future to make such things more visible. That said, there is a finite amount of resources that can go into Collector management at any one time (though with recent growth that should improve over what it is now). I actively triage the Collector before each release looking for reports that are high-impact (like memory leaks) to make sure they get fixed. I do plan to get to this for 2.3. We can't, however, always get to each issue the day it is submitted :) http://classic.zope.org:8080/Collector/1772/view This is causing big problems with our site, and we need to resolve it pretty desperately. I guess the another solution would be to ensure that all calls to SQL columns are the same case as in the database, but with around 500 ZSQLMethods - that is a daunting task. I have a test suite written that verfies this - it actually appears to be some interaction with the Record extension class (as SQLAliases don't leak if you access them from a plain ExtensionClass.Base-derived class). Hopefully I'll have a patch or workaround for you today. Brian Lloyd[EMAIL PROTECTED] Software Engineer 540.371.6909 Digital Creations http://www.digicool.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SQLAlias Memory Leak?
OK, I've put this into the Collector. I'm intrigued as to what the operating procedure is with the Collector though - there appear to be a large number of Pending jobs. Does that mean that Digital Creations haven't even looked at them, or that they just haven't done anything about them? http://classic.zope.org:8080/Collector/1772/view This is causing big problems with our site, and we need to resolve it pretty desperately. I guess the another solution would be to ensure that all calls to SQL columns are the same case as in the database, but with around 500 ZSQLMethods - that is a daunting task. -d On Wed, Dec 06, 2000 at 03:08:33PM -0800, Jon Prettyman wrote: I've seen this as well under DCOracle. -jon Dyon Balding [EMAIL PROTECTED] writes: OK, I've had more of a hack at this, and I've narrowed the problem down a little. It appears that any access to an SQLAlias object causes it to not be correctly released. The SQLAlias objects means that you can access column names using a different case to how they are stored in the database. For example, we store users in a users table in the database, with column names like USERNAME, but they are accessed with: dtml-var username, this causes the SQLAlias objects to stick around. One solution to our problem is to just do a dtml-var USERNAME instead. However that doesn't solve the problem with this "feature" of being able to call it with a different case. This is a fairly easy bug to reproduce, on my setup at least :). I have tested it with ZSQLRelay and ZODBDA both talking to SQLServer, and with ZPyGreSQLDA talking to PostgreSQL. I would appreciate it if someone else can reproduce the bug, and I'll put it in the Collector I guess. thanks -d -- Dyon Balding [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SQLAlias Memory Leak?
OK, I've had more of a hack at this, and I've narrowed the problem down a little. It appears that any access to an SQLAlias object causes it to not be correctly released. The SQLAlias objects means that you can access column names using a different case to how they are stored in the database. For example, we store users in a users table in the database, with column names like USERNAME, but they are accessed with: dtml-var username, this causes the SQLAlias objects to stick around. One solution to our problem is to just do a dtml-var USERNAME instead. However that doesn't solve the problem with this "feature" of being able to call it with a different case. This is a fairly easy bug to reproduce, on my setup at least :). I have tested it with ZSQLRelay and ZODBDA both talking to SQLServer, and with ZPyGreSQLDA talking to PostgreSQL. I would appreciate it if someone else can reproduce the bug, and I'll put it in the Collector I guess. thanks -d On Wed, Dec 06, 2000 at 09:06:47AM +1100, Dyon Balding wrote: Hi, I have an application that stores state data in a SQLServer database. The Zope and database servers run on different windows machines, using the ZODBCDA product to connect Zope to the ODBC machinery. What I have been observing is that the Zope client's memory usage increases consistently when ZSQL methods are being called. In the Debug screens, the culprit appears to be Shared.DC.ZRDB.RDB.SQLAlias. Here's a dump of the first few lines after hitting a ZSQL method a few hundred times using ab. The SQLAlias figures never seem to go down. Class December 5, 2000 1:58 pm December 5, 2000 2:01 pm Delta Shared.DC.ZRDB.RDB.SQLAlias 1048 4068 +3020 ZClasses.ZClass.ZClass 73 108 +35 OFS.DTMLMethod.DTMLMethod 63 94 +31 In fact most of our servers run out of memory and need to be restarted every couple of days when they get sufficient use. I've done a bit of a search of the mailing lists, and noone else appears to be having the same problem. So does anyone have an idea of what we might be doing differently to everyone else? We appear to have the latest versions installed. I've had a bit of a look through the code, and it appears that SQLAlias is used to case desensitize the column names of the returned data? Any pointers would be appreciated. thanks, d -- |Dyon Balding [EMAIL PROTECTED] | ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) -- Dyon Balding [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SQLAlias Memory Leak?
I've seen this as well under DCOracle. -jon Dyon Balding [EMAIL PROTECTED] writes: OK, I've had more of a hack at this, and I've narrowed the problem down a little. It appears that any access to an SQLAlias object causes it to not be correctly released. The SQLAlias objects means that you can access column names using a different case to how they are stored in the database. For example, we store users in a users table in the database, with column names like USERNAME, but they are accessed with: dtml-var username, this causes the SQLAlias objects to stick around. One solution to our problem is to just do a dtml-var USERNAME instead. However that doesn't solve the problem with this "feature" of being able to call it with a different case. This is a fairly easy bug to reproduce, on my setup at least :). I have tested it with ZSQLRelay and ZODBDA both talking to SQLServer, and with ZPyGreSQLDA talking to PostgreSQL. I would appreciate it if someone else can reproduce the bug, and I'll put it in the Collector I guess. thanks -d On Wed, Dec 06, 2000 at 09:06:47AM +1100, Dyon Balding wrote: Hi, I have an application that stores state data in a SQLServer database. The Zope and database servers run on different windows machines, using the ZODBCDA product to connect Zope to the ODBC machinery. What I have been observing is that the Zope client's memory usage increases consistently when ZSQL methods are being called. In the Debug screens, the culprit appears to be Shared.DC.ZRDB.RDB.SQLAlias. Here's a dump of the first few lines after hitting a ZSQL method a few hundred times using ab. The SQLAlias figures never seem to go down. Class December 5, 2000 1:58 pm December 5, 2000 2:01 pm Delta Shared.DC.ZRDB.RDB.SQLAlias 1048 4068 +3020 ZClasses.ZClass.ZClass 73 108 +35 OFS.DTMLMethod.DTMLMethod 63 94 +31 In fact most of our servers run out of memory and need to be restarted every couple of days when they get sufficient use. I've done a bit of a search of the mailing lists, and noone else appears to be having the same problem. So does anyone have an idea of what we might be doing differently to everyone else? We appear to have the latest versions installed. I've had a bit of a look through the code, and it appears that SQLAlias is used to case desensitize the column names of the returned data? Any pointers would be appreciated. thanks, d -- |Dyon Balding [EMAIL PROTECTED] | ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) -- Dyon Balding [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
