I updated the program to make it easier for me to gather data. Probably too much information below.…
Looks like I am still seeing the problem on my Ubuntu 16.04. This run simulates approximately 1 week's worth of calls. Wrapping the Connection class, subclassing the Cursor class (WeeWX simulation). Before looping 2419200 times: mem_size: 31.593750 mem_rss: 7.300781 mem_share: 4.250000 After loop:: mem_size: 256.062500 mem_rss: 232.277344 mem_share: 4.878906 Garbage collected 0 objects After garbage collection:: mem_size: 256.062500 mem_rss: 232.277344 mem_share: 4.878906 The next 6 runs are different implementations of the connect, get cursor, execute statement, fetchone, close cursor loop. This simulates what WeeWX does. A wrapper around the Connection and a subclass for the Cursor. This is what I originally saw. Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.980469 mem_share: 3.933594 After loop:: mem_size: 15.535156 mem_rss: 11.933594 mem_share: 4.742188 Garbage collected 0 objects After garbage collection:: mem_size: 15.535156 mem_rss: 11.933594 mem_share: 4.742188 Using the sqlite3 module directly. Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.996094 mem_share: 3.949219 After loop:: mem_size: 10.597656 mem_rss: 6.875000 mem_share: 4.707031 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.875000 mem_share: 4.707031 Wrapping the Connection class, using sqlite3 Cursor class. Still leaking. So doesn’t appear to be the Cursor subclass. Before looping 100000 times: mem_size: 10.472656 mem_rss: 6.000000 mem_share: 3.953125 After loop:: mem_size: 15.535156 mem_rss: 11.937500 mem_share: 4.746094 Garbage collected 0 objects After garbage collection:: mem_size: 15.535156 mem_rss: 11.937500 mem_share: 4.746094 Change the Connection class to subclass Sqlite Connection class and override the cursor method to use a factory to get an instance of the cursor. This looks promising, but we still need a wrapper of the connection to fit into the WeeWX architecture to support multiple database implementations (Sqlite, MySQL, etc) Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.972656 mem_share: 3.925781 After loop:: mem_size: 10.597656 mem_rss: 6.906250 mem_share: 4.738281 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.906250 mem_share: 4.738281 Create a subclass of the Sqlite Connection class that overrides the cursor method. Have the Connection wrapper instantiate this instead of the SQlite Connection class. This would fit into WeeWx quite easily. Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.996094 mem_share: 3.949219 After loop:: mem_size: 10.597656 mem_rss: 6.937500 mem_share: 4.769531 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.937500 mem_share: 4.769531 Subclass the SQLite Connection class and also act as a wrapper. This would also fit into WeeWx using multiple inheritance in the Connection class. Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.984375 mem_share: 3.937500 After loop:: mem_size: 10.597656 mem_rss: 6.914062 mem_share: 4.746094 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.914062 mem_share: 4.746094 The shim class to provide a cursor via a factory seems the best to me. The next three are just upping the loop size to see if anything shows up. Looks like a loop of 100,000 is reasonable to show possible issues. Before looping 100000 times: mem_size: 10.472656 mem_rss: 5.503906 mem_share: 3.601562 After loop:: mem_size: 10.597656 mem_rss: 6.878906 mem_share: 4.710938 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.878906 mem_share: 4.710938 Before looping 200000 times: mem_size: 10.472656 mem_rss: 6.003906 mem_share: 3.957031 After loop:: mem_size: 10.597656 mem_rss: 6.949219 mem_share: 4.781250 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.949219 mem_share: 4.781250 Before looping 2500000 times: mem_size: 10.472656 mem_rss: 5.496094 mem_share: 3.593750 After loop:: mem_size: 10.597656 mem_rss: 6.871094 mem_share: 4.703125 Garbage collected 0 objects After garbage collection:: mem_size: 10.597656 mem_rss: 6.871094 mem_share: 4.703125 Next up is another program that uses the weedb APIs. The first is 3.9.1 as is. The second is with an updated sqlite.py with the shim class. Each is using a loop of 100,000. ./amemtest2.py mem_size: 11.859375 mem_rss: 7.367188 mem_share: 4.089844 mem_size: 16.804688 mem_rss: 13.269531 mem_share: 4.832031 Garbage collected 0 objects mem_size: 16.804688 mem_rss: 13.269531 mem_share: 4.832031 ./amemtest2.py mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719 mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719 Garbage collected 0 objects mem_size: 11.609375 mem_rss: 7.410156 mem_share: 4.136719 The update to sqlite.py is about 10 LOC. I want to run it in my ‘production’ environment for a few days. If all seems to be behaving, I will submit a pull request. - Rich -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/13e95fb7-f57e-4fa8-b002-776cdcdeae2e%40googlegroups.com.
