[issue1469] SSL tests leak memory

Bill Janssen added the comment: The server isn't handling the close event properly. I'll fix that. On Dec 13, 2007 9:06 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Guido van Rossum added the comment: I spoke too soon. In a debug build, this hangs forever during the second iteration:

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Added file: http://bugs.python.org/file8955/patch-4 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469

[issue1469] SSL tests leak memory

Changes by Bill Janssen: Removed file: http://bugs.python.org/file8949/patch-3 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Changes by Bill Janssen: Removed file: http://bugs.python.org/file8954/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Perfect! Check it in. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Done. On Dec 14, 2007 9:44 AM, Guido van Rossum [EMAIL PROTECTED] wrote: Guido van Rossum added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Perfect! Check it in.

[issue1469] SSL tests leak memory

Changes by Guido van Rossum: -- resolution: - fixed status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: Hm, when I run the full test_ssl test suite with ./python Lib/test/regrtest.py -v -uall test_ssl it always hangs in testAsyncoreServer after printing this: testAsyncoreServer (test.test_ssl.ThreadedTests) ... server: new connection from 127.0.0.1:59781

[issue1469] SSL tests leak memory

Christian Heimes added the comment: It hangs on my Linux box, too. I've not yet tried on Windows. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Good catch. I flipped a bit cleaning up the C code. Here's a fixed patch. Added file: http://bugs.python.org/file8949/patch-3 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: Good catch. I flipped a bit cleaning up the C code. Here's a fixed patch. Added file: http://bugs.python.org/file8949/patch-3 Great! Go ahead and check it in. Sorry for deleting the _real_close() earlier BTW. Enjoy your time away!

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: I spoke too soon. In a debug build, this hangs forever during the second iteration: ./python.exe Lib/test/regrtest.py -uall -R1:1 test_ssl Adding -v, it looks like two iterations are carried out perfectly (one must be a trial run, one the warm-up run), but

[issue1469] SSL tests leak memory

Changes by Giampaolo Rodola': -- nosy: +giampaolo.rodola __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe:

[issue1469] SSL tests leak memory

Bill Janssen added the comment: I think I've figured it out. My initial patch to socket.py and ssl.py had an extra method defined on socket.socket, _real_close(), which did' the cleanup work of deallocating the underlying socket, and in the SSL subclass, of releasing the SSL context. Guido

[issue1469] SSL tests leak memory

Bill Janssen added the comment: So, what's the final status of __del__ in py3K? The other bit of leak is due to _real_close() not being called when a socket is dropped on the floor (say, you try to connect, fail, and raise the exception back to the caller, without ever explicitly calling

[issue1469] SSL tests leak memory

Amaury Forgeot d'Arc added the comment: Is it possible that you have the same problem as in http://bugs.python.org/issue1540 ? To be sure, check the gc.garbage variable at the end of the test. It may contain the references you are looking for... -- nosy: +amaury.forgeotdarc

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: So, what's the final status of __del__ in py3K? The other bit of leak is due to _real_close() not being called when a socket is dropped on the floor (say, you try to connect, fail, and raise the exception back to the caller, without ever explicitly

[issue1469] SSL tests leak memory

Bill Janssen added the comment: The other leak comes from this code: s = ssl.wrap_socket(socket.socket(), ...) s.connect((SOME BOGUS ADDRESS OR SERVER)) The connect() fails, and the SSLSocket sgets dropped on the floor, but never seems to be GC'd, (or the GC never seems to call the

[issue1469] SSL tests leak memory

Bill Janssen added the comment: gc.garbage is always empty. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe:

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: I wonder if Christian Heimes was correct that the ssl object needs GC support? This was part of his patch (which I checked in and then reverted because the other part of it didn't work as advertised). Alternatively, if 's' is involved in a cycle *and* any of

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Ah, I see what's going on. The revision of the socket code (nice job, by the way) removed the distinction between the C socket object and the Python socket object. The C SSLContext keeps a pointer to the C socket object, which is now the Python socket object, or

[issue1469] SSL tests leak memory

Christian Heimes added the comment: Guido van Rossum wrote: Guido van Rossum added the comment: I wonder if Christian Heimes was correct that the ssl object needs GC support? This was part of his patch (which I checked in and then reverted because the other part of it didn't work as

[issue1469] SSL tests leak memory

Bill Janssen added the comment: I think Christian analysis is right, in that it takes a bit of GC support, but not perhaps in the specifics of his approach. I've done two things to fix this: 1) Put _real_close() back in socket.py, and then override it in ssl.SSLSocket to release the

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Here's a patch -- take a look and let me know. I also added a real asyncore server test. Added file: http://bugs.python.org/file8919/a __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469

[issue1469] SSL tests leak memory

Bill Janssen added the comment: Now, about the confused signature of SSLSocket.read(). I'm not sure how confused it is. It's sui generis; SSLSocket doesn't inherit from some other class with a different read() method with a different signature. But we could change the name, perhaps to

[issue1469] SSL tests leak memory

Changes by Bill Janssen: Removed file: http://bugs.python.org/file8919/a __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe:

[issue1469] SSL tests leak memory

Changes by Guido van Rossum: Removed file: http://bugs.python.org/file8892/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: Don't worry, I did back it out before releasing 3.0a2. I believe I hacked your code a bit before checking it in (or after you checked it in, can't remember). Did you see my bug report with a TypeError? __ Tracker [EMAIL

[issue1469] SSL tests leak memory

Changes by Guido van Rossum: Removed file: http://bugs.python.org/file8845/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Bill Janssen added the comment: I'm still not sure what the problem is. This code was working fine when I committed it, no leaks, so something else must have changed under the covers. I don't believe that adding GC to the C code is the answer; it should be automatically GC'd. So I'd back that

[issue1469] SSL tests leak memory

Christian Heimes added the comment: I can reproduce the problem when I run the ssl tests with -unetwork. I've used PYTHONDUMREFS and combinerefs.py to find the leaking objects. I'm seeing lots of 0x9fd3e4c [1] SSLSocket 0x9fd2928 [1] ssl.SSLContext ssl.SSLContext object at 0x9fd2928 0x9fd5b74

[issue1469] SSL tests leak memory

Christian Heimes added the comment: I may have found the error. The PySSL_Type doesn't support GC but it should in order to clean up self-Socket. I've started to fix it but I don't have time to work on the problem 'til tonight. The patch causes a seg fault. I may have overlooked or

[issue1469] SSL tests leak memory

Christian Heimes added the comment: I found the problem! I've to use PyObject_GC_New instead of PyObject_New __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: I get a segfault when I run it like this: $ ./python.exe Lib/test/regrtest.py -uall test_ssl test_ssl Segmentation fault $ (Without -uall it passes.) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469

[issue1469] SSL tests leak memory

Changes by Christian Heimes: Removed file: http://bugs.python.org/file8884/ssl_gc.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Christian Heimes added the comment: New patch The new patch doesn't cause a seg fault but it doesn't help. It *increases* the number of reference leaks. test_ssl leaked [1610, 1610] references, sum=3220 Added file: http://bugs.python.org/file8885/ssl_gc.patch

[issue1469] SSL tests leak memory

Changes by Christian Heimes: Removed file: http://bugs.python.org/file8885/ssl_gc.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list

[issue1469] SSL tests leak memory

Christian Heimes added the comment: The new patch works and fixes the ref leak. I had to move the call self._real_close() from __del__ to PySSL_dealloc(). I don't know if you are going to like it. :] Added file: http://bugs.python.org/file8886/ssl_gc.patch __

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: I will look at this in an hour or so, after I bring Orlijn to school and drive to work. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __

[issue1469] SSL tests leak memory

Christian Heimes added the comment: Guido van Rossum wrote: I will look at this in an hour or so, after I bring Orlijn to school and drive to work. I found two problems with the _real_close() call in PySSL_dealloc. I'm digging into it now. o = PyObject_CallMethod((PyObject*)self-Socket,

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: Oops, I just committed revision 59394. Please advise. -- priority: high - immediate __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __

[issue1469] SSL tests leak memory

Guido van Rossum added the comment: I'm giving up on this for now. There are other weird bugs in the code, e.g. this simple piece of code fails: x = urllib.urlopen(https://mail.google.com;).read() Traceback (most recent call last): File stdin, line 1, in module File

[issue1469] SSL tests leak memory

Christian Heimes added the comment: Guido van Rossum wrote: I just reverted it. I put a bit of debugging in the call to _real_close(), and even with Christian's corrections it fails miserably. I prefer leaking. I agree! I've not enough time to work on the patch. A working patch requires some

[issue1469] SSL tests leak memory

Christian Heimes added the comment: Are you sure that the SSL tests are still leaking memory? The tests aren't leaking references on Windows nor on Linux. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __

[issue1469] SSL tests leak memory

Bill Janssen added the comment: I'm seeing the leaks on my Leopard machine. I haven't had a chance to look into it yet. On Dec 1, 2007 11:20 AM, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: Are you sure that the SSL tests are still leaking memory? The

[issue1469] SSL tests leak memory

Bill Janssen added the comment: What platform? Bill On Nov 20, 2007 11:21 PM, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: I don't see leaks when I run the tests with $ ./python Lib/test/regrtest.py -R:: test_ssl.py test_ssl beginning 9 repetitions

[issue1469] SSL tests leak memory

Christian Heimes added the comment: Ubuntu Linux x86 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe:

[issue1469] SSL tests leak memory

Christian Heimes added the comment: I don't see leaks when I run the tests with $ ./python Lib/test/regrtest.py -R:: test_ssl.py test_ssl beginning 9 repetitions 123456789 . 1 test OK. [80034 refs] -- nosy: +tiran priority: - normal __ Tracker

[issue1469] SSL tests leak memory

New submission from Bill Janssen: I'm seeing leaks in the test_ssl run, after various socket.py changes. I'm looking into it. -- assignee: janssen components: Library (Lib) keywords: py3k messages: 57667 nosy: janssen severity: normal status: open title: SSL tests leak memory versions: