[Python-Dev] Possible error in Python 2.4 garbage collector
Hi, we're running into a problem with the Python interpreter, which we suspect triggers a bug in the garbage collector or a related part of the memory management. We are using Python 2.4.4 in the version from Debian 4.0. Our C application imports various Python plugins using PyNode_Compile(). This used to work very well (it's a feature we've been using for years now), but if we import more than 23 modules we're running a segmentation fault with the backtrace below: (gdb) bt #0 0xb7ee9cad in visit_decref (op=0xb7287ec0, data=0x0) at ../Modules/gcmodule.c:269 #1 0xb7e716d1 in list_traverse (o=0xb724e62c, visit=0xb7ee9ca0 visit_decref, arg=0x0) at ../Objects/listobject.c:2280 #2 0xb7eea729 in collect (generation=0) at ../Modules/gcmodule.c:294 #3 0xb7eeb0a3 in _PyObject_GC_Malloc (basicsize=20) at ../Modules/gcmodule.c:872 #4 0xb7eeb0eb in _PyObject_GC_NewVar (tp=0xb7f25720, nitems=2) at ../Modules/gcmodule.c:1283 #5 0xb7e913e4 in PyTuple_New (size=2) at ../Objects/tupleobject.c:68 #6 0xb7e91bb5 in PyTuple_Pack (n=2) at ../Objects/tupleobject.c:143 #7 0xb7ebfded in com_add (c=0xbfc5d3ac, list=0x0, dict=0xb7ee9ca0, v=0xb770fc20) at ../Python/compile.c:1375 #8 0xb7ec4b1e in com_addop_varname (c=0xbfc5d3ac, kind=0, name=0xb71f9218 new) at ../Python/compile.c:1410 #9 0xb7ecea53 in com_atom (c=0xbfc5d3ac, n=0xb71f8ca0) at ../Python/compile.c:2213 #10 0xb7ecefe0 in com_power (c=0x70732f72, n=0xb71fb5c0) at ../Python/compile.c:2587 #11 0xb7ecf4c3 in com_term (c=0x70732f72, n=0xb71fb590) at ../Python/compile.c:2727 #12 0xb7ecf639 in com_shift_expr (c=0xbfc5d3ac, n=0xb71fb560) at ../Python/compile.c:2762 #13 0xb7ecf924 in com_xor_expr (c=0x70732f72, n=0xb71fb530) at ../Python/compile.c:2814 #14 0xb7ec5064 in com_comparison (c=0x70732f72, n=0xb71fb2a8) at ../Python/compile.c:2858 #15 0xb7ec5e5a in com_test (c=0xbfc5d3ac, n=0xb71f8714) at ../Python/compile.c:2986 #16 0xb7eca88a in com_if_stmt (c=0xbfc5d3ac, n=0xb71fb278) at ../Python/compile.c:3844 #17 0xb7ec9101 in com_node (c=0xbfc5d3ac, n=0xb71f4d2c) at ../Python/compile.c:4181 #18 0xb7eca926 in com_if_stmt (c=0xbfc5d3ac, n=0xb71f7398) at ../Python/compile.c:3848 #19 0xb7ec9101 in com_node (c=0xbfc5d3ac, n=0xb720a688) at ../Python/compile.c:4181 #20 0xb7ecd1af in com_try_except (c=0xbfc5d3ac, n=0xb71f5728) at ../Python/compile.c:3998 #21 0xb7ec9101 in com_node (c=0xbfc5d3ac, n=0xb722ecdc) at ../Python/compile.c:4181 #22 0xb7eca926 in com_if_stmt (c=0xbfc5d3ac, n=0xb71dcb60) at ../Python/compile.c:3848 #23 0xb7ec9101 in com_node (c=0xbfc5d3ac, n=0xb71d8318) at ../Python/compile.c:4181 #24 0xb7ecac18 in com_if_stmt (c=0xbfc5d3ac, n=0xb71d4110) at ../Python/compile.c:3855 #25 0xb7ec9101 in com_node (c=0xbfc5d3ac, n=0xb737c610) at ../Python/compile.c:4181 #26 0xb7ec6d4c in compile_node (c=0xbfc5d3ac, n=value optimized out) at ../Python/compile.c:4755 #27 0xb7ec7e80 in jcompile (n=0xb71d4098, filename=value optimized out, base=0xbfc5d72c, flags=0x0) at ../Python/compile.c:5006 #28 0xb7ec8776 in com_funcdef (c=0xbfc5d72c, n=0xb71d4098) at ../Python/compile.c:4942 #29 0xb7ec6c8e in compile_node (c=0xbfc5d72c, n=value optimized out) at ../Python/compile.c:4728 #30 0xb7ec7e80 in jcompile (n=0xb7341668, filename=value optimized out, base=0x0, flags=0x0) at ../Python/compile.c:5006 #31 0xb7ec86ca in PyNode_Compile (n=0xb71a58f0, filename=0x82d5aa0 /usr/lib/univention-directory-listener/system/cyrus-sieve.py) at ../Python/compile.c:4913 #32 0x0804e5de in module_import (filename=0x82d5aa0 /usr/lib/univention-directory-listener/system/cyrus-sieve.py) at handlers.c:98 #33 0x0804ec6e in handler_import (filename=0x82d5aa0 /usr/lib/univention-directory-listener/system/cyrus-sieve.py) at handlers.c:184 #34 0x0804fb37 in handlers_load_path (path=0x8062130 /usr/lib/univention-directory-listener/system) at handlers.c:472 #35 0x0804fbc0 in handlers_load_all_paths () at handlers.c:491 #36 0x0804fff0 in handlers_init () at handlers.c:568 #37 0x0804bfb9 in main (argc=16, argv=0xbfc5dce4) at main.c:489 (gdb) Given the function names I would suspect a bug in the garbage collector? I'm not familiar with the internals of Python's memory managent, so I can't really tell for sure. It could also be that some default internal limits of the memory manager are triggered. Shall I open a bug in the tracker for this does anyone suspect a different cause or even a workaround? Cheers, Moritz -- Moritz Muehlenhoff [EMAIL PROTECTED] fon: +49 421 22 232- 0 DevelopmentLinux for Your Business fax: +49 421 22 232-99 Univention GmbHhttp://www.univention.de/ mobil: +49 175 22 999 23 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible error in Python 2.4 garbage collector
This is almost certainly not a bug in python. A cursory look indicates that a list being traversed in list_traverse has a NULL member. I'd suggest examining the other members of the list to figure out what this list is. Use the debugger for this, that is what it is for. It is probably a list you have created in your C code but forgot to check for exceptions when adding members to it, thus leaving one of them with a NULL pointer. see for example http://wiki.python.org/moin/DebuggingWithGdb for tips on debugging. K -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Moritz Mühlenhoff Sent: Monday, November 26, 2007 10:58 To: python-dev@python.org; [EMAIL PROTECTED] Subject: [Python-Dev] Possible error in Python 2.4 garbage collector Hi, we're running into a problem with the Python interpreter, which we suspect triggers a bug in the garbage collector or a related part of the memory management. We are using Python 2.4.4 in the version from Debian 4.0. Our C application imports various Python plugins using PyNode_Compile(). This used to work very well (it's a feature we've been using for years now), but if we import more than 23 modules we're running a segmentation fault with the backtrace below: (gdb) bt #0 0xb7ee9cad in visit_decref (op=0xb7287ec0, data=0x0) at ../Modules/gcmodule.c:269 #1 0xb7e716d1 in list_traverse (o=0xb724e62c, visit=0xb7ee9ca0 visit_decref, arg=0x0) at ../Objects/listobject.c:2280 #2 0xb7eea729 in collect (generation=0) at ../Modules/gcmodule.c:294 #3 0xb7eeb0a3 in _PyObject_GC_Malloc (basicsize=20) at ../Modules/gcmodule.c:872 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Build Notes for building trunk with Visual Studio 2008 Express Edition
On Nov 23, 2007 8:53 AM, Paul Moore [EMAIL PROTECTED] wrote: I have just built the current trunk version of Python on Windows, using the new PCBuild9 directory, and Visual Studio 2008 Express Edition. Everything went extremely well. I include below my notes on what I did, for reference. To be honest, there's nothing in here that really warrants a change to the readme.txt - to all intents and purposes, the process just works. OK, here are my notes: Install Visual C++ 2008 Express Edition. Only select Silverlight from the options (no documentation or SQL Server - Silverlight probably isn't actually needed either). I already had the Platform SDK installed, but did nothing to tell VS about it, or to integrate it. I doubt this is relevant. I am using Python trunk at revision 59132. (But I hit an issue fixed in 59136, so better to use that). Start VC. Open project PCBuild9\pcbuild.sln Message Solution Folders are not supported in this version of Visual Studio. Solution folder 'Solution Items' will be displayed as unavailable. Select OK. Select the release build (Build Configuration Manager) Right click pythoncore Build make_buildinfo - succeeded 1 warning (unlink vs _unlink) make_versioninfo - succeeded pythoncore - I hit an error in ast.c so I needed to svn up (to 59136). Succeeded. Right click python Build. Succeeded. Right click pythonw Build. Succeeded. Right click _socket Build. Succeeded. Right click _testcapi Build. Succeeded. Right click pyexpat Build. Succeeded. Right click select Build. Succeeded. Right click unicodedata Build. Succeeded. Right click winsound Build. Succeeded. At this point, we have finished the modules documented as build out of the box in PCBuild9\readme.txt. The modules _tkinter, bz2, _bsddb, _sqlite3, _ssl are documented as having dependencies. See below. Modules _ctypes, _ctypes_test, _elementtree, _msi, w9xpopen are not mentioned in readme.txt. They all build without error. bz2 --- The include dir is a macro, and I can't work out how to override the default (which is bzip2-1.0.3). So I stuck to 1.0.3 and exported it from Python svn, as per the instructions. Built OK. _sqlite3 Again, couldn't work out how to change the macro, so I stuck with the external from svn (sqlite-source-3.3.4). The pre-link step failed with an error about not finding TCL. I edited the prelink step to include a /DNO_TCL flag on the cl command. There may be a better approach - I don't know if not having TCL is an issue for Python's use of sqlite. _tkinter and _bsddb --- The instructions suggest using VS 2003 to build the dependencies. I don't have VS 2003 and don't have the time at the moment to investigate further. _ssl Christian has been making changes to allow this to build without Perl, so I gave it a try. I used openssl 0.9.8g, which I extracted to the build directory (I noticed afterwards that this is the same version as in Python svn, so I could have used the svn external!) I needed to download nasm (nasm.sf.net) version 2.00rc1, and rename nasm.exe to nasmw.exe and put it on my PATH. Build succeeded, no issues. Tests - Running the tests, all succeed except test_tcl and test_bsddb, which are because I didn't build those two extensions, and test_os. The test_os failure is just because it looks for C:\pagefile.sys and my pagefile is on D:\. (There's also a problem with test_socket_ssl hanging, but this is present in the standard 2.6 snapshot build. I've raised a separate bug report for this). I was able to build both 2.6 and py3k (trunk checkouts, revision 59136 and 59130 respectively) out of the box, except for openssl and tcl, which I didn't try to install. The external libraries I placed as sibling directories to the python source root dir, just as with the previous PCBuild instructions. I didn't need to build in any specific order, Build solution correctly resolved dependencies. I got the alert box about solution folders but it didn't harm anything. If anyone is curious, I ran pybench for both 2.6 and 3k build against VS 2003 and VS 2008. This is using the out of the box settings, and no PGO for 2008 (Express version doesn't have it). MSVC 9 was slightly faster for 2.6, but somewhat slower for py3k. I'm not sure how relevant that is, but (surprisingly to me) it's not an automatic speed win over MSVC 7.1. The results of my pybench runs available here: http://code.google.com/p/wxpsvg/wiki/PyBenchRuns ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible error in Python 2.4 garbage collector
Kristján Valur Jónsson wrote: This is almost certainly not a bug in python. A cursory look indicates that a list being traversed in list_traverse has a NULL member. I'd suggest examining the other members of the list to figure out what this list is. Use the debugger for this, that is what it is for. It is probably a list you have created in your C code but forgot to check for exceptions when adding members to it, thus leaving one of them with a NULL pointer. Thank you for your comment. A longer debugging session showed a stray Py_DECREF in an unexpected code path, which lead to the error above. The Python GC is indeed fine. Thanks, Moritz -- Moritz Muehlenhoff [EMAIL PROTECTED] fon: +49 421 22 232- 0 DevelopmentLinux for Your Business fax: +49 421 22 232-99 Univention GmbHhttp://www.univention.de/ mobil: +49 175 22 999 23___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Build Notes for building trunk with Visual Studio 2008 Express Edition
Chris Mellon wrote: If anyone is curious, I ran pybench for both 2.6 and 3k build against VS 2003 and VS 2008. This is using the out of the box settings, and no PGO for 2008 (Express version doesn't have it). MSVC 9 was slightly faster for 2.6, but somewhat slower for py3k. I'm not sure how relevant that is, but (surprisingly to me) it's not an automatic speed win over MSVC 7.1. The results of my pybench runs available here: I haven't enabled all optimization flags yet. We may get some speed ups by enable more flags. I leave that to the optimization experts. I'm glad that everybody is happy with the new PCbuild9 directory. Tcl/Tk is the last obstacle. I'm not able to build the 64bit version with the cross compiler of VS 2008 Standard Edition. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssl module integration with asyncore
Hi there, since ssl module is still in development I thought it would have been better asking such question here instead of on comp.lang.python. I'm interested in using the ssl module with asyncore but since there's no real documentation about it yet I've been not able to write something useful with it. http://docs.python.org/dev/library/ssl.html -- it's not quite up-to-date, but close. Currently I'm trying to add the SSL support to an asyncore-based FTP server I wrote. I tried to write some code by watching the ssl-related test-suite scripts with no luck since there are no available tests for asyncore available. Take a look at the SSL 1.12 test suite -- there's an asyncore-based server in there. I tried to play with the wrap_socket() function a little bit but different type of errors are raised every time. I'm sure this happens because I'm not doing things in the right way. Could someone please show me an example code about how the ssl module could be integrated with asyncore? Hope that helps. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Build Notes for building trunk with Visual Studio 2008 Express Edition
I'm glad that everybody is happy with the new PCbuild9 directory. Tcl/Tk is the last obstacle. I'm not able to build the 64bit version with the cross compiler of VS 2008 Standard Edition. I'll look into it. I'll probably try to build some beta of Tcl 8.5, hoping that they manage to release Tcl/Tk 8.5 before Python gets released. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssl module integration with asyncore
On 26 Nov, 19:23, Bill Janssen [EMAIL PROTECTED] wrote: Hi there, since ssl module is still in development I thought it would have been better asking such question here instead of on comp.lang.python. I'm interested in using the ssl module with asyncore but since there's no real documentation about it yet I've been not able to write something useful with it. http://docs.python.org/dev/library/ssl.html-- it's not quite up-to-date, but close. Currently I'm trying to add the SSL support to an asyncore-based FTP server I wrote. I tried to write some code by watching the ssl-related test-suite scripts with no luck since there are no available tests for asyncore available. Take a look at the SSL 1.12 test suite -- there's an asyncore-based server in there. I downloaded this one: http://pypi.python.org/pypi/ssl/1.12 ...which seems to contain the same test-suite used in the current Python 2.6 distribution available here: http://svn.python.org/snapshots/ I looked into test/test_ssl.py but I didn't find any test referencing to the asyncore module. I see a class named AsyncoreHTTPSServer but it does not use the asyncore or asynchat modules. Or maybe... am I'm looking in the wrong place? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssl module integration with asyncore
I downloaded this one: http://pypi.python.org/pypi/ssl/1.12 Yes, that's the one. ...which seems to contain the same test-suite used in the current Not quite. Python 2.6 distribution available here: http://svn.python.org/snapshots/ I looked into test/test_ssl.py but I didn't find any test referencing to the asyncore module. I see a class named AsyncoreHTTPSServer but it does not use the asyncore or asynchat modules. Yes, that's right, it uses SocketServer. But that's very close to the same thing. I'll change the name to make it clear. Really, the only interesting part is the get_request method, which overrides the get_request method from SocketServer.TCPServer, and is very similar to the asyncore.dispatcher accept method. You may also need to override the readable method on asyncore.dispatcher: def readable(self): if isinstance(self.socket, ssl.SSLSocket): # dispatch any bytes left in the SSL buffer while self.socket.pending() 0: self.handle_read_event() return True Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
