Bugs item #1413192, was opened at 2006-01-23 12:35 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1413192&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Alex Roitman (rshura) Assigned to: Neal Norwitz (nnorwitz) Summary: bsddb: segfault on db.associate call with Txn and large data Initial Comment: Problem confirmed on Python2.3.5/bsddb4.2.0.5 and Python2.4.2/bsddb4.3.0 on Debian sid and Ubuntu Breezy. It appears, that the associate call, necessary to create a secondary index, segfaults when: 1. There is a large amount of data 2. Environment is transactional. The http://www.gramps-project.org/files/bsddb/testcase.tar.gz contains the example code and two databases, pm.db and pm_ok.db -- both have the same number of keys and each data item is a pickled tuple with two elements. The second index is created over the unpickled data[1]. The pm.db segfaults and the pm_ok.db does not. The second db has much smaller data items in data[0]. If the environment is set up and opened without TXN then pm.db is also fine. Seems like a problem in associate call in a TXN environment, that is only seen with large enough data. Please let me know if I can be of further assistance. This is a show-stopper issue for me, I would go out of my way to help resolving this or finding a work-around. Thanks! Alex P.S. I could not attach the large file, probably due to the size limit on the upload, hence a link to the testcase. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2006-01-24 11:40 Message: Logged In: YES user_id=33168 Could you pull the version of Modules/_bsddb.c out of SVN and then apply my patch? I believe your new problem was fixed recently. You can look in the Misc/NEWS file to find the exact patch. ---------------------------------------------------------------------- Comment By: Alex Roitman (rshura) Date: 2006-01-24 11:37 Message: Logged In: YES user_id=498357 Done same tests on another Debian sid machine, exact same results (up to one line number, due to my extra fprintf statement): (gdb) run test2.py Starting program: /usr/bin/python2.4-dbg test2.py [Thread debugging using libthread_db enabled] [New Thread -1210390848 (LWP 5865)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1210390848 (LWP 5865)] 0xb7b01eb4 in DB_associate (self=0xb7d63df0, args=0xb7d67234, kwargs=0xb7d5ee94) at /home/shura/src/python2.4-2.4.2/Modules/_bsddb.c:1218 1218 Py_DECREF(self->associateCallback); (gdb) ---------------------------------------------------------------------- Comment By: Alex Roitman (rshura) Date: 2006-01-24 11:31 Message: Logged In: YES user_id=498357 OK, built and installed all kinds of python packages with the patch. All tests are fine. Here goes: 1. Your testcase goes just fine, no segfault with the patched version. 2. Mine still segfaults. 3. I ran mine under gdb with python2.4-dbg package, here's the output (printed "shurafine" is my addition, to make sure that the correct code is being run): $ gdb python2.4-dbg GNU gdb 6.4-debian Copyright 2005 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i486-linux-gnu"...Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1". (gdb) run test2.py Starting program: /usr/bin/python2.4-dbg test2.py [Thread debugging using libthread_db enabled] [New Thread -1210038592 (LWP 29629)] shurafine Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1210038592 (LWP 29629)] 0xb7b57f3e in DB_associate (self=0xb7db9f58, args=0xb7dbd3b4, kwargs=0xb7db5e94) at /home/shura/src/python2.4-2.4.2/Modules/_bsddb.c:1219 1219 Py_DECREF(self->associateCallback); (gdb) Please let me know if I can be of further assistance. ---------------------------------------------------------------------- Comment By: Alex Roitman (rshura) Date: 2006-01-24 10:50 Message: Logged In: YES user_id=498357 Thanks for a quick response! OK, first thing first: your simpler testcase seems to expose yet another problem, not the one I had. In particular, your testcase segfaults for me on python2.4.2/bsddb4.3.0 but *does not* segfault with python2.3.5/bsddb4.2.0.5 In my testcase, I can definitely blame the segfault on the associate call, not on open. I can demonstrate it by either commenting out the associate call (no segfault) or by inserting a print statement right before the associate. So your testcase does not seem to have an exact same problem than my testcase. In my testcase nothing seems to depend on variable names (as one would expect). I am rebuilding python2.4 with your patch, will post results soon. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-01-23 23:03 Message: Logged In: YES user_id=33168 I spoke too soon. The attached patch works for me or your original test case and my pared down version. It also passes the tests. It also fixes a potential memory leak. Let me know if this works for you. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-01-23 22:45 Message: Logged In: YES user_id=33168 I've got a much simpler test case. The problem seems to be triggered when the txn is deleted after the env (in Modules/_bsddb.c 917 vs 966). If I change the variable names in python, I don't get the same behaviour (ie, it doesn't crash). I removed the original data file, but if you change the_txn to txn, that might "fix" the problem. If not, try playing with different variable names and see if you can get it to not crash. Obviously there needs to be a real fix in C code, but I'm not sure what needs to happen. It doesn't look like we keep enough info to do this properly. ---------------------------------------------------------------------- Comment By: Alex Roitman (rshura) Date: 2006-01-23 12:41 Message: Logged In: YES user_id=498357 Attaching test3.py containing same code without transactions. Works fine with either pm.db or pm_ok.db ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1413192&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
