Re: [Zope] Frequent ZOPE crashes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Krasa wrote: Hi Tres, thank you very much for your reply! Am 29.11.09 21:57, schrieb Tres Seaver: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at we're right in the process of tracking down the error outside of ZOPE. We have completely installed a new server from scratch with RHEL 5.4 and have re-installed python 2.4.6 and the latest versions of libxml2 and libxslt there. We double checked the LD config, and made sure that te correct shared objects get loaded (via lsof). We also reinstalled a few other modules that contain C-code (such as python-ldap) which we need for being able to do authenitcation. Unfortunately that didn't really help much. We still experience crashes. Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that could cause these problems? The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). We're unfortunately still not able to reproduce these crashes. Can you set 'ulimit -c' to get a core file, which might at least help point to the extension which is to blame (although it may just show the downstream victim of a heap munge). What versions of libxml2 / libxslt are you using? How about lxml? Yes, we did set the ulimit and were indeed able to produce a coredump for each crash happening (each having something between 300 and 700 MB). We tried to debug using gdb but unfortunaley they only reveal two cases when the crashes occur: 1) During garbage collection where the gc tries to clean up damaged python objects 2) During some ceval process, also related to accessing damaged python objects Unfortunately it doesn't reveal what exactly trashes the objects. To us it seems that this could happen some time earlier before either of the two processes mentioned above tries to access the objects and crashes ZOPE. For now, we don't really see a reproduceable pattern as it seems to be a somewhat more complex user behavior which leads to this. We were able to extract a few URLs out of the coredumps but directly accessing those does nothing. Also the last logged access in the Z2.log before the coredump triggers nothing, when directly accessing it. We're running ZOPE-2.11.2 with an eggified version of ZODB3-3.8.4 plus libxml2-2.7.6, libxslt-1.1.26 and lxml-2.2.4 now, the crashes still happen. Previously we've been running with ZOPE-2.11.2, libxml2-2.7.3, libxslt-1.1.24 and lxml-2.1.5. That also crashed ZOPE occasionally. Does your application ever use the libxml2 / libxslt Python bindings directly? If so, I would go over that part of your app with a microscope: it is incredibly easy to trigger segfaults from those bindings. If not, then I would look for help on the lxml mailing list. This only happened since we switched to a new layout (probably in combination with a few minor Silva updates). By new layout, to you mean a new site them? If so, how do lxml / libxml2 / lbixslt interact with your application to generate the theme? What is structurally different about the new theme? We have been using the same system software (RHEL5), hardware, python version and libxml2/libxslt/lxml versions with our old old layout, where everything worked fine for years. I would be happy to paste any particular gdb outputs if that is of any help...? I'm afraid that won't help: the GC segfaults indicate somebody is munging the heap way before the segfault is triggered. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksT65cACgkQ+gerLs4ltQ5swACgsSuScLIAfFtd1d9TMznaQEeu 7JEAoJBetJHX3KOCbinGlyV5F/7DWjqK =qGv5 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Am 25.11.09 17:37, schrieb Jaroslav Lukesh: At first, try to eliminate error outside of the Zope itself. Try to install it all into plain whole new (and reliable!) machine. Do not use restore of any backups! - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at A week ago we switched to a new layout (for corporate reasons) and now we're experiencing frequent crashes of the Zope servers. Fortunately Hi Jaroslav, we're right in the process of tracking down the error outside of ZOPE. We have completely installed a new server from scratch with RHEL 5.4 and have re-installed python 2.4.6 and the latest versions of libxml2 and libxslt there. We double checked the LD config, and made sure that te correct shared objects get loaded (via lsof). We also reinstalled a few other modules that contain C-code (such as python-ldap) which we need for being able to do authenitcation. Unfortunately that didn't really help much. We still experience crashes. Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that could cause these problems? The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). We're unfortunately still not able to reproduce these crashes. Kind regards, Andreas ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Try to cache page segments selectively and then you should identify problematic piece of page. If You does not use page segments, then divide page to some pieces and get them together. - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Krasa wrote: Am 25.11.09 17:37, schrieb Jaroslav Lukesh: At first, try to eliminate error outside of the Zope itself. Try to install it all into plain whole new (and reliable!) machine. Do not use restore of any backups! - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at A week ago we switched to a new layout (for corporate reasons) and now we're experiencing frequent crashes of the Zope servers. Fortunately Hi Jaroslav, we're right in the process of tracking down the error outside of ZOPE. We have completely installed a new server from scratch with RHEL 5.4 and have re-installed python 2.4.6 and the latest versions of libxml2 and libxslt there. We double checked the LD config, and made sure that te correct shared objects get loaded (via lsof). We also reinstalled a few other modules that contain C-code (such as python-ldap) which we need for being able to do authenitcation. Unfortunately that didn't really help much. We still experience crashes. Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that could cause these problems? The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). We're unfortunately still not able to reproduce these crashes. Can you set 'ulimit -c' to get a core file, which might at least help point to the extension which is to blame (although it may just show the downstream victim of a heap munge). What versions of libxml2 / libxslt are you using? How about lxml? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksS4FEACgkQ+gerLs4ltQ4d+ACeJom2eaL9cKTU5qq0zYd+4wjT syMAoI+2rPjBOJtz/HlQnbQgvDEghfGP =gOdT -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Hi Tres, thank you very much for your reply! Am 29.11.09 21:57, schrieb Tres Seaver: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at we're right in the process of tracking down the error outside of ZOPE. We have completely installed a new server from scratch with RHEL 5.4 and have re-installed python 2.4.6 and the latest versions of libxml2 and libxslt there. We double checked the LD config, and made sure that te correct shared objects get loaded (via lsof). We also reinstalled a few other modules that contain C-code (such as python-ldap) which we need for being able to do authenitcation. Unfortunately that didn't really help much. We still experience crashes. Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that could cause these problems? The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). We're unfortunately still not able to reproduce these crashes. Can you set 'ulimit -c' to get a core file, which might at least help point to the extension which is to blame (although it may just show the downstream victim of a heap munge). What versions of libxml2 / libxslt are you using? How about lxml? Yes, we did set the ulimit and were indeed able to produce a coredump for each crash happening (each having something between 300 and 700 MB). We tried to debug using gdb but unfortunaley they only reveal two cases when the crashes occur: 1) During garbage collection where the gc tries to clean up damaged python objects 2) During some ceval process, also related to accessing damaged python objects Unfortunately it doesn't reveal what exactly trashes the objects. To us it seems that this could happen some time earlier before either of the two processes mentioned above tries to access the objects and crashes ZOPE. For now, we don't really see a reproduceable pattern as it seems to be a somewhat more complex user behavior which leads to this. We were able to extract a few URLs out of the coredumps but directly accessing those does nothing. Also the last logged access in the Z2.log before the coredump triggers nothing, when directly accessing it. We're running ZOPE-2.11.2 with an eggified version of ZODB3-3.8.4 plus libxml2-2.7.6, libxslt-1.1.26 and lxml-2.2.4 now, the crashes still happen. Previously we've been running with ZOPE-2.11.2, libxml2-2.7.3, libxslt-1.1.24 and lxml-2.1.5. That also crashed ZOPE occasionally. This only happened since we switched to a new layout (probably in combination with a few minor Silva updates). We have been using the same system software (RHEL5), hardware, python version and libxml2/libxslt/lxml versions with our old old layout, where everything worked fine for years. I would be happy to paste any particular gdb outputs if that is of any help...? Kind regards, Andreas ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Frequent ZOPE crashes
Hello Mailinglist, we've been using ZOPE in combination with the Silva CMS for around four years now to serve our University's homepage. And everything worked fine so far. A week ago we switched to a new layout (for corporate reasons) and now we're experiencing frequent crashes of the Zope servers. Fortunately enough the reconnect themselves to the ZODB but since this is now happening around every five minutes, I'm rather worried that this might permanently damage the ZODB. I have absolutely no idea how this can happen, as we're using the same python, libxml2, libxslt and other module versions as with the old homepage - in fact the new site even runs on the same hardware. We never experienced any problems like these up until now. As far as I understood so far, it requires some C modules to successfully cause ZOPE to segfault? Versions we're using: Python 2.4.6 Zope 2.11.2 LibXML2 2.7.3 LibXSLT 1.1.24 Python-LDAP 2.3.6 Setuptools 0.6c9 and a Kerberos Module plus the Silva CMS (2.1) on top. We have four ZOPE servers, each running two ZEO processes and a separate ZODB. The machines all run RedHat Enterprise Linux 5.4. In front of that Apache, Squid and Pound take care of the caching. What we did was to examine the coredump-files with gdb but unfortunately this didn't prove to be very helpful because either things go wrong during garbage collection or some ceval stuff. So basically something trashes certain python-objects at time before. Do you have *any* hinst in how to track down this problem? Or are there any known problems with the versions above? The changelogs didn't reveal any plausible cause for me... Kind regards, Andreas Krasa ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
At first, try to eliminate error outside of the Zope itself. Try to install it all into plain whole new (and reliable!) machine. Do not use restore of any backups! - Original Message - From: Andreas Krasa andreas.kr...@wu-wien.ac.at A week ago we switched to a new layout (for corporate reasons) and now we're experiencing frequent crashes of the Zope servers. Fortunately ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
RE : Re: [Zope] Frequent Zope crashes (Zope 2.9.8)
Hi, --- Chris Withers [EMAIL PROTECTED] a écrit : Hi Paul, It's certainly worth filing a bug over in launchpad about this. I have now filed a report: https://bugs.launchpad.net/zope2/+bug/194788. Thanks, Paul _ Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail http://mail.yahoo.fr ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent Zope crashes (Zope 2.9.8)
Hi Paul, It's certainly worth filing a bug over in launchpad about this. I wonder if this is 64-bit related? cheers, Chris Paul Brettschneider wrote: Hello, my Zope 2.9.8 instance crashes up to 6 times per hour. This is very unfortunate since the constant restarting brings performance to its knees. It runs under Linux in 64 bit mode on an AMD64 . I managed to catch two backtraces with gdb (see end of the mail). Both backtraces show a crash in cc_oid_unreferenced(ccobject *self, PyObject *oid) in persistent/cPickleCache.c: Either in line 576: v = PyDict_GetItem(self-data, oid); or in line 607: Py_DECREF((ccobject *)((cPersistentObject *)v)-cache); v and v-cache seem to point to heap: (gdb) print v $1 = (PyObject *) 0x5f8920 (gdb) print ((cPersistentObject *)v)-cache $2 = (PerCache *) 0x613620 Always called from Per_dealloc(cPersistentObject *self) in persistent/cPersistence.c in line 578: cPersistenceCAPI-percachedel(self-cache, self-oid); Is this a known issue? Thank you for any help, Paul #0 0x00436777 in PyDict_Contains () #1 0x004369ad in PyDict_GetItem () #2 0x2b56466e6f37 in cc_oid_unreferenced (self=0x2b564b71c808, oid=0x2ce4ecc0) at persistent/cPickleCache.c:576 #3 0x2b56464ded28 in Per_dealloc (self=0x2ce50050) at persistent/cPersistence.c:578 #4 0x00446bf3 in PyType_GenericAlloc () #5 0x00436bdc in PyDict_GetItem () #6 0x00446c4c in PyType_GenericAlloc () #7 0x00436bdc in PyDict_GetItem () #8 0x00446c4c in PyType_GenericAlloc () #9 0x00436bdc in PyDict_GetItem () #10 0x00446c4c in PyType_GenericAlloc () #11 0x00436bdc in PyDict_GetItem () #12 0x00446c4c in PyType_GenericAlloc () #13 0x00438ddb in _PyTrash_destroy_chain () #14 0x2b56466e772a in cc_clear (self=0x2b564b71c808) at persistent/cPickleCache.c:756 #15 0x0049f212 in _PyObject_GC_UnTrack () #16 0x0049fab5 in _PyObject_GC_New () #17 0x004bc6c8 in PyFunction_New () #18 0x004715ac in PyEval_EvalFrame () #19 0x00474f48 in PyEval_EvalCodeEx () #20 0x00472ca5 in PyEval_EvalFrame () #21 0x00472d99 in PyEval_EvalFrame () #22 0x00472d99 in PyEval_EvalFrame () #23 0x00472d99 in PyEval_EvalFrame () #24 0x00474f48 in PyEval_EvalCodeEx () #25 0x004bc293 in PyClassMethod_New () #26 0x004139f0 in PyObject_Call () #27 0x004196ee in PyClass_IsSubclass () #28 0x004139f0 in PyObject_Call () #29 0x2b5646121a1e in fast_save_leave () from /usr/lib/python2.4/lib-dynload/cPickle.so #30 0x2b5646124a4e in fast_save_leave () from /usr/lib/python2.4/lib-dynload/cPickle.so #31 0x0047453c in PyEval_EvalFrame () #32 0x00472d99 in PyEval_EvalFrame () #33 0x00472d99 in PyEval_EvalFrame () #34 0x00472d99 in PyEval_EvalFrame () #35 0x00474f48 in PyEval_EvalCodeEx () #36 0x004bc293 in PyClassMethod_New () #37 0x004139f0 in PyObject_Call () #38 0x004196ee in PyClass_IsSubclass () #39 0x00415d93 in PyObject_CallMethod () #40 0x2b56464de888 in unghostify (self=0x2de96aa0) at persistent/cPersistence.c:100 #41 0x2b56464de909 in Per_setstate (self=0x24fb740) at persistent/cPersistence.c:1125 #42 0x2b56482ebcb8 in P_getattr (self=0x2de96aa0, name=0x2aafafb0) at Persistence/_Persistence.c:108 #43 0x2b5646b80242 in Wrapper_findattr (self=0x2b61c310, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:479 #44 0x2b5646b81031 in Wrapper_acquire (self=0x2b61cb50, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #45 0x2b5646b8049c in Wrapper_findattr (self=0x2b61cb50, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #46 0x2b5646b81031 in Wrapper_acquire (self=0x2b61cb90, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #47 0x2b5646b8049c in Wrapper_findattr (self=0x2b61cb90, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #48 0x2b5646b81031 in Wrapper_acquire (self=0x2b61c9d0, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #49 0x2b5646b8049c in Wrapper_findattr (self=0x2b61c9d0, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #50 0x2b5646b81031 in Wrapper_acquire (self=0x2b61c490, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out,
[Zope] Frequent Zope crashes (Zope 2.9.8)
Hello, my Zope 2.9.8 instance crashes up to 6 times per hour. This is very unfortunate since the constant restarting brings performance to its knees. It runs under Linux in 64 bit mode on an AMD64 . I managed to catch two backtraces with gdb (see end of the mail). Both backtraces show a crash in cc_oid_unreferenced(ccobject *self, PyObject *oid) in persistent/cPickleCache.c: Either in line 576: v = PyDict_GetItem(self-data, oid); or in line 607: Py_DECREF((ccobject *)((cPersistentObject *)v)-cache); v and v-cache seem to point to heap: (gdb) print v $1 = (PyObject *) 0x5f8920 (gdb) print ((cPersistentObject *)v)-cache $2 = (PerCache *) 0x613620 Always called from Per_dealloc(cPersistentObject *self) in persistent/cPersistence.c in line 578: cPersistenceCAPI-percachedel(self-cache, self-oid); Is this a known issue? Thank you for any help, Paul #0 0x00436777 in PyDict_Contains () #1 0x004369ad in PyDict_GetItem () #2 0x2b56466e6f37 in cc_oid_unreferenced (self=0x2b564b71c808, oid=0x2ce4ecc0) at persistent/cPickleCache.c:576 #3 0x2b56464ded28 in Per_dealloc (self=0x2ce50050) at persistent/cPersistence.c:578 #4 0x00446bf3 in PyType_GenericAlloc () #5 0x00436bdc in PyDict_GetItem () #6 0x00446c4c in PyType_GenericAlloc () #7 0x00436bdc in PyDict_GetItem () #8 0x00446c4c in PyType_GenericAlloc () #9 0x00436bdc in PyDict_GetItem () #10 0x00446c4c in PyType_GenericAlloc () #11 0x00436bdc in PyDict_GetItem () #12 0x00446c4c in PyType_GenericAlloc () #13 0x00438ddb in _PyTrash_destroy_chain () #14 0x2b56466e772a in cc_clear (self=0x2b564b71c808) at persistent/cPickleCache.c:756 #15 0x0049f212 in _PyObject_GC_UnTrack () #16 0x0049fab5 in _PyObject_GC_New () #17 0x004bc6c8 in PyFunction_New () #18 0x004715ac in PyEval_EvalFrame () #19 0x00474f48 in PyEval_EvalCodeEx () #20 0x00472ca5 in PyEval_EvalFrame () #21 0x00472d99 in PyEval_EvalFrame () #22 0x00472d99 in PyEval_EvalFrame () #23 0x00472d99 in PyEval_EvalFrame () #24 0x00474f48 in PyEval_EvalCodeEx () #25 0x004bc293 in PyClassMethod_New () #26 0x004139f0 in PyObject_Call () #27 0x004196ee in PyClass_IsSubclass () #28 0x004139f0 in PyObject_Call () #29 0x2b5646121a1e in fast_save_leave () from /usr/lib/python2.4/lib-dynload/cPickle.so #30 0x2b5646124a4e in fast_save_leave () from /usr/lib/python2.4/lib-dynload/cPickle.so #31 0x0047453c in PyEval_EvalFrame () #32 0x00472d99 in PyEval_EvalFrame () #33 0x00472d99 in PyEval_EvalFrame () #34 0x00472d99 in PyEval_EvalFrame () #35 0x00474f48 in PyEval_EvalCodeEx () #36 0x004bc293 in PyClassMethod_New () #37 0x004139f0 in PyObject_Call () #38 0x004196ee in PyClass_IsSubclass () #39 0x00415d93 in PyObject_CallMethod () #40 0x2b56464de888 in unghostify (self=0x2de96aa0) at persistent/cPersistence.c:100 #41 0x2b56464de909 in Per_setstate (self=0x24fb740) at persistent/cPersistence.c:1125 #42 0x2b56482ebcb8 in P_getattr (self=0x2de96aa0, name=0x2aafafb0) at Persistence/_Persistence.c:108 #43 0x2b5646b80242 in Wrapper_findattr (self=0x2b61c310, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:479 #44 0x2b5646b81031 in Wrapper_acquire (self=0x2b61cb50, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #45 0x2b5646b8049c in Wrapper_findattr (self=0x2b61cb50, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #46 0x2b5646b81031 in Wrapper_acquire (self=0x2b61cb90, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #47 0x2b5646b8049c in Wrapper_findattr (self=0x2b61cb90, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #48 0x2b5646b81031 in Wrapper_acquire (self=0x2b61c9d0, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #49 0x2b5646b8049c in Wrapper_findattr (self=0x2b61c9d0, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, sob=1, sco=1, explicit=0, containment=0) at Acquisition/_Acquisition.c:514 #50 0x2b5646b81031 in Wrapper_acquire (self=0x2b61c490, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0, explicit=value optimized out, containment=0) at Acquisition/_Acquisition.c:544 #51 0x2b5646b8049c in Wrapper_findattr (self=0x2b61c490, oname=0x2aafafb0, filter=0x0, extra=0x0, orig=0x0,
