Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
Heimo Laukkanen wrote: Hi, I am hitting my head agains wall - and witnessing strange behaviour, where after time most of load focuses only to one thread. I am not sure what is the cause of this, but so far we have seen different things before load has concentrated to one single thread. We also have this behaviour, but I have thought that it was our overusage of DCOracle2 connections that was the culprit, along with out bad TTW code. Do you use any DA, nad have long running SQL queries? We have no fix for this, but have resigned to do restarts every now and then. We use Zope 2.5.1, Redhat 7.3 /dario -- -- --- Dario Lopez-Ksten, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
On Wed, 26 May 2004 08:09:53 +0200, Dario Lopez-Kästen [EMAIL PROTECTED] wrote: Heimo Laukkanen wrote: Hi, I am hitting my head agains wall - and witnessing strange behaviour, where after time most of load focuses only to one thread. I am not sure what is the cause of this, but so far we have seen different things before load has concentrated to one single thread. We also have this behaviour, but I have thought that it was our overusage of DCOracle2 connections that was the culprit, along with out bad TTW code. Do you use any DA, nad have long running SQL queries? Yes, extensively. We use DCOracle2 that had to be patched to work with UTF-8. And we combine it with badly scripted code too. We started to do prototype by using own Archetypes storage layer that stores data from content objects to the Oracle database in certain format - so every access to an object actually creates a load of queries to the Oracle database. Connections to the databse are handled by the Oracle adapter in Zope, so we have not done any own connections but rather just run queries through Zope adapter. We have no fix for this, but have resigned to do restarts every now and then. We use Zope 2.5.1, Redhat 7.3 How did you debug or pinpoint the culprit to be DCOracle connection? Have you tried cx_Oracle? -- -huima ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
Heimo Laukkanen wrote: Do you use any DA, nad have long running SQL queries? Yes, extensively. We use DCOracle2 that had to be patched to work with UTF-8. And we combine it with badly scripted code too. We started to do prototype by using own Archetypes storage layer that stores data from content objects to the Oracle database in certain format - so every access to an object actually creates a load of queries to the Oracle database. Connections to the databse are handled by the Oracle adapter in Zope, so we have not done any own connections but rather just run queries through Zope adapter. We use a lot of queries (minimum 10-15) for each request, accros around 7 DA connected to various Oracle schemas. Most of the queries are fairsly simple, but there are a couple of queries that are often used that are just plain bad, however they are so bad that it is very difficult to fix them. These queries generate a lot of load on the DB server (they are *very* bad SQL). So these tend to take a long tiem to respond. We have no fix for this, but have resigned to do restarts every now and then. We use Zope 2.5.1, Redhat 7.3 How did you debug or pinpoint the culprit to be DCOracle connection? Gut feeling, doing some basic optimisation of the not-so complicated cases. I did receive a few tips on how to do real debugging from Matt, Dieter and others, but those tips assume a level of knowlegde that I do not posses yet - also I have until recently not had any hardware to do that kind of debugging on. I just made some basic observations - some times Zope would stop responding and angry users would call, and I woudl find threads that had been running for more thatn 7 seconds, while at the same time I could observe queries on the database that had been running for similar lenght of time, occasionally even blocking the database altoghther. Some times it seemed like Zope wold lose the connection to the database (we still get that randomly from now and then) but I am not 100% sure that this is only zope's fault - it may be that the database was under so much load that it lost the connection to zope, thus triggering some kidn of wait for reply loop on zope's side. Chris withers has done some work in improving DCOracle2's connections and general bug-fixning. If you haven't used it, grab the latest DCOracle2 from cvs - it is much better. We see this behaviour mostly under heavy load - many users accessing the database all the time. Using the latest DCOracle and improving parts of our code has removed a lot of the problems, however it believe it is still unclcear what the problem is - in our setting, my stance is that we have only cured the symptoms, not the real problems... Have you tried cx_Oracle? No I was not aware that they had a Zope adaptor. /dario -- -- --- Dario Lopez-Kästen, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
On Wed, 26 May 2004 08:44:44 +0200, Dario Lopez-Kästen [EMAIL PROTECTED] wrote: We use a lot of queries (minimum 10-15) for each request, accros around 7 DA connected to various Oracle schemas. Most of the queries are fairsly simple, but there are a couple of queries that are often used that are just plain bad, however they are so bad that it is very difficult to fix them. Ok. In our case we use very simple queries, but this stupid code generates a lot of those. One thing to do is to throw away stupidity and Archetypes tsorage layer that is an bad idea - and put object to flush and read it's data from the oracle only when actually necessary. Chris withers has done some work in improving DCOracle2's connections and general bug-fixning. If you haven't used it, grab the latest DCOracle2 from cvs - it is much better. We did take it from CVS. Has there been much work on lately? We see this behaviour mostly under heavy load - many users accessing the database all the time. Using the latest DCOracle and improving parts of our code has removed a lot of the problems, however it believe it is still unclcear what the problem is - in our setting, my stance is that we have only cured the symptoms, not the real problems... Sounds familiar. Have you tried cx_Oracle? No I was not aware that they had a Zope adaptor. Well Dieter and others suggested that it should be doable to to write adaptor based on any python db api compliant adaptor - either using DCOracle, psycopg or similar as base. However since I've never done that kind of work before I am reluctant to take the step and be bitten, hence the question if you have had more courage. -- -huima ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
Heimo Laukkanen wrote: On Wed, 26 May 2004 08:44:44 +0200, Dario Lopez-Kästen [EMAIL PROTECTED] wrote: Ok. In our case we use very simple queries, but this stupid code generates a lot of those. One thing to do is to throw away stupidity and Archetypes tsorage layer that is an bad idea - and put object to flush and read it's data from the oracle only when actually necessary. I agree. We cannot do it however, because ost of my code is mostly TTW, so w have no custom obejcts. However, this is a good strategy in gerenal - do not read from external soruces unless it is necessary... Chris withers has done some work in improving DCOracle2's connections and general bug-fixning. If you haven't used it, grab the latest DCOracle2 from cvs - it is much better. We did take it from CVS. Has there been much work on lately? not that I am aware of. (I haven't checked :) Have you tried cx_Oracle? No I was not aware that they had a Zope adaptor. Well Dieter and others suggested that it should be doable to to write adaptor based on any python db api compliant adaptor - either using DCOracle, psycopg or similar as base. However since I've never done that kind of work before I am reluctant to take the step and be bitten, hence the question if you have had more courage. No, sorry. For me, this too is beyond what I can do at the moment, mostly due to time constraints. Hiopefully this will chagne as more people become invlved with Zope at my work... /dario -- -- --- Dario Lopez-Kästen, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
Dario Lopez-Kästen wrote: Chris withers has done some work in improving DCOracle2's connections and general bug-fixning. If you haven't used it, grab the latest DCOracle2 from cvs - it is much better. My work is still on a branch... Have you tried cx_Oracle? No I was not aware that they had a Zope adaptor. I think they do, if it's significantly better than DCOracle, then I'll tell people to switch instead of trying to make DCOracle work further ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Python 2.3.4c1] nasty LinuxThread problem not solved
On Mon, 24 May 2004 11:29:01 -0400, Tim Peters wrote: [Dieter Maurer] I will reraise the question and see what my colleagues feel as the less problematic way: use NPTL or our own Python version. It would sure help if people running Zope on an NPTL system spoke up here! My development machine uses NPTL : $ getconf GNU_LIBPTHREAD_VERSION NPTL 0.60 I am now starting to create a test deployment. That machine, which will be the production machine, also uses the same version of NPTL. Both systems are Debian with kernel 2.6.5. (FWIW) -- mailhost:/etc/mail# less sendmail.cf less: syntax of file sendmail.cf may induce nausea, show anyway? [n] www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem not solved
Dieter Maurer wrote: http://sourceforge.net/tracker/index.php? func=detailaid=949332group_id=5470atid=305470 indeed fixes the problem. It might introduce other subtle problems but at least none that are revealed by Python's regression test suite... Moreover, I doubt that such problems will be significant in practise: You might want to take a look at: http://sourceforge.net/tracker/?func=detailaid=960406group_id=5470atid=305470 and comment on its effectiveness. It fixes the original problem that lead to having signals blocked in spawned threads. (The problem essentially being that a Control-C in readline might be caught by the wrong thread.) but without the blocking. (besides the segfault problem, it also fixes the way that os.system() and os.popen() spawn children with blocked signals, and similar problems.) It does so in a more general purpose way that should help python on all supported platforms, and not just the specifics of LinuxThreads. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Read-only root database doesn't work ... bug orfeature?
Paul Winkler wrote: The fix is in the latest Formulator CVS; Paul, please test it if you can and let me know if you still see the untowards behavior. It seems good, thanks!! Great! Thanks everybody! Regards, Martijn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope3-dev] Re: SVN: Zope3/trunk/functional_tests/ Remove unused directory.
On Wednesday 26 May 2004 08:42 am, Philipp von Weitershausen wrote: That works for me too, but please *above* the diff. Yes; I should have been more clear and said *immediately* following the commit message. No, the CVS should stay in there to tell them apart, for the very same reason you stated (patches in both CVS and svn). Without the 'SVN:', it will still be unique. I would even argue that it would make it visually easier. Right now, my email client lists a lot of: Ok; like I said, I don't feel very strongly about this. Sure. I'm not saying that we should convert everything *right* now, but moving relatively independent stuff makes sense, IMO. I'm not suggesting to move CMF or something else with a gazillion branches... Cool. zpkg is a good candidate, since it's not shared with other projects, it's a stand-alone tool. -Fred -- Fred L. Drake, Jr. fred at zope.com PythonLabs at Zope Corporation ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem not solved
[EMAIL PROTECTED] wrote at 2004-5-26 11:28 -0400: ... You might want to take a look at: http://sourceforge.net/tracker/?func=detailaid=960406group_id=5470atid=305470 and comment on its effectiveness. It fixes the original problem that lead to having signals blocked in spawned threads. (The problem essentially being that a Control-C in readline might be caught by the wrong thread.) but without the blocking. (besides the segfault problem, it also fixes the way that os.system() and os.popen() spawn children with blocked signals, and similar problems.) It does so in a more general purpose way that should help python on all supported platforms, and not just the specifics of LinuxThreads. Note the following extract from the Python Library Reference doc-2.3/lib/module-signal.html: and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). Thus, beside the Control-C issue, we respect/update this assertion about signal delivery. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope threads on 2.7.0, python 2.3.3
Heimo Laukkanen wrote at 2004-5-25 22:46 +0300:
...
What does Control_Panel -- Debug information tells you
about the use of your connections (at the bottom of the page)?
At the moment it said that only one opened connection and others were
none. I have no terminal access to the machines at the moment to check how
threads are behing at the moment.
That looks strange...
Maybe, the only running thread does not release the GIL?
And then backtrace with gdb from those quiet threads
My most essential tools when analysing Python code in gdb
are the following two GDB command definitions.
def ps
x/s ({PyStringObject}$arg0)-ob_sval
end
def pfr
ps f-f_code-co_filename
ps f-f_code-co_name
p f-f_lineno
end
ps allows you to look at a Python String variable
and pfr can be called in eval_frame frames.
It tells you what code this frame is executing -- identified
by module and function. You cannot trust the lineno in
Python 2.3 (it is the start of the function not where you actually
are).
~# gdb program 14552
backtrace
...
#7 0x080a22f6 in eval_frame (f=0x9b5ea24) at Python/ceval.c:2116
use fr 7 to select this frame and then pfr to see where this is.
...
Another thread ( 14548 )
#0 0x401153c4 in read () from /lib/libc.so.6
#1 0x40029ae0 in __DTOR_END__ () from /lib/libpthread.so.0
#2 0x412474f0 in nttrd () from
/opt/portaali/comp/oracle/lib/libclntsh.so.9.0
#3 0x410fdcf8 in nsprecv () from
Obviously, this is a call to Oracle.
Maybe, the GIL is not released?
...
#14 0x40e13631 in Cursor_execute (self=0x4216fcd0, args=0x4017002c) at
src/dco2.c:3740
The GIL should have been released in the function above.
Check it for Py_BEGIN_ALLOW_THREADS, Py_END_ALLOW_THREADS
around line 3740.
PID: 14547
#0 0x401153c4 in read () from /lib/libc.so.6
#1 0x40029ae0 in __DTOR_END__ () from /lib/libpthread.so.0
#2 0x412474f0 in nttrd () from
/opt/portaali/comp/oracle/lib/libclntsh.so.9.0
This, too is waiting for Oracle...
PID: 14545
#0 0x401153c4 in read () from /lib/libc.so.6
#1 0x40029ae0 in __DTOR_END__ () from /lib/libpthread.so.0
#2 0x412474f0 in nttrd () from
/opt/portaali/comp/oracle/lib/libclntsh.so.9.0
#3 0x410fdcf8 in nsprecv () from
This, too...
PID: 14531
#0 0x4011c7ee in select () from /lib/libc.so.6
#1 0x40568cb4 in __DTOR_END__ () from
/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/lib-dynload/select.so
#2 0x080e0e3f in PyCFunction_Call (func=0x4049636c, arg=0x426476e4,
kw=0x0) at Objects/methodobject.c:73
#3 0x080a3beb in call_function (pp_stack=0xb11c, oparg=4) at
Python/ceval.c:3439
#4 0x080a22f6 in eval_frame (f=0x9033f64) at Python/ceval.c:2116
Check where you are here (-- pfr). It may be the medusa/asyncore main loop.
You have several threads waiting for responses from Oracle.
Are you sure that Control_Panel -- Debug information
tells you only about a single ZODB connection?
--
Dieter
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
