Re: Too Much Context Switching? - FIXED
[EMAIL PROTECTED] wrote: [snip] I'll probably be upgrading to 7.0 in the next month or so, given that this is obviously a thread issue and that that release has much improved thread code. However, for the time being, the pressing issue is fixed, and for anyone in my position stuck on 6.2...this is night day. It has been over a year and a half, or so, since I last experimented with Zope. I only have 9 FreeBSD servers, but in my circumstance I've had good results with 7. When you build Python set the HUGE_STACK_SIZE to yes. I believe the default is to have thread support already on. Even when Python app code is written with multithreading, the execution still runs through the 'Global_Interpreter_Lock' when run. So with this limitation in mind, should you still observe Zope/Python only utilizing only one core in an SMP machine an alternative may be to see if you can run Zope as FastCGI and start more than one instance, ie one for each core. Keep in mind that FastCGI brings in a whole new dimension of it's own problems and instabilities. A problem that may arise in such a situation is loss of session if a request should switch instances somehow. Test for this if you can. Before attempting such I would profile/bench the box as it is now. Since this might be considered expiremental I'd *not* use the production box unless you are in a downtime, and have sufficient time to play around. Best would be to try this kind of stuff on a second box and not mess with the production one, ala - if it works don't FIX it! :-) If you succeed in getting multiple Zope instances using multiple cores, and have lots of RAM you may also consider giving memcached a go. As I said earlier I'd only play with these ideas in a lab testing scenario and *not* the production box. YMMV -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Too Much Context Switching? - FIXED
Quoting Kris Kennaway [EMAIL PROTECTED]: [EMAIL PROTECTED] wrote: Quoting Kris Kennaway [EMAIL PROTECTED]: Michel Talon wrote: On Mon, Jun 30, 2008 at 07:53:00PM +0200, Kris Kennaway wrote: Yep, it could be that -- what confuses me though is that it is claimed that performance suddenly regressed. If so then this cannot be the underlying cause. It may be that the load has augmented to the point that contention imposes a rapid regression on throughput. Yes, it could be that. I don't know off-hand whether multiple threads are counted separately by vmstat (at a guess I'd say no), but ps/top/etc should show how many are active in the python process. Just ran ktrace, and a bit of Googling seems to confirm my initial suspicion that the results I'm seeing are abnormal. The first several screenfulls of output look like this: 52929 python2.4 1214867016.469416 CALL kse_wakeup(0x811740c) 52929 python2.4 0.60 RET kse_wakeup 0 52929 python2.4 0.08 RET kse_release 0 52929 python2.4 0.40 CALL kse_release(0x811df4c) 52929 python2.4 0.000515 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.12 CALL kse_release(0x811df4c) 52929 python2.4 0.000365 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.03 RET kse_release 0 52929 python2.4 0.10 CALL kse_release(0x811df4c) 52929 python2.4 0.000413 CALL kse_wakeup(0x811740c) 52929 python2.4 0.11 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) 52929 python2.4 0.000393 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) I may be mistaken, but it seems like that's a lot of unnecessary activity managing the threads; the confirmation I found came from http://arkiv.freebsd.se/?ml=freebsd-threadsa=2007-02t=3178634. Am I correct that this is abnormal behavior? If so, any idea what I may need to do to fix the issue? Looks exactly like the python thread problem Michel described. You will get some improvement by switching to libthr and/or updating to 7.0 as I discussed, but ultimately you're hitting limits of python, not FreeBSD. WOW...it's *amazing* how much of a difference a single sysctl can make. I went ahead and set kern.threads.virtual_cpu=1, as suggested in the thread above, and the difference is ridiculous -- Zope is now faster than I've ever seen. More importantly, my ktracing shows that all of the kse_* garabage is now gone. I'll probably be upgrading to 7.0 in the next month or so, given that this is obviously a thread issue and that that release has much improved thread code. However, for the time being, the pressing issue is fixed, and for anyone in my position stuck on 6.2...this is night day. Alex ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Too Much Context Switching? - FIXED
[EMAIL PROTECTED] wrote: Quoting Kris Kennaway [EMAIL PROTECTED]: [EMAIL PROTECTED] wrote: Quoting Kris Kennaway [EMAIL PROTECTED]: Michel Talon wrote: On Mon, Jun 30, 2008 at 07:53:00PM +0200, Kris Kennaway wrote: Yep, it could be that -- what confuses me though is that it is claimed that performance suddenly regressed. If so then this cannot be the underlying cause. It may be that the load has augmented to the point that contention imposes a rapid regression on throughput. Yes, it could be that. I don't know off-hand whether multiple threads are counted separately by vmstat (at a guess I'd say no), but ps/top/etc should show how many are active in the python process. Just ran ktrace, and a bit of Googling seems to confirm my initial suspicion that the results I'm seeing are abnormal. The first several screenfulls of output look like this: 52929 python2.4 1214867016.469416 CALL kse_wakeup(0x811740c) 52929 python2.4 0.60 RET kse_wakeup 0 52929 python2.4 0.08 RET kse_release 0 52929 python2.4 0.40 CALL kse_release(0x811df4c) 52929 python2.4 0.000515 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.12 CALL kse_release(0x811df4c) 52929 python2.4 0.000365 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.03 RET kse_release 0 52929 python2.4 0.10 CALL kse_release(0x811df4c) 52929 python2.4 0.000413 CALL kse_wakeup(0x811740c) 52929 python2.4 0.11 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) 52929 python2.4 0.000393 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) I may be mistaken, but it seems like that's a lot of unnecessary activity managing the threads; the confirmation I found came from http://arkiv.freebsd.se/?ml=freebsd-threadsa=2007-02t=3178634. Am I correct that this is abnormal behavior? If so, any idea what I may need to do to fix the issue? Looks exactly like the python thread problem Michel described. You will get some improvement by switching to libthr and/or updating to 7.0 as I discussed, but ultimately you're hitting limits of python, not FreeBSD. WOW...it's *amazing* how much of a difference a single sysctl can make. I went ahead and set kern.threads.virtual_cpu=1, as suggested in the thread above, and the difference is ridiculous -- Zope is now faster than I've ever seen. More importantly, my ktracing shows that all of the kse_* garabage is now gone. I'll probably be upgrading to 7.0 in the next month or so, given that this is obviously a thread issue and that that release has much improved thread code. However, for the time being, the pressing issue is fixed, and for anyone in my position stuck on 6.2...this is night day. Seriously, try libthr. No matter what you do to libkse it is going to suck. That's why we removed it. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Too Much Context Switching? - FIXED
Quoting Kris Kennaway [EMAIL PROTECTED]: [EMAIL PROTECTED] wrote: Quoting Kris Kennaway [EMAIL PROTECTED]: [EMAIL PROTECTED] wrote: Quoting Kris Kennaway [EMAIL PROTECTED]: Michel Talon wrote: On Mon, Jun 30, 2008 at 07:53:00PM +0200, Kris Kennaway wrote: Yep, it could be that -- what confuses me though is that it is claimed that performance suddenly regressed. If so then this cannot be the underlying cause. It may be that the load has augmented to the point that contention imposes a rapid regression on throughput. Yes, it could be that. I don't know off-hand whether multiple threads are counted separately by vmstat (at a guess I'd say no), but ps/top/etc should show how many are active in the python process. Just ran ktrace, and a bit of Googling seems to confirm my initial suspicion that the results I'm seeing are abnormal. The first several screenfulls of output look like this: 52929 python2.4 1214867016.469416 CALL kse_wakeup(0x811740c) 52929 python2.4 0.60 RET kse_wakeup 0 52929 python2.4 0.08 RET kse_release 0 52929 python2.4 0.40 CALL kse_release(0x811df4c) 52929 python2.4 0.000515 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.12 CALL kse_release(0x811df4c) 52929 python2.4 0.000365 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.03 RET kse_release 0 52929 python2.4 0.10 CALL kse_release(0x811df4c) 52929 python2.4 0.000413 CALL kse_wakeup(0x811740c) 52929 python2.4 0.11 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) 52929 python2.4 0.000393 CALL kse_wakeup(0x811740c) 52929 python2.4 0.12 RET kse_wakeup 0 52929 python2.4 0.04 RET kse_release 0 52929 python2.4 0.09 CALL kse_release(0x811df4c) I may be mistaken, but it seems like that's a lot of unnecessary activity managing the threads; the confirmation I found came from http://arkiv.freebsd.se/?ml=freebsd-threadsa=2007-02t=3178634. Am I correct that this is abnormal behavior? If so, any idea what I may need to do to fix the issue? Looks exactly like the python thread problem Michel described. You will get some improvement by switching to libthr and/or updating to 7.0 as I discussed, but ultimately you're hitting limits of python, not FreeBSD. WOW...it's *amazing* how much of a difference a single sysctl can make. I went ahead and set kern.threads.virtual_cpu=1, as suggested in the thread above, and the difference is ridiculous -- Zope is now faster than I've ever seen. More importantly, my ktracing shows that all of the kse_* garabage is now gone. I'll probably be upgrading to 7.0 in the next month or so, given that this is obviously a thread issue and that that release has much improved thread code. However, for the time being, the pressing issue is fixed, and for anyone in my position stuck on 6.2...this is night day. Seriously, try libthr. No matter what you do to libkse it is going to suck. That's why we removed it. I will, probably as part of upgrading to 7.0 (which I may accelerate, given this point). I'm just ecstatic at the difference I'm already seeing, and specifically wanted to make note of it in the archives. Point very much taken, though. :-) Alex ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Too Much Context Switching? - FIXED
I will, probably as part of upgrading to 7.0 (which I may accelerate, given this point). I'm just ecstatic at the difference I'm already seeing, and specifically wanted to make note of it in the archives. Point very much taken, though. :-) It's trivial to change to libthr, as pointed out earlier in this thread. You simply add an entry/entries to /etc/libmap.conf (see man libmap.conf for details) and then restart whatever it is that is currently running against libkse. I'll second Kris' recommendation to move to libthr. I saw a drastic improvement in MySQL and ffmpeg performance on 6.2 when I switched from libkse to libthr. Certainly 7.0 would give it to you automatically, but there's no reason not to use libmap to use it now, as an interim solution. Josh ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
