Re: [Zope] Zope + Apache on Quad Debian machine
Thanks Kevin, It's a very good architecture option. I'm afraid a bit too late to test.. I'm already in production with the new architecture and CPU affinity. It's running smoothly though.. :-) Regards Hugo On 3/22/06, ksmith99 <[EMAIL PROTECTED]> wrote: > > Hi Hugo, > > I've got a similar setup, Compaq DL580 on a Quad Xeon with 3gb of ram on > Debian. > To overcome the GIL and avoid dealing with CPU affiinity and to optimize > server utilization, I started using Xen virtualization to create multiple > virtual servers. It's open source and works great. > > Check out http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ or > http://www.xensource.com. For performance see > http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html > > This tutorial worked for me: > http://www.debian-administration.org/articles/320 I also recommend > installing xen-tools http://www.steve.org.uk/Software/xen-tools/ > > It's very easy to create new virtual servers: > xen-create-image --host=apache > xen-create-image --host=zclient1 > xen-create-image --host=zclient2 > xen-create-image --host=zeo > > To create and attach to a virtual server, just do "xm create zclient2.cfg > -c" log in and install python, Zope and whatever else you need. > > So instead of running 1 plone instance at 22pps, you can have serveral plone > instances running at 22pps each and no GIL problem. > > HTH, > > Kevin Smith > > > > > > > > > -- > View this message in context: > http://www.nabble.com/Zope-%2B-Apache-on-Quad-Debian-machine-t1287183.html#a3540420 > Sent from the Zope - General forum at Nabble.com. > > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > -- Hugo Ramos - [EMAIL PROTECTED] http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
Hi Hugo, I've got a similar setup, Compaq DL580 on a Quad Xeon with 3gb of ram on Debian. To overcome the GIL and avoid dealing with CPU affiinity and to optimize server utilization, I started using Xen virtualization to create multiple virtual servers. It's open source and works great. Check out http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ or http://www.xensource.com. For performance see http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html This tutorial worked for me: http://www.debian-administration.org/articles/320 I also recommend installing xen-tools http://www.steve.org.uk/Software/xen-tools/ It's very easy to create new virtual servers: xen-create-image --host=apache xen-create-image --host=zclient1 xen-create-image --host=zclient2 xen-create-image --host=zeo To create and attach to a virtual server, just do "xm create zclient2.cfg -c" log in and install python, Zope and whatever else you need. So instead of running 1 plone instance at 22pps, you can have serveral plone instances running at 22pps each and no GIL problem. HTH, Kevin Smith -- View this message in context: http://www.nabble.com/Zope-%2B-Apache-on-Quad-Debian-machine-t1287183.html#a3540420 Sent from the Zope - General forum at Nabble.com. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
Hugo Ramos wrote at 2006-3-15 17:45 -0300: >I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. >I've noticed that the CPU's never go beyond 30% top occupation... but >on rush hours the site takes too long to load... > >I've been reading about process affinity and how it could speed up >everything by making zope run on 1 CPU, Apache on another and so on... A further, more promissing option, is to run several Zope processes on the same machine (using different ports) with a load balancer in front feeding them with work. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
On Thu, Mar 16, 2006 at 02:38:04PM +0100, Stefan H. Holek wrote: > a) Your system may be I/O bound - not CPU bound - so you will never > see the CPU max out because the limiting factors are memory and disk/ > network access. On that note, if running linux on a box with IDE disks, make sure that the disks have dma turned on. This can have a big impact on overall system responsiveness. man hdparm for details. -PW -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
Thank you all !! (very insane multiple exclamation marks) I've been trying different scenarios and Zope performance increased 10 to 11 times faster... Cheers Hugo On 3/16/06, Stefan H. Holek <[EMAIL PROTECTED]> wrote: > a) Your system may be I/O bound - not CPU bound - so you will never > see the CPU max out because the limiting factors are memory and disk/ > network access. > > b) Make sure to tune Python's checkinterval. While you should > *always* do that, it is especially important on multi-processor/multi- > core hardware. The rule of thumb is to run the pystone benchmark, > divide the "pystones" your machine benchmarks at by 50, and put the > result into etc/zope.conf. > > e.g: > python-check-interval 1000 > > Once you get the checkinterval right, playing with processor affinity > becomes a pointless exercise, in my experience. > > c) Always use profiling tools for performance analysis. "Guessing" > rarely leads to effective optimizations. ZopeProfiler and PTProfiler > come to mind. > > HTH, > Stefan > > > On 15. Mär 2006, at 21:45, Hugo Ramos wrote: > > > Yellow, > > > > I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. > > I've noticed that the CPU's never go beyond 30% top occupation... but > > on rush hours the site takes too long to load... > > > > I've been reading about process affinity and how it could speed up > > everything by making zope run on 1 CPU, Apache on another and so on... > > > > Has any1 tried this before? Can you point me to some documentation? > > What's your experience? is it true that not doing this the 4 cpu's > > will not be 100% used? > > -- > Anything that happens, happens. --Douglas Adams > > > -- Hugo Ramos - [EMAIL PROTECTED] http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
a) Your system may be I/O bound - not CPU bound - so you will never see the CPU max out because the limiting factors are memory and disk/ network access. b) Make sure to tune Python's checkinterval. While you should *always* do that, it is especially important on multi-processor/multi- core hardware. The rule of thumb is to run the pystone benchmark, divide the "pystones" your machine benchmarks at by 50, and put the result into etc/zope.conf. e.g: python-check-interval 1000 Once you get the checkinterval right, playing with processor affinity becomes a pointless exercise, in my experience. c) Always use profiling tools for performance analysis. "Guessing" rarely leads to effective optimizations. ZopeProfiler and PTProfiler come to mind. HTH, Stefan On 15. Mär 2006, at 21:45, Hugo Ramos wrote: Yellow, I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load... I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on... Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used? -- Anything that happens, happens. --Douglas Adams ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope + Apache on Quad Debian machine
Why do you want to do this? Do you believe that this is more effective or efficient than the scheduling algorithm that Linux uses? Binding an application to a single processor makes very little sense at all. If Zope really is your limiting factor, then you need to spread that load over as many processors as possible, not lock it down to a single cpu - if you do that, then the rest of the processors sit idle while waiting for Zope to respond. Read up on ZEO. Then set yourself a ZEO server up and a couple of ZEO clients, balance your requests between the two using pound, apache or squid. I can guarantee you that if your system is Zope limited, then this will greatly increase responsiveness and the number of requests you can process simultaneously - I run over a dozen dual processor machines hosting approx 60 Zope applications across them, all using ZEO - an average of 5 or 6 instances per system, we have no problem with Zope only using one processor; the applications that are across most machines (up to 10 in our largest application) are very, very fast indeed and can cope with all the concurrent users we can throw at it. Your problem is not Zope itself... you definitely need to investigate further before coming to the conclusion that binding a Zope instance to a single processor is a solution - if you'll excuse me, that's a rather simplistic view to take. I was also suggest reading up on Apache, Linux, Proxying/Caching, Clustering and Load Balancing - they're all factors to consider when developing and deploying a web application. John Snowdon - IT Support Specialist -==- School of Medical Education Development Faculty of Medical Sciences Computing University of Newcastle Email : [EMAIL PROTECTED] >-Original Message- >From: [EMAIL PROTECTED] >[mailto:[EMAIL PROTECTED] On >Behalf Of Hugo Ramos >Sent: 15 March 2006 20:45 >To: [Zope] >Subject: [Zope] Zope + Apache on Quad Debian machine > > >Yellow, > >I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. >I've noticed that the CPU's never go beyond 30% top occupation... but >on rush hours the site takes too long to load... > >I've been reading about process affinity and how it could speed up >everything by making zope run on 1 CPU, Apache on another and so on... > >Has any1 tried this before? Can you point me to some documentation? >What's your experience? is it true that not doing this the 4 cpu's >will not be 100% used? > > >Cheers >-- >Hugo Ramos - [EMAIL PROTECTED] >http://www.orkut.com/Profile.aspx?uid=10082105466310142690 >___ >Zope maillist - Zope@zope.org >http://mail.zope.org/mailman/listinfo/zope >** No cross posts or HTML encoding! ** >(Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
You can do something like: /usr/bin/taskset -c 0 zeo/bin/zeoctl start /usr/bin/taskset -c 1 zope.a/bin/zopectl start /usr/bin/taskset -c 2 zope.b/bin/zopectl start /usr/bin/taskset -c 3 zope.c/bin/zopectl start You then need to load-balance across the ZOPE instances with (eg Pound or Squid). AFAIK it's only Python-based programs that need binding to a particular CPU, so Apache, Pound and Squid are fine without setting any affinity. Since ZEO is not very CPU intensive, you can share it's CPU with another ZOPE instance should you wish. We're running twin ZOPE instances and ZEO on a dual CPU Debian box, load balancing with Pound and caching and URL rewriting with Apache. The feeling these days seems to be that load balancing and caching with Squid gives better performance. Steven Steven Hayles - Computer Systems Developer, [EMAIL PROTECTED] Learning Technology Section, Computer Centre, University of Leicester, University Rd, Leicester, LE1 7RH Fax (0/+44)116 2525027 WWW http://www.le.ac.uk/home/sh23> On Wed, 15 Mar 2006, Hugo Ramos wrote: Yellow, I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load... I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on... Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used? Cheers -- Hugo Ramos - [EMAIL PROTECTED] http://www.orkut.com/Profile.aspx?uid=10082105466310142690 ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope + Apache on Quad Debian machine
On Wed, Mar 15, 2006 at 05:45:23PM -0300, Hugo Ramos wrote: > Yellow, > > I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. > I've noticed that the CPU's never go beyond 30% top occupation... but > on rush hours the site takes too long to load... > > I've been reading about process affinity and how it could speed up > everything by making zope run on 1 CPU, Apache on another and so on... > > Has any1 tried this before? Can you point me to some documentation? > What's your experience? is it true that not doing this the 4 cpu's > will not be 100% used? Have you considered that your setup might be IO bound? -- __ "Nothing is as subjective as reality" Reinoud van Leeuwen[EMAIL PROTECTED] http://www.xs4all.nl/~reinoud __ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
