Apologies folks, as Stas just pointed out, I was not replying to all
We thought that ithreads would be the ideal tool - should solve the issues of processes communicating with each other for us, so that we don't have to. Yes, (pre) forking would work and running a separate "server" to handle the concurrent roboting would work, but we thought that it would be neat and easy to maintain if we wrapped it all up one app with shared variables. For us the point of ithreads is that you don't have to mess about passing messages or whatever - all those problems have been solved by the dudes who wrote the threading module and you and they have given you shared variables for your convenience. I had supposed that the entire rationale of ithreads / shared variables is to give you shared variables to make the programmer's life easier. Apart from the memory usage we it has done everything we want and performs quickly and reliably. > > > ----- Original Message ----- > From: "Perrin Harkins" <[EMAIL PROTECTED]> > To: "Simon Clewer" <[EMAIL PROTECTED]> > Cc: <[EMAIL PROTECTED]> > Sent: Friday, January 09, 2004 2:10 AM > Subject: Re: ithreads with modperl > > > > On Thu, 2004-01-08 at 20:22, Simon Clewer wrote: > > > Huge memory usage ... each ithread uses about 10M of ram ( image of > Apache, > > > image of mod perl and image of our deep-link robot ), and as we use 5 > > > ithreads plus the original thread that means that each Apache is using > 60 M > > > and because we trade on being the best and the fastest at what we do we > need > > > to keep plenty of Apaches ready and waiting ( about 20 ) - so we're > using > > > heaps of memory. > > > > My question would be, why are you using Perl threads for this? The talk > > about the 5.8 threads sounds pretty bad, both for memory and > > performance. I can't imagine ithreads were a whole lot better on either > > front. I think you'd be better off forking. > > > > > Does anybody know how we can reduce the amount of memory we use ? > > > > If I were designing a system to do what you're doing, I would make it > > asynchronous. Do the form interaction with mod_perl, then let another > > program do the data collection without typing up an apache process, and > > let your user wait on a "working..." page until it's done. > > > > You can do this by forking, or by pushing a request onto a queue (like a > > database table or dbm file) that another process monitors. > > > > The techniques for forking from mod_perl are described here: > > > http://perl.apache.org/docs/1.0/guide/performance.html#Forking_and_Executing > _Subprocesses_from_mod_perl > > > > An example of a "working..." page is given in this column by Randal: > > http://www.stonehenge.com/merlyn/WebTechniques/col20.html > > > > - Perrin > > > > > -- Reporting bugs: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html
