Greetings...
I do something similar, pulling across 400+ log files, 3GB+ total size every
night. However, I am using considerably fewer drones than you. We found
that we reached a saturation point long before that. Disk I/O and network
I/O both became bottlenecks before we reached 20 processes. Server has 8
processors and 16GB real, 16GB swap. I also chose to use the Net::FTP
module rather than continually forking off more sub-processes using the UNIX
ftp client. Plus, I found that I have better control and real
error-checking with the module.
I have not seen an real memory problem inherently due to forking. However,
the drone process will pre-allocate the same amount of memory as the queen.
So, if the queen is 10MB and you create 100 drones...10x100... The only way
to reduce this is to spawn the drones before pulling anything 'extra' into
memory.
Not sure if any of that helps you at all, but there is my two cents.
Jeremy Elston
Sr. Staff Unix System Administrator
Electronic Brokerage Technology
Charles Schwab & Co., Inc.
(602.431.2087 [EMAIL PROTECTED])
"Life is the final exam where no one can cheat, because we all have
different problems to solve."
WARNING: All email sent to this address will be received by the Charles
Schwab Corporate email system and is subject to archival and review by
someone other than the recipient
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 21, 2001 9:30 AM
To: [EMAIL PROTECTED]
Subject: RE: [Perl-unix-users] memory usage of perl processes
>I guess you could use 'top' in a unix window.
>
>Then kick off 1 drone only and look at the memory usage. Then * it by the
>number of processes you expect.
>55 is a lot of processes!!!! On Any unix system.
>
>Perhaps you could stagger the number of processes. so maybe spawn 30
drones
>and have each drone process 2 sites in serial, one after the other?
>
>I've seen some pretty large perl processes on very simillar spec machines
as
>yours.
>
>Marty
>> -----Original Message-----
>>
>> I am designing a system to process almost 4000 remote sites in a nightly
>> sweep. This process is controlled from a database which maintains site
>> status in realtime (or at least that's the goal). I am
>> attempting to fork
>> off around 100 "drones" to process 100 concurrent sites. Each drone
will
>> need a connection to the database. In doing some impromptu testing I've
>> had the following results...
>>
>> Writing a queen, who does nothing, and sets nothing (no variables or
>> filehandles are open) except fork off drones, and writing a
>> drone who only
>> connects to the database and nothing else had the following
>> results on this
>> machine config:
>>
>> RedHat Linux 6.2, PIII 600, 256MB RAM, 256MB Swap - Anything more than
55
>> drones and the system is entirely out of memory.
>>
>> Is Perl really using that much memory? There is approx 190MB of RAM
free,
>> and nearly ALL the swap space free when I kick off the Queen process.
>>
>> Do there results seem typical? Is there any way to optimize this?
>>
>> Thanks
>>
>> Chuck
Marty,
top is in fact what I have been using to watch it, but I have heard that
you cannot fully trust it's memory reporting because of shared memory usage
and other things, but of course that may be fixed.
The production box this system will live on is same OS, but dual PIII 733,
1GB RAM, 2GB Swap BUT! there's always a but! it's not as though THIS
system has full access to that box, another nightly system runs around the
same time, so there will surely be overlap. I am looking to optimize this
system as much as possible to avoid bringing the box down cause it's out of
memory. The reason for wanting so many concurrent drones is because the
process runs for several hours currently (using 100 plain FTP's launched
from shell scripts) and we are adding a new site approx every 17 hours - no
joke. So we don't want to make it take any longer than it does already, in
fact we'd like to be able to scale up if need be as the number of sites
increase.
I guess my real question here is: Is this much memory usage just a fact of
life when using perl and fork()ing? If so, I can try to ask for more RAM,
and size the production box accordingly. Obviously my testing is doing
NOTHING compared to what will really be getting processed in the real
system, so RAM usage is likely to much larger than it is in these tests -
which is my fear. I intened to use good practices, close any handles, and
free as many variables as possible before forking, etc. but I have fears of
bringing this box to it's knees, so if anyone can offer some tips to
optimize the system I am all ears!
Thanks
Chuck
_______________________________________________
Perl-Unix-Users mailing list. To unsubscribe go to
http://listserv.ActiveState.com/mailman/subscribe/perl-unix-users
_______________________________________________
Perl-Unix-Users mailing list. To unsubscribe go to
http://listserv.ActiveState.com/mailman/subscribe/perl-unix-users
