On Mon, 2006-12-11 at 15:03 -0500, Lobsingerp,Peter [CIS] wrote:
> Hello,
>
> I am running a script that needs to check its results with a lot of
> random data (Monte Carlo methods). I am trying to use ithreads to do
> this. However, on an 8 CPU server, Perl never uses more than 200% CPU. I
> am running Perl 5.8.5 on Red Hat EL. Is there some configuration that
> could allow Perl to use more CPU while using ithreads?
Recent releases of perl have some major improvements in ithreads
stability. You might try compiling perl 5.8.8 and see if things
improve. That said, your limitation is probably locking between
threads, which isn't going to get better with more threads. I haven't
really had much luck with large numbers of threads in perl. IIRC,
about 10 threads pretty neatly consumes two HT processors and their
siblings just running LWP and parsing the returned HTML (most of the CPU
time).
Have you tried less (~10) threads?
Are you yielding a lot? Yielding too often at best causes context
switching to eat precious CPU time.
What approach are you using? Pooling? Master/Worker? Spawn &
Forget?
> Also, how well do threads scale? My initial attempts are using 100
> threads but this may need to rise to 1000. As it stands it is taking
> forever to complete, but this is most likely caused by not having access
> to all CPUs that are available.
You'll need a LOT of RAM. Depending on your data sharing needs, it may
be way more efficient to use fork() and SysV IPC (message queues are
pretty easy).
use IPC::SysV qw(IPC_PRIVATE S_IRWXU);
use IPC::Msg;
use constant WORKER_COUNT => 25;
my $msg = IPC::Msg->new( IPC_PRIVATE, S_IRWXU );
...
for ( 0..WORKER_COUNT ) {
my $pid = fork();
if ( $pid ) {
... # parent
}
else {
while (1) {
$msg->rcv( $data, 8192 ); # will block on empty queue
compute_stuff( $data );
}
}
}
# load up the queue (see perldoc IPC::Msg)
...
Perl's threads don't benefit in any significant way from Linux's CoW
semantics because it manually clones the interpreter for every thread,
so fork can actually use less memory.
I have a web load testing tool that uses ithreads that consumes easily
5mb or more per/thread. It scales linearly in memory usage, but large
numbers of threads kill it with context switching and locking. (look at
vmstat 1 during runtime).
-Al Tobey
>
> Any help would be much appreciated,
>
> Peter Lobsinger
** ** ** PRIVILEGED AND CONFIDENTIAL ** ** **
This email transmission contains privileged and confidential information
intended only for the use of the individual or entity named above. Any
unauthorized review, use, disclosure or distribution is prohibited and may be a
violation of law. If you are not the intended recipient or a person
responsible for delivering this message to an intended recipient, please delete
the email and immediately notify the sender via the email return address or
mailto:[EMAIL PROTECTED] Thank you.
- end -
