Re: [Beowulf] Strange SGE scheduling problem

[Shannon V. Davidson]

Schoenefeld, Keith wrote:
This definitely looked promising, but unfortunately it didn't work.  I
both added the appropriate export lines to my qsub file, and then when
that didn't work I checked the mvapich.conf file and confirmed that the
processor affinity was disabled.  I wonder if I can turn it on and make
it work, but unfortunately the cluster is full at the moment, so I can't
test it.
  

You may want to verify that the environment variable was actually passed 
down to the MPI task.  To set environment variables for MPI jobs, I 
usually either specify the environment variable on the mpirun command 
line or in a wrapper script:

  mpirun -np 32 -hostfile nodes  VIADEV_ENABLE_AFFINITY=0 a.out

  mpirun -np 32 -hostfile nodes run.sh a.out

      where run.sh sets up the local environment including environment 
variables.

The second method is more portable to various shells and MPI versions.

Shannon

-- KS

-----Original Message-----
From: Shannon V. Davidson [EMAIL PROTECTED] 
Sent: Wednesday, July 23, 2008 4:02 PM
To: Schoenefeld, Keith
Cc: beowulf@beowulf.org
Subject: Re: [Beowulf] Strange SGE scheduling problem

Schoenefeld, Keith wrote:
  
My cluster has 8 slots (cores)/node in the form of two quad-core
processors. Only recently we've started running jobs on it that
    
require
  
12 slots.  We've noticed significant speed problems running multiple
    
12
  
slot jobs, and quickly discovered that the node that was running 4
    
slots
  
on one job and 4 slots on another job was running both jobs on the
    
same
  
processor cores (i.e. both job1 and job2 were running on CPU's #0-#3,
and the CPUs #4-#7 were left idling.  The result is that the jobs were
competing for time on half the processors that were available.

In addition, a 4 slot job started well after the 12 slot job has
    
ramped
  
up results in the same problem (both the 12 slot job and the four slot
job get assigned to the same slots on a given node).

Any insight as to what is occurring here and how I could prevent it
    
from
  
happening?  We were are using SGE + mvapich 1.0 and a PE that has the
$fill_up allocation rule.

I have also posted this question to the [EMAIL PROTECTED]
mailing list, so my apologies for people who get this email multiple
times.
Any insight as to what is occurring here and how I could prevent it
    
from
  
happening?  We were are using SGE + mvapich 1.0 and a PE that has the
$fill_up allocation rule.
  
    

This sounds like MVAPICH is assigning your MPI tasks to your CPUs 
starting with CPU#0.  If you are going to run multiple MVAPICH jobs on 
the same host, turn off CPU affinity by starting the MPI tasks with the 
environment variable VIADEV_USE_AFFINITY=0 and VIADEV_ENABLE_AFFINITY=0.

Cheers,
Shannon

  
Any help is appreciated.

-- KS

_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
    
http://www.beowulf.org/mailman/listinfo/beowulf
  
  
    

  

_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf