Gary, This is spot on. I've tried various ways and haven't allocated a 1 GPU job and a 2 GPU job and the second job always gets split between K80's.
On Thu, Feb 11, 2016 at 11:35 AM, Gary Brown <[email protected]> wrote: > FYI, the Nvidia K80 contains two K40 GPUs and they appear to the OS as two > separate GPUs, notwithstanding they are peers on the same board that can > communicate very quickly with each other. > > What users want for their jobs that request 2 GPUs is the ability to > schedule the two GPUs within a single K80 so the job's application can take > advantage of the fast peer-to-peer communication. > > Gary D. Brown > Adaptive Computing > > > On Thu, Feb 11, 2016 at 7:31 AM, Diego Zuccato <[email protected]> > wrote: > >> >> Il 11/02/2016 12:25, Michael Senizaiz ha scritto: >> >> > This doesn't enforce keeping the jobs on a K80. There are only 4 K80's >> > in the system. If I submit a 1 gpu job and a 2 gpu job after the first >> > will get GPU0 (0 and 1 are a K80, 2 and 3 are a K80, etc). The 2gpu job >> > will then get GPU 1 and GPU 2. Then the user will complain that their >> > peer-to-peer code isn't working and the job performance is bad because >> > they are running across two discreet K80's and not the 2 GPU's on a >> > single K80. >> Like allocating multithread jobs across different hosts. >> >> > gres.conf >> > NodeName=node[001-008] Name=gpu Type=k80 File=/dev/nvidia[0-7] >> CPUs=0-19 >> Shouldn't you have >> --8<-- >> NodeName=node[001-008] Name=Gpu Type=k80 File=/dev/nvidia[0-1] >> NodeName=node[001-008] Name=Gpu Type=k80 File=/dev/nvidia[2-3] >> NodeName=node[001-008] Name=Gpu Type=k80 File=/dev/nvidia[4-5] >> NodeName=node[001-008] Name=Gpu Type=k80 File=/dev/nvidia[6-7] >> --8<-- >> ? >> (I omitted CPUs since I don't know if in your case they're significant >> or not) >> IIUC, you should define each K80 as a different resource. But I started >> with SLURM about a week ago, so I could be way off target! >> HiH >> >> -- >> Diego Zuccato >> Servizi Informatici >> Dip. di Fisica e Astronomia (DIFA) - Università di Bologna >> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy >> tel.: +39 051 20 95786 >> mail: [email protected] >> > >
