http://www.schedmd.com/slurmdocs/troubleshoot.html
Quoting Mads Boye <[email protected]>: > Hi i upgraded my slurm installation to 2.5.3, and slurm is starting > again, but now all jobs stays in the completing state > > mb@bengal1:~$ squeue > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 10 bengal hellowor mb CG 0:00 1 bengal1 > 12 bengal hellowor mb CG 0:00 1 bengal2 > 15 bengal test2 jhs CG 0:00 1 bengal3 > 17 bengal hellowor mb CG 0:00 1 bengal4 > 9 lion ivan ibb CG 0:00 4 lion[1-4] > 13 lion ivan ibb CG 0:00 4 lion[5-8] > 14 lion ivan ibb CG 0:00 4 lion[9-12] > 16 munchkin hello zhzhang CG 0:00 1 munchkin1 > mb@bengal1:~$ > > I tired to mark the nodes as down and re run the job but no luck. > > De bedste hilsner / Best regards, > > Mads Boye. > >> -----Original Message----- >> From: Moe Jette [mailto:[email protected]] >> Sent: 19. februar 2013 17:27 >> To: slurm-dev; Mads Boye >> Subject: Re: [slurm-dev] Cannot submit jobs to slurm >> >> Look at your slurmd log file on bengal30 >> >> I would also recommend upgrading. Slurm v2.2 is two years old. >> >> Quoting Mads Boye <[email protected]>: >> >> > Hi. >> > I've got slurm-2.2.0 running, and until yesterday it worked fine. >> > When it try to run my test script "test.sh": >> > >> > #!/bin/sh >> > >> > ### Job name >> > #SBATCH --job-name=mads >> > ### Name of queue to submit to >> > ### #SBATCH --partition munchkin >> > ### Time limit of total run time of the job #SBATCH --time=2:00 ### >> > Number of nodes required for the job #SBATCH --nodes=1 ### Number of >> > processor cores #SBATCH --ntasks=1 ### Amount of memory per core >> > #SBATCH --mem-per-cpu=200 >> > >> > #SBATCH --output=slurm-%j.out >> > #SBATCH --error=slurm-%j.err >> > >> > >> > srun hostname >> > echo "Hello World" >> > >> > exit 0 >> > >> > I get the following error: >> > >> > srun: WARNING: We will use a much slower algorithm with >> > proctrack/pgid, use Proctracktype=proctrack/linuxproc or >> > Proctracktype=proctrack/rms with Job accounting gather LINUX plugin >> > srun: error: Task launch for 8900.0 failed on node bengal30: >> > Unspecified error >> > srun: error: Application launch failed: Unspecified error >> > srun: Job step aborted: Waiting up to 2 seconds for job step to finish. >> > srun: error: Timed out waiting for job step to complete Hello World >> > >> > I have the following partitions: >> > lion up infinite 1 alloc* lion1 >> > lion up infinite 19 alloc lion[2-12,14-21] >> > lion up infinite 4 idle lion[13,22-24] >> > bengal* up infinite 8 drain bengal[22-23,27-29,81-83] >> > bengal* up infinite 10 alloc bengal[20-21,24-26,44-45,78-80] >> > bengal* up infinite 65 idle bengal[1-19,30-43,46-77] >> > persian up infinite 16 idle persian[14-29] >> > munchkin up infinite 13 idle munchkin[1-13] >> > >> > The only think I've change since yesterday is that I've change >> > persian1-13 to be named munchkin[1-13] but I fail to see this being >> > the issue, as it also failes on the Bengal partition, which main. >> > >> > Best regards >> > >> > Mads. >> > >> >> > >
