Attached is a perl script a threw together yesterday to identity jobs that are
indicated as running in the slurmdbd, but not by the slurmctld.
Please feel free to use or modify as needed if you find it helpful.
Phil Eckert
LLNL.
From: Julien <jaulagn...@gmail.com<mailto:jaulagn...@gmail.com>>
Reply-To: slurm-dev <slurm-dev@schedmd.com<mailto:slurm-dev@schedmd.com>>
Date: Thursday, March 13, 2014 11:46 PM
To: slurm-dev <slurm-dev@schedmd.com<mailto:slurm-dev@schedmd.com>>
Subject: [slurm-dev] RE: error: We have more allocated time than is possible...
Hi Loris,
We have the same problem and we are also writing a script to "clean" the
database.
We are interesting by your code. Of course when our script will be written we
will also share it.
Thanks a lot.
Regards
--
Julien
2014-03-14 8:15 GMT+01:00 Loris Bennett
<loris.benn...@fu-berlin.de<mailto:loris.benn...@fu-berlin.de>>:
"Lipari, Don" <lipa...@llnl.gov<mailto:lipa...@llnl.gov>> writes:
> I wanted to add another reason (just discovered today) for the "We have more
> allocated time than is possible" error emitted to the slurmdbd.log.
>
> Disclaimer: I found this in an old version (v2.3.3) of Slurm, and can't
> confirm that the problem can still happen.
>
> The slurmctld submits job records to the slurmdbd in a one-way fashion.
> There is no code that validates jobs captured to the slurmdbd against jobs
> actually running. If for some reason, a job is completed by the slurmctld
> and a record of that event fails to make its way to the slurmdbd, the
> slurmdbd will consider that job still running - forever! Usage for this
> missing job will be rolled up and accounted as real usage for every hour
> after the job left the slurmctld's memory.
>
> As the number of these phantom, running jobs accumulates, the usage
> calculated for the cluster begins to exceed the theoretical maximum and the
> "We have more allocated time than is possible" error begins to appear every
> hour in the slurmdbd.log.
>
> The solution is to manually modify the job record in mysql to change its
> state from 1 to 3 and update time_end to something reasonable.
>
> Here's what I used to discover the jobs. Look for jobs with very old start
> times (or with excessive elapsed times) that are still running.
>
> sacct -a -X -S 1/1/13 -L -sR -o
> jobid,user,account,cluster,partition,nnodes,state,start,end,elapsed,exitcode
>
> Don
>
>> -----Original Message-----
>> From: Lipari, Don
>> Sent: Thursday, May 02, 2013 8:55 AM
>> To: slurm-dev
>> Subject: RE: [slurm-dev] error: We have more allocated time than is
>> possible...
>>
>> Have a look at http://bugs.schedmd.com/show_bug.cgi?id=247 and the patches
>> that resulted.
>> Don
>>
>> > -----Original Message-----
>> > From: Luis Alves [mailto:luis.al...@csc.fi<mailto:luis.al...@csc.fi>]
>> > Sent: Thursday, May 02, 2013 5:07 AM
>> > To: slurm-dev
>> > Subject: [slurm-dev] error: We have more allocated time than is
>> > possible...
>> >
>> >
>> > Hi all,
>> >
>> > Relating to this thread:
>> > http://comments.gmane.org/gmane.comp.distributed.slurm.devel/1755
>> >
>> > Is there any conclusion or mitigation solution that can be taken from
>> > that thread?
>> > In our case we are finding this lines in slurmdbd.log (1/hour
>> frequency):
>> >
>> > [2013-05-02T13:00:01] error: We have more allocated time than is
>> > possible (7033618 > 3211200) for cluster alcyone(892) from
>> > 2013-05-02T12:00:00 - 2013-05-02T13:00:00
>> > [2013-05-02T14:00:01] error: We have more allocated time than is
>> > possible (7351765 > 3211200) for cluster alcyone(892) from
>> > 2013-05-02T13:00:00 - 2013-05-02T14:00:00
>> >
>> > we're using:
>> >
>> > $ sacct -V
>> > slurm 2.4.3
>> >
>> > and I have no clue of what can be wrong.
>> > Can anybody please throw some light on this?
>> >
>> > Cheers,
>> >
>> > luis
>> >
>> > --
>> > Luï¿1Ž2s Alves
>> >
>> > ï¿1Ž2 System Specialist ï¿1Ž2 Computing Environments Group
>> > ï¿1Ž2 Finnish Grid Infrastucture (FGI) ï¿1Ž2ï¿1Ž2 NGI_FI
>> > ï¿1Ž2 Phone: +358 (0) 503 812 519 ï¿1Ž2ï¿1Ž2 http://www.csc.fi
>> > ï¿1Ž2 CSC - IT Center for Science, Ltd. ï¿1Ž2 Finland
We have experience this running 2.2.7. I have written a script which
will identify jobs in the database which have no end-time, but in fact a
no-longer running. There is a bit of logic to guess what an appropriate
end-time might be and then an SQL file is generated with which the
changes can be inserted into the database.
If anyone is interested, I can provide the code.
Regards
Loris
--
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email
loris.benn...@fu-berlin.de<mailto:loris.benn...@fu-berlin.de>
#! /usr/bin/perl
#
# Author: Phil Eckert
# Date: 03/14/2014
#
# A script to look for lost jobs.
# Jobs that are still listed as running
# in the slurmdbd, but no longer present
# in slurm.
#
use strict;
#
# set the time to right now.
#
my $date = `date +%H:%M`;
chomp $date;
#
# format for sacct output.
#
my $format = "-o 'JobID,User,JobName,Account,State,Cluster,NodeList,NNodes,Partition,Elapsed'";
#
# Get all the clusters we know about.
#
my @clusters = `sacctmgr show clusters -n format=cluster`;
#
# Loop through the clusters getting and comparing data.
#
foreach my $clust (@clusters) {
chomp $clust;
$clust =~ s/ //g;
printf("\nCLUSTER:$clust\n\n");
#
# I tried to use the "-s R" to capture only running jobs, but
# the output has more than just running jobs, so I just
# grep for RUNNING instead.
#
my @sjobs = `sacct -a -M $clust -S $date $format | grep RUNNING`;
my @jobs = `squeue -M $clust -o %i -h`;
#
# Compare running jobs in slurm with what the slurmdbd lists as running.
#
foreach my $j1 (@sjobs) {
chomp $j1;
my ($jobid) = ($j1 =~ m/(\S+) /);
#
# Ignore the steps.
#
next if ($jobid =~ /\./);
my $found = 0;
foreach my $j2 (@jobs) {
chomp $j2;
$j2 =~ s/ //g;
if ($jobid eq $j2) {
$found++;
last;
}
}
#
# Didn't find the job.
#
if (!$found) {
printf("cant find job $j1\n");
}
}
}
