> -----Original Message----- > From: [email protected] [mailto:owner-slurm- > [email protected]] On Behalf Of Michael Di Domenico > Sent: Tuesday, April 19, 2011 12:26 PM > To: [email protected] > Subject: Re: [slurm-dev] JobComp scripts > > Thanks. > > We're using 2.2.4, just upgraded over the weekend. > > Right now i'm using the jobcomp script plugin to push the environment > variables into a far end database. we then have a dashboard'ish > webpage that shows the jobs run on the partitions, by whom, how long > they took, how many nodes, etc. (the webpage is an aggregate of all > the clusters on our network (since we use slurm everywhere))
You can get the same effect from the sview tool when only using 1 slurmdbd (the preferred way of operation for most sites). While not a handy webpage it does work cross cluster. Also using 1 slurmdbd will give your users the power to do multi-cluster options or stat information anywhere in the enterprise. All the SLURM user query tools and sbatch work cross-cluster. > > the newest instance of slurm (2.2.4) where we added > accounting/slurmdbd the mysql database is local to the slurm instance, > ideally, I'd like to just have my webpage query the slurm (via > perl/DBI) tables and pull out the necessary info. but if those > columns change that often perhaps that's not a good idea Yeah, it probably isn't a good idea, put you can use the perl apis in already in slurm would probably give you what you want. I know others have done this exact thing to do web pages which sound very similar to what you have going. > > On Tue, Apr 19, 2011 at 12:56 PM, Auble, Danny <[email protected]> wrote: > > > > > >> -----Original Message----- > >> From: [email protected] [mailto:owner-slurm- > >> [email protected]] On Behalf Of Michael Di Domenico > >> Sent: Tuesday, April 19, 2011 9:38 AM > >> To: [email protected] > >> Subject: Re: [slurm-dev] JobComp scripts > >> > >> Danny, we are using the accounting/slurmdbd module to push the > records > >> into a database (just starting with it), but it was/is not clear to > me > >> that this provides a duplication of the jobcomp information. If > >> that's the case, then yes, i could change the jobcomp/mysql plugin > to > >> jobcomp/script. > > > > Yes, the jobcomp is, outside of the fields you mentioned a small > subset of what the accounting plugin stores. There is not much sense > using both to write to the slurmdbd. > > > >> > >> Since we're on the accounting/slurmdbd subject, for a moment, i > >> noticed in the accounting table, there is/are a field missing. in > the > >> jobcomp/script output you're setting a "BATCH" environment variable > to > >> tell whether a program was produced as a batch or single, that > column > >> seems to be missing from the database table. it would also be nice > if > >> this table logged the clustername and what nodes were done on a > >> NODE_FAIL exit status as additional columns. > > > > I am not sure which version of SLURM you are using, but all job > tables in 2.2+ are prefixed with the name of the cluster, hence why it > isn't a column in the table. We don't store anything special about the > nodes in the job table outside of the nodes actually ran on, but you > could easily figure out which nodes had issues by using sacctmgr to > list the events that happened on those nodes at the end of the job. > > > > As for the batch flag, if a user uses sbatch or salloc there is a > flag set called track_steps in the database. If that isn't set the job > was started directly from srun. As for telling if it really was a > batch script or not, the only way to really tell if that was the case > it isn't fool proof, but if the user does specify the name of the job > with salloc the name of the job will be "allocation", you could query > against that to find batch jobs outside of interactive jobs, but if the > user states a job name that doesn't work. > > > > On another note, I am not sure how you are getting this data to your > other database, but it is probably a good idea to either use the C api, > perl api, or sacct[mgr] to get the data since that interface is pretty > static. If you are looking at the actual database tables they could > change often and would probably be difficult to keep up to sync. > Especially going from 2.1 to 2.2 where the entire database scheme > changed dramatically. > > > > Danny > > > >> > >> having these extra columns would eliminate my need for a script at > all > >> as i could sql pull the data into our site-wide tracking databases > >> > >> On Tue, Apr 19, 2011 at 11:59 AM, Danny Auble <[email protected]> wrote: > >> > Hey Michael, is there any reason you don't want to use the > >> > accounting_storage/slurmdbd plugin for accounting? It provides an > >> outrageous > >> > amount more information about each job and step than the basic > >> jobcomp > >> > plugin. It also allows the ability for fairshare scheduling and > >> limits if > >> > you want. Since you are already using the slurmdbd what would be > >> considered > >> > the hard part is already over. Also, if you just want straight job > >> records > >> > no real setup is required. > >> > > >> > That would leave the jobcomp/script open for your other database. > >> > > >> > Danny > >> > > >> >> Yes, i guess in my expedience I wasn't clear. We're using the > >> jobcomp > >> > > >> >> module with slurmdbd now, i want to continue using that, but also > >> run > >> > > >> >> a script. effectively having two JobCompType parameters > >> > > >> >> > >> > > >> >> On Tue, Apr 19, 2011 at 11:37 AM, Jette, Moe <[email protected]> > >> wrote: > >> > > >> >> > We do something like that today. Just configure > >> > > >> >> > > >> > > >> >> > JobCompType=jobcomp/script > >> > > >> >> > JobCompLoc=<pathname> > >> > > >> >> > > >> > > >> >> > Your script can easily write information into a database and do > >> anything > >> >> > else you want. > >> > > >> >> > > >> > > >> >> > ________________________________________ > >> > > >> >> > From: [email protected] [owner-slurm- > >> [email protected]] On > >> >> > Behalf Of Michael Di Domenico [[email protected]] > >> > > >> >> > Sent: Tuesday, April 19, 2011 8:21 AM > >> > > >> >> > To: slurm-dev > >> > > >> >> > Subject: [slurm-dev] JobComp scripts > >> > > >> >> > > >> > > >> >> > Is it possible to have the JobComp logic insert it's records > into > >> a > >> > > >> >> > database, but also run a script for each job run? > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > >> > > >> > > >> > > > > >
