Maciej was in touch off-list, but just to give my reply here as well...
No, we didn't implement any time limits in sbank, as it would have meant modifying the slurm database schema, which we don't want to do! The best we can suggest is to have an external database of accounts, including start/end dates, and then have a separate procedure (e.g. a crontab) to query the external system and adjust the allocations in slurm. For example, you could have a `normal' QoS and a low-value `expired' QoS, and use the crontab to assign the `expired' QoS to accounts/jobs which have past their end date. Or if you just want to prevent jobs from running at all, you could delete the association. In our own setup, we are still using sbank, but have changed it slightly from the SUG2012 description. We found that without a PriorityDecayHalfLife that the fairshare parameter would tend to 0 for everyone, and so effectively became useless. So now we allow the usage to decay, and fairshare can play a part again in priorities. Note that these changes have been pushed to GitHub (https://github.com/jcftang/slurm-bank) This works reasonably well for us, but we are investigating other tweaks. We would actually like to emphasise accounting/reporting more than restricting use; we want to encourage as much usage of the systems as possible, and don't want project limits to prevent people from running jobs when the resources are idle. In our current sbank setup, people's `available balance' can still drop to 0 when their usage is high enough, and before the half-life decays it, and so with a 0 balance their jobs will wait in the queue until their usage decays enough. We are investigating the expired/normal QoS approach, to see if that will give us a better system usage. I think there's a friction between giving different groups (who may have contributed different amounts) fair usage of the system, but keeping the system utilisation high. My own preference is to move away from limits, but there could be a pressure from higher-up to have limits. :) I'd be happy to share our experiences with the QoS idea once I have something firm to report. Paddy On Tue, Apr 21, 2015 at 10:38:50AM -0700, Danny Auble wrote: > > Currently there isn't any straightforward way to expire a limit. > > Based on the talk you referenced it looks like expiry was something > they wanted to add. If you knew the end time of the project the > easiest way to make it so the account couldn't run is set the > GrpCPUMins=0 when the end time happens, or just remove the account > from Slurm. I am unaware of anything that does this automatically > with Slurm, but a cronjob running a script every day or so might > give you what you are looking for. > > > On 04/21/15 10:06, Danny Auble wrote: > >Have you looked at sreport? I think the Cluster > >UserUtilizationByAccount report would give you what you are > >looking for. > > > >On 04/21/15 05:53, Maciej L. Olchowik wrote: > >>Dear all, > >> > >>For our accounting needs, we are currently running slurmdbd with > >>the sbank scripts: > >>http://slurm.schedmd.com/slurm_ug_2012/SUG2012-TCHPC-sbank.pdf > >> > >>We find that this setup does almost everything we need, apart > >>from one thing. We need to be able to track core-hour > >>allocations on per project basis. For example, a project would > >>be allowed to run for 3 months and it's remaining core-hour > >>allocation will expire after that. Is it possible to set an > >>expiry date on the slurmdbd association (specifically we use > >>GrpCPUMins parameter)? > >> > >>We know that we can somewhat track allocations via: > >> sacctmgr list transactions > >> > >>but how do we keep track and enforce the expiry date? Does > >>anyone have a neat solution to this problem? > >> > >>many thanks for any pointers, > >> > >>Maciej > >> > >>-- > >>Maciej Olchowik > >>HPC Systems Administrator > >>KAUST Supercomputing Laboratory (KSL) > >>Al Khawarizmi Bldg. (1) Room 0134 > >>Thuwal, Kingdom of Saudi Arabia > >>tel +966 2 808 0684 > >> > >>________________________________ > >> > >>This message and its contents including attachments are intended > >>solely for the original recipient. If you are not the intended > >>recipient or have received this message in error, please notify > >>me immediately and delete this message from your computer > >>system. Any unauthorized use or distribution is prohibited. > >>Please consider the environment before printing this email. > > > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
