Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package golang-github-vpenso-prometheus_slurm_exporter for openSUSE:Leap:16.0 checked in at 2025-05-16 08:30:03 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Leap:16.0/golang-github-vpenso-prometheus_slurm_exporter (Old) and /work/SRC/openSUSE:Leap:16.0/.golang-github-vpenso-prometheus_slurm_exporter.new.30101 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "golang-github-vpenso-prometheus_slurm_exporter" Fri May 16 08:30:03 2025 rev:1 rq:1210980 version:0.20 Changes: -------- New Changes file: --- /dev/null 2025-04-02 01:05:25.176000000 +0200 +++ /work/SRC/openSUSE:Leap:16.0/.golang-github-vpenso-prometheus_slurm_exporter.new.30101/golang-github-vpenso-prometheus_slurm_exporter.changes 2025-05-16 08:30:04.631522138 +0200 @@ -0,0 +1,70 @@ +------------------------------------------------------------------- +Tue Jun 27 11:19:12 UTC 2023 - Egbert Eich <e...@suse.com> + +- Adjust-GPU-data-gathering-to-work-with-all-Slurm-versions-since-18.08.patch + Fix to make GPU data gathering works with all Slurm versions since 18.08. +- Add a sysconfig settings to make daemon command line args easily. +- Fix service file names to match upstream: + prometheus-slurm_exporter.service -> prometheus-slurm-exporter.service + rcprometheus-slurm_exporter -> rcprometheus-slurm-exporter + To support updates, the old name is still kept as sym-link. + Add README.SUSE to inform about sysconfig file. + +------------------------------------------------------------------- +Thu Jun 16 11:35:34 UTC 2022 - ecsos <ec...@opensuse.org> + +- Update to 0.20 + * Multiple README fix. + * Improve build process. + +------------------------------------------------------------------- +Thu Jul 22 13:23:12 UTC 2021 - Egbert Eich <e...@suse.com> + +- Update to version 0.19 + * GPUs accounting has to be activated explicitly via cmd line option. + * Export detailed usage info for every node (CPU, Memory). + NOTE: With the present version of Slurm (20.11), GPU accounting + in the prometheus-slurm-exporter will cause the exporter to + terminate, thus it must not be enabled for the time being. +- Do not ship sources. + +------------------------------------------------------------------- +Thu Mar 18 15:04:01 UTC 2021 - Ana Guerrero Lopez <aguerr...@suse.com> + +- New version 0.17 + * Export information about shares via sshare +- New version 0.16 + * Add support to provide information about GPU GREs usage + +------------------------------------------------------------------- +Wed Dec 2 12:51:19 UTC 2020 - Ana Guerrero Lopez <aguerr...@suse.com> + +- New version 0.15 + * Metric updates + * Bugfixes +- Upstream now provides go.mod, update packaging accordingly. + +------------------------------------------------------------------- +Wed Dec 18 16:54:26 UTC 2019 - Ana Guerrero Lopez <aguerr...@suse.com> + +- List all source files. + +------------------------------------------------------------------- +Wed Dec 11 13:52:46 UTC 2019 - Ana Guerrero Lopez <aguerr...@suse.com> + +- Remove grafana-dashboards and do not add the dashboard in the tarball, + its license needs to be clarified. It's also easier for users + to install the board from grafana.com + +------------------------------------------------------------------- +Fri Dec 6 10:21:32 UTC 2019 - Ana Guerrero Lopez <aguerr...@suse.com> + +- Add new package with grafana-dashboards +- Add README.packaging +- Document use_sbin.patch + +------------------------------------------------------------------- +Mon Dec 2 11:57:46 UTC 2019 - Ana Guerrero Lopez <aguerr...@suse.com> + +- Initial release. + New: ---- Adjust-GPU-data-gathering-to-work-with-all-Slurm-versions-since-18.08.patch README.SUSE _service golang-github-vpenso-prometheus_slurm_exporter-0.20.tar.gz golang-github-vpenso-prometheus_slurm_exporter.changes golang-github-vpenso-prometheus_slurm_exporter.spec use_sbin.patch vendor.tar.gz BETA DEBUG BEGIN: New: - Adjust-GPU-data-gathering-to-work-with-all-Slurm-versions-since-18.08.patch Fix to make GPU data gathering works with all Slurm versions since 18.08. New:- Add README.packaging - Document use_sbin.patch BETA DEBUG END: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ golang-github-vpenso-prometheus_slurm_exporter.spec ++++++ # # spec file for package golang-github-vpenso-prometheus_slurm_exporter # # Copyright (c) 2021 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed # upon. The license for this file, and modifications and additions to the # file, is the same license as for the pristine package itself (unless the # license for the pristine package is not an Open Source License, in which # case the license is the MIT License). An "Open Source License" is a # license that conforms to the Open Source Definition (Version 1.9) # published by the Open Source Initiative. # Please submit bugfixes or comments via https://bugs.opensuse.org/ # %{go_nostrip} Name: golang-github-vpenso-prometheus_slurm_exporter Version: 0.20 Release: 0 Summary: Prometheus exporter for Slurm metrics License: GPL-3.0-or-later Group: System/Management URL: https://github.com/vpenso/prometheus-slurm-exporter Source0: https://github.com/vpenso/prometheus-slurm-exporter/archive/%{version}.tar.gz#/%{name}-%{version}.tar.gz Source1: vendor.tar.gz Source2: README.SUSE Patch0: use_sbin.patch Patch1: Adjust-GPU-data-gathering-to-work-with-all-Slurm-versions-since-18.08.patch BuildRequires: golang-packaging %{?systemd_requires} Requires(pre): shadow %{go_provides} Requires: /usr/bin/sinfo Requires: /usr/bin/squeue %description Prometheus collector and exporter for metrics extracted from the Slurm resource scheduling system. %prep %setup -q -n prometheus-slurm-exporter-%{version} %setup -q -n prometheus-slurm-exporter-%{version} -T -D -a 1 %autopatch -p1 %build %{goprep} github.com/vpenso/prometheus-slurm-exporter %{gobuild} -mod=vendor "" %install %{goinstall} # No %%{gosrc} install -D -m 0644 lib/systemd/prometheus-slurm-exporter.service %{buildroot}%{_unitdir}/prometheus-slurm-exporter.service sed -i -e '/ExecStart/s@^\(.*\)$@\1 $PROMETHEUS_SLURM_EXPORTER_ARGS@' \ -e '/\[Service\]/a EnvironmentFile=-%{_sysconfdir}/sysconfig/prometheus-slurm-exporter' \ %{buildroot}%{_unitdir}/prometheus-slurm-exporter.service # To handle service file rename ln -sf prometheus-slurm-exporter.service %{buildroot}%{_unitdir}/prometheus-slurm_exporter.service install -m 0755 -d %{buildroot}%{_fillupdir} echo -e '# prometheus-slurm-exporter args\n'\ '# possible values: -gpus-acct, -listen-address string\n'\ 'PROMETHEUS_SLURM_EXPORTER_ARGS=""' > \ %{buildroot}%{_fillupdir}/sysconfig.prometheus-slurm-exporter mv %{buildroot}%{_bindir}/ %{buildroot}%{_sbindir}/ ln -s %{_sbindir}/service %{buildroot}%{_sbindir}/rcprometheus-slurm-exporter cp %{S:2} . %{gofilelist} %pre %service_add_pre prometheus-slurm-exporter.service getent group prometheus >/dev/null || %{_sbindir}/groupadd -r prometheus getent passwd prometheus >/dev/null || %{_sbindir}/useradd -r -g prometheus -d %{_localstatedir}/lib/prometheus -M -s /sbin/nologin prometheus %post %service_add_post prometheus-slurm-exporter.service %{fillup_only -n prometheus-slurm-exporter} %preun %service_del_preun prometheus-slurm-exporter.service %postun %service_del_postun prometheus-slurm-exporter.service %files -f file.lst %license LICENSE %doc README.md %doc %{basename:cp %{S:2}} %{_fillupdir}/sysconfig.prometheus-slurm-exporter %{_sbindir}/prometheus-slurm-exporter %{_unitdir}/prometheus-slurm-exporter.service %{_unitdir}/prometheus-slurm_exporter.service %{_sbindir}/rcprometheus-slurm-exporter %changelog ++++++ Adjust-GPU-data-gathering-to-work-with-all-Slurm-versions-since-18.08.patch ++++++ From: Egbert Eich <e...@suse.com> Date: Tue Jun 27 13:09:51 2023 +0200 Subject: Adjust GPU data gathering to work with all Slurm versions since 18.08 Patch-mainline: Not yet Git-commit: dad0d76a32c784e658573a0025d423df610ab9e1 References: These changes have been ported from the development branch of https://github.com/vpenso/prometheus-slurm-exporter. Signed-off-by: Egbert Eich <e...@suse.com> Signed-off-by: Egbert Eich <e...@suse.de> --- gpus.go | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++++------------ node.go | 19 ++++---- 2 files changed, 147 insertions(+), 41 deletions(-) diff --git a/gpus.go b/gpus.go index ca3bcaf..7db5ab3 100644 --- a/gpus.go +++ b/gpus.go @@ -1,4 +1,4 @@ -/* Copyright 2020 Joeri Hermans, Victor Penso, Matteo Dessalvi +/* Copyright 2022 Joeri Hermans, Victor Penso, Matteo Dessalvi, Iztok Lebar Bajec This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by @@ -16,17 +16,19 @@ along with this program. If not, see <http://www.gnu.org/licenses/>. */ package main import ( - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/common/log" - "io/ioutil" "os/exec" - "strings" + "regexp" "strconv" + "strings" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/common/log" ) type GPUsMetrics struct { alloc float64 idle float64 + other float64 total float64 utilization float64 } @@ -35,6 +37,11 @@ func GPUsGetMetrics() *GPUsMetrics { return ParseGPUsMetrics() } +/* TODO: + sinfo has gresUSED since slurm>=19.05.0rc01 https://github.com/SchedMD/slurm/blob/master/NEWS + revert to old process on slurm<19.05.0rc01 + --format=AllocGRES will return gres/gpu=8 + --format=AllocTRES will return billing=16,cpu=16,gres/gpu=8,mem=256G,node=1 func ParseAllocatedGPUs() float64 { var num_gpus = 0.0 @@ -53,21 +60,106 @@ func ParseAllocatedGPUs() float64 { return num_gpus } +*/ -func ParseTotalGPUs() float64 { +func ParseAllocatedGPUs(data []byte) float64 { var num_gpus = 0.0 + // sinfo -a -h --Format="Nodes: ,GresUsed:" --state=allocated + // 3 gpu:2 # slurm>=20.11.8 + // 1 gpu:(null):3(IDX:0-7) # slurm 21.08.5 + // 13 gpu:A30:4(IDX:0-3),gpu:Q6K:4(IDX:0-3) # slurm 21.08.5 + + sinfo_lines := string(data) + re := regexp.MustCompile(`gpu:(\(null\)|[^:(]*):?([0-9]+)(\([^)]*\))?`) + if len(sinfo_lines) > 0 { + for _, line := range strings.Split(sinfo_lines, "\n") { + // log.info(line) + if len(line) > 0 && strings.Contains(line, "gpu:") { + nodes := strings.Fields(line)[0] + num_nodes, _ := strconv.ParseFloat(nodes, 64) + node_active_gpus := strings.Fields(line)[1] + num_node_active_gpus := 0.0 + for _, node_active_gpus_type := range strings.Split(node_active_gpus, ",") { + if strings.Contains(node_active_gpus_type, "gpu:") { + node_active_gpus_type = re.FindStringSubmatch(node_active_gpus_type)[2] + num_node_active_gpus_type, _ := strconv.ParseFloat(node_active_gpus_type, 64) + num_node_active_gpus += num_node_active_gpus_type + } + } + num_gpus += num_nodes * num_node_active_gpus + } + } + } - args := []string{"-h", "-o \"%n %G\""} - output := string(Execute("sinfo", args)) - if len(output) > 0 { - for _, line := range strings.Split(output, "\n") { - if len(line) > 0 { - line = strings.Trim(line, "\"") - descriptor := strings.Fields(line)[1] - descriptor = strings.TrimPrefix(descriptor, "gpu:") - descriptor = strings.Split(descriptor, "(")[0] - node_gpus, _ := strconv.ParseFloat(descriptor, 64) - num_gpus += node_gpus + return num_gpus +} + +func ParseIdleGPUs(data []byte) float64 { + var num_gpus = 0.0 + // sinfo -a -h --Format="Nodes: ,Gres: ,GresUsed:" --state=idle,allocated + // 3 gpu:4 gpu:2 # slurm 20.11.8 + // 1 gpu:8(S:0-1) gpu:(null):3(IDX:0-7) # slurm 21.08.5 + // 13 gpu:A30:4(S:0-1),gpu:Q6K:40(S:0-1) gpu:A30:4(IDX:0-3),gpu:Q6K:4(IDX:0-3) # slurm 21.08.5 + + sinfo_lines := string(data) + re := regexp.MustCompile(`gpu:(\(null\)|[^:(]*):?([0-9]+)(\([^)]*\))?`) + if len(sinfo_lines) > 0 { + for _, line := range strings.Split(sinfo_lines, "\n") { + // log.info(line) + if len(line) > 0 && strings.Contains(line, "gpu:") { + nodes := strings.Fields(line)[0] + num_nodes, _ := strconv.ParseFloat(nodes, 64) + node_gpus := strings.Fields(line)[1] + num_node_gpus := 0.0 + for _, node_gpus_type := range strings.Split(node_gpus, ",") { + if strings.Contains(node_gpus_type, "gpu:") { + node_gpus_type = re.FindStringSubmatch(node_gpus_type)[2] + num_node_gpus_type, _ := strconv.ParseFloat(node_gpus_type, 64) + num_node_gpus += num_node_gpus_type + } + } + num_node_active_gpus := 0.0 + node_active_gpus := strings.Fields(line)[2] + for _, node_active_gpus_type := range strings.Split(node_active_gpus, ",") { + if strings.Contains(node_active_gpus_type, "gpu:") { + node_active_gpus_type = re.FindStringSubmatch(node_active_gpus_type)[2] + num_node_active_gpus_type, _ := strconv.ParseFloat(node_active_gpus_type, 64) + num_node_active_gpus += num_node_active_gpus_type + } + } + num_gpus += num_nodes * (num_node_gpus - num_node_active_gpus) + } + } + } + + return num_gpus +} + +func ParseTotalGPUs(data []byte) float64 { + var num_gpus = 0.0 + // sinfo -a -h --Format="Nodes: ,Gres:" + // 3 gpu:4 # slurm 20.11.8 + // 1 gpu:8(S:0-1) # slurm 21.08.5 + // 13 gpu:A30:4(S:0-1),gpu:Q6K:40(S:0-1) # slurm 21.08.5 + + sinfo_lines := string(data) + re := regexp.MustCompile(`gpu:(\(null\)|[^:(]*):?([0-9]+)(\([^)]*\))?`) + if len(sinfo_lines) > 0 { + for _, line := range strings.Split(sinfo_lines, "\n") { + // log.Info(line) + if len(line) > 0 && strings.Contains(line, "gpu:") { + nodes := strings.Fields(line)[0] + num_nodes, _ := strconv.ParseFloat(nodes, 64) + node_gpus := strings.Fields(line)[1] + num_node_gpus := 0.0 + for _, node_gpus_type := range strings.Split(node_gpus, ",") { + if strings.Contains(node_gpus_type, "gpu:") { + node_gpus_type = re.FindStringSubmatch(node_gpus_type)[2] + num_node_gpus_type, _ := strconv.ParseFloat(node_gpus_type, 64) + num_node_gpus += num_node_gpus_type + } + } + num_gpus += num_nodes * num_node_gpus } } } @@ -77,29 +169,40 @@ func ParseTotalGPUs() float64 { func ParseGPUsMetrics() *GPUsMetrics { var gm GPUsMetrics - total_gpus := ParseTotalGPUs() - allocated_gpus := ParseAllocatedGPUs() + total_gpus := ParseTotalGPUs(TotalGPUsData()) + allocated_gpus := ParseAllocatedGPUs(AllocatedGPUsData()) + idle_gpus := ParseIdleGPUs(IdleGPUsData()) + other_gpus := total_gpus - allocated_gpus - idle_gpus gm.alloc = allocated_gpus - gm.idle = total_gpus - allocated_gpus + gm.idle = idle_gpus + gm.other = other_gpus gm.total = total_gpus gm.utilization = allocated_gpus / total_gpus return &gm } +func AllocatedGPUsData() []byte { + args := []string{"-a", "-h", "--Format=Nodes: ,GresUsed:", "--state=allocated"} + return Execute("sinfo", args) +} + +func IdleGPUsData() []byte { + args := []string{"-a", "-h", "--Format=Nodes: ,Gres: ,GresUsed:", "--state=idle,allocated"} + return Execute("sinfo", args) +} + +func TotalGPUsData() []byte { + args := []string{"-a", "-h", "--Format=Nodes: ,Gres:"} + return Execute("sinfo", args) +} + // Execute the sinfo command and return its output func Execute(command string, arguments []string) []byte { cmd := exec.Command(command, arguments...) - stdout, err := cmd.StdoutPipe() + out, err := cmd.CombinedOutput() if err != nil { log.Fatal(err) } - if err := cmd.Start(); err != nil { - log.Fatal(err) - } - out, _ := ioutil.ReadAll(stdout) - if err := cmd.Wait(); err != nil { - log.Fatal(err) - } return out } @@ -111,9 +214,10 @@ func Execute(command string, arguments []string) []byte { func NewGPUsCollector() *GPUsCollector { return &GPUsCollector{ - alloc: prometheus.NewDesc("slurm_gpus_alloc", "Allocated GPUs", nil, nil), - idle: prometheus.NewDesc("slurm_gpus_idle", "Idle GPUs", nil, nil), - total: prometheus.NewDesc("slurm_gpus_total", "Total GPUs", nil, nil), + alloc: prometheus.NewDesc("slurm_gpus_alloc", "Allocated GPUs", nil, nil), + idle: prometheus.NewDesc("slurm_gpus_idle", "Idle GPUs", nil, nil), + other: prometheus.NewDesc("slurm_gpus_other", "Other GPUs", nil, nil), + total: prometheus.NewDesc("slurm_gpus_total", "Total GPUs", nil, nil), utilization: prometheus.NewDesc("slurm_gpus_utilization", "Total GPU utilization", nil, nil), } } @@ -121,6 +225,7 @@ func NewGPUsCollector() *GPUsCollector { type GPUsCollector struct { alloc *prometheus.Desc idle *prometheus.Desc + other *prometheus.Desc total *prometheus.Desc utilization *prometheus.Desc } @@ -129,6 +234,7 @@ type GPUsCollector struct { func (cc *GPUsCollector) Describe(ch chan<- *prometheus.Desc) { ch <- cc.alloc ch <- cc.idle + ch <- cc.other ch <- cc.total ch <- cc.utilization } @@ -136,6 +242,7 @@ func (cc *GPUsCollector) Collect(ch chan<- prometheus.Metric) { cm := GPUsGetMetrics() ch <- prometheus.MustNewConstMetric(cc.alloc, prometheus.GaugeValue, cm.alloc) ch <- prometheus.MustNewConstMetric(cc.idle, prometheus.GaugeValue, cm.idle) + ch <- prometheus.MustNewConstMetric(cc.other, prometheus.GaugeValue, cm.other) ch <- prometheus.MustNewConstMetric(cc.total, prometheus.GaugeValue, cm.total) ch <- prometheus.MustNewConstMetric(cc.utilization, prometheus.GaugeValue, cm.utilization) } diff --git a/node.go b/node.go index bf2f759..ae7dc90 100644 --- a/node.go +++ b/node.go @@ -27,12 +27,12 @@ import ( // NodeMetrics stores metrics for each node type NodeMetrics struct { - memAlloc uint64 - memTotal uint64 - cpuAlloc uint64 - cpuIdle uint64 - cpuOther uint64 - cpuTotal uint64 + memAlloc uint64 + memTotal uint64 + cpuAlloc uint64 + cpuIdle uint64 + cpuOther uint64 + cpuTotal uint64 nodeStatus string } @@ -60,7 +60,6 @@ func ParseNodeMetrics(input []byte) map[string]*NodeMetrics { memAlloc, _ := strconv.ParseUint(node[1], 10, 64) memTotal, _ := strconv.ParseUint(node[2], 10, 64) - cpuInfo := strings.Split(node[3], "/") cpuAlloc, _ := strconv.ParseUint(cpuInfo[0], 10, 64) cpuIdle, _ := strconv.ParseUint(cpuInfo[1], 10, 64) @@ -82,7 +81,7 @@ func ParseNodeMetrics(input []byte) map[string]*NodeMetrics { // NodeData executes the sinfo command to get data for each node // It returns the output of the sinfo command func NodeData() []byte { - cmd := exec.Command("sinfo", "-h", "-N", "-O", "NodeList,AllocMem,Memory,CPUsState,StateLong") + cmd := exec.Command("sinfo", "-h", "-N", "-O", "NodeList: ,AllocMem: ,Memory: ,CPUsState: ,StateLong:") out, err := cmd.Output() if err != nil { log.Fatal(err) @@ -102,7 +101,7 @@ type NodeCollector struct { // NewNodeCollector creates a Prometheus collector to keep all our stats in // It returns a set of collections for consumption func NewNodeCollector() *NodeCollector { - labels := []string{"node","status"} + labels := []string{"node", "status"} return &NodeCollector{ cpuAlloc: prometheus.NewDesc("slurm_node_cpu_alloc", "Allocated CPUs per node", labels, nil), @@ -128,7 +127,7 @@ func (nc *NodeCollector) Collect(ch chan<- prometheus.Metric) { nodes := NodeGetMetrics() for node := range nodes { ch <- prometheus.MustNewConstMetric(nc.cpuAlloc, prometheus.GaugeValue, float64(nodes[node].cpuAlloc), node, nodes[node].nodeStatus) - ch <- prometheus.MustNewConstMetric(nc.cpuIdle, prometheus.GaugeValue, float64(nodes[node].cpuIdle), node, nodes[node].nodeStatus) + ch <- prometheus.MustNewConstMetric(nc.cpuIdle, prometheus.GaugeValue, float64(nodes[node].cpuIdle), node, nodes[node].nodeStatus) ch <- prometheus.MustNewConstMetric(nc.cpuOther, prometheus.GaugeValue, float64(nodes[node].cpuOther), node, nodes[node].nodeStatus) ch <- prometheus.MustNewConstMetric(nc.cpuTotal, prometheus.GaugeValue, float64(nodes[node].cpuTotal), node, nodes[node].nodeStatus) ch <- prometheus.MustNewConstMetric(nc.memAlloc, prometheus.GaugeValue, float64(nodes[node].memAlloc), node, nodes[node].nodeStatus) ++++++ README.SUSE ++++++ The Prometheus Slurm exporter currenly accepts two command line options: -gpus-acct Enable GPUs accounting -listen-address string The port to listen on for HTTP requests. (default ":8080") These can be set by changing the PROMETHEUS_SLURM_EXPORTER_ARGS setting in /etc/sysconfig/prometheus_slurm_exporter. ++++++ _service ++++++ <services> <service name="go_modules" mode="disabled"> <param name="archive">golang-github-vpenso-prometheus_slurm_exporter-0.20.tar.gz</param> </service> </services> ++++++ use_sbin.patch ++++++ From: Ana Guerrero López <aguerr...@suse.com> Date: 2019-12-04 10:20:20 +0200 Subject: Install the exporter under /usr/sbin Upstream: tbd Install the exporter binary under /usr/sbin/ to keep the same logic than golang-github-prometheus-node_exporter and other exporters. diff -Nrua a/lib/systemd/prometheus-slurm-exporter.service b/lib/systemd/prometheus-slurm-exporter.service --- a/lib/systemd/prometheus-slurm-exporter.service +++ b/lib/systemd/prometheus-slurm-exporter.service @@ -2,7 +2,7 @@ Description=Prometheus SLURM Exporter [Service] -ExecStart=/usr/bin/prometheus-slurm-exporter +ExecStart=/usr/sbin/prometheus-slurm-exporter Restart=always RestartSec=15