We tried to do this originally with cpusets but had issues. Rather than continuing on in that direction we now use a system reservation where we can assign the first X cores. This reservation basically just holds those X cores for an infinite time period allowing any slurm jobs to use only the remaining cores. This way cgroups work to contain users' jobs onto the correct memories and cores while allowing the first X cores to be used for system function.
Not the most elegant or protected way to contain system processes but it has been working for us for years.
Bill
