Thanks Tim. We'll try out OMP_NUM_THREADS and see if we see a change.
We also do core binding for our jobs. The problem, in case you or others aren't aware, is that if a job asks e.g. for two cores and is thus bound to two cores (i.e. won't ever be able to use more than the two cores assigned to it), and the code it's running spawns more than 2 computational threads (because it sees e.g. 16 cores on the whole machine, or has some hard-coded number of threads to spawn), then the job will most likely run very inefficiently as all those threads are crammed onto the two cores, resulting in inefficient memory use as cpu cache misses make the cpu swap cache memory as each thread takes its time on the core. It'd be great to see a central way to control the # of computational threads, but I realize it might be impractical given the number of tools called by the scripts. -M On Fri, Jun 19, 2015 at 4:36 PM, Timothy Coalson <[email protected]> wrote: > Some operations in wb_command automatically use OpenMP without being > enabled by an argument, using however many cores it can find. You can > change the number it uses by setting the environment variable > OMP_NUM_THREADS. I'm not as sure about multithreading in other software > used in the scripts. > > As I understand it, in our cluster, we use the job control system to > restrict the number of cores the process can see, in order to reduce things > like cross-socket memory access. This won't help with a fixed number of > threads set in a script, but it does let us keep a single process > restricted to the amount of hardware we want it to use. > > Tim > > > On Fri, Jun 19, 2015 at 1:04 PM, m s <[email protected]> wrote: > >> Version 3.4.0 >> >> Hi, >> >> We're wondering if there's a clear global way to manage how many compute >> threads the HCP pipeline uses. We're running on a Rocks/SGE(OGE) cluster. >> >> My colleague recently ran the perfusion pre-processing pipeline with 9 >> slots/cores allocated via SGE/OGE and we observed the job's cpu usage on >> the compute node to which the job was assigned fluctuate in steps between >> 200%, 800% and 900%. >> >> We found one spot in an HCP script where the -openmp argument is >> explicitly called and was hardcoded to 8. We've changed that to be set to >> the SGE/OGE NSLOTS env var. But it seems there may be other spots where >> this is hardcoded to 8, or other spots where other threading options are >> set? >> >> Thanks, >> Michael >> >> _______________________________________________ >> HCP-Users mailing list >> [email protected] >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> > > _______________________________________________ HCP-Users mailing list [email protected] http://lists.humanconnectome.org/mailman/listinfo/hcp-users
