Re: [OMPI users] Binding to thread 0
HIi Nathan, Isn't this a case for --map-by core --bind-to hwthread? Because you want to map each process by core but bind the the first hwthread. It is, but indicating a rankfile I can map to certain core numbers. I cannot use both --map-by and --rankfile: Conflicting directives for mapping policy are causing the policy to be redefined: New policy: RANK_FILE Prior policy: BYCORE Regards, L From the looks of it your process is both binding and mapping by hwthread now. -Nathan On Sep 11, 2023, at 10:20 AM, Luis Cebamanos via users wrote: @Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
Isn't this a case for --map-by core --bind-to hwthread? Because you want to map each process by core but bind the the first hwthread.From the looks of it your process is both binding and mapping by hwthread now. -NathanOn Sep 11, 2023, at 10:20 AM, Luis Cebamanos via users wrote:@Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote:In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread.From: users on behalf of Luis Cebamanos via users Sent: Friday, September 8, 2023 7:10 AM To: Ralph Castain via users Cc: Luis Cebamanos Subject: [OMPI users] Binding to thread 0 Hello,Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen?The command I am using is pretty simple:mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks likerank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36Thanks!
Re: [OMPI users] Binding to thread 0
@Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
Hi, @Gilles Awesome. --bind-to hwthread does the trick. I thought binding and mapping would not be read if using --rankfile. @Jeff I am running on a shared cluster. Some apps do use HT for achieving better performance. Nothing we could do with BIOS settings. Regards On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. From: users on behalf of Luis Cebamanos via users Sent: Friday, September 8, 2023 7:10 AM To: Ralph Castain via users Cc: Luis Cebamanos Subject: [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
Luis, you can pass the --bind-to hwthread option in order to bind on the first thread of each core Cheers, Gilles On Fri, Sep 8, 2023 at 8:30 PM Luis Cebamanos via users < users@lists.open-mpi.org> wrote: > Hello, > > Up to now, I have been using numerous ways of binding with wrappers > (numactl, taskset) whenever I wanted to play with core placing. Another way > I have been using is via -rankfile, however I notice that some ranks jump > from thread 0 to thread 1 on SMT chips. I can control this with numactl for > instance, but it would be great to see similar behaviour when using > -rankfile. Is there a way to pack all ranks to one of the threads of each > core (preferibly to thread 0) so I can nicely see all ranks with htop on > either left or right of the screen? > > The command I am using is pretty simple: > > mpirun -np $MPIRANKS --rankfile ./myrankfile > > and ./myrankfile looks like > > rank 33=argon slot=33 > rank 34=argon slot=34 > rank 35=argon slot=35 > rank 36=argon slot=36 > > Thanks! >
