Re: [petsc-users] prevent linking to multithreaded BLAS?

[Jed Brown]

Barry Smith  writes:

>> We could test at runtime whether child threads exist/are created when 
>> calling BLAS and deliver a warning. 
>
>   How does one test for this? Some standard Unix API for checking this?

I'm not sure, the ids of child threads are in /proc/$pid/task/ and (when opened 
by a process) /proc/self/task/. See man procfs for details.

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Barry Smith]

> On Dec 7, 2022, at 11:56 PM, Jed Brown  wrote:
> 
> It isn't always wrong to link threaded BLAS. For example, a user might need 
> to call threaded BLAS on the side (but the application can only link one) or 
> a sparse direct solver might want threading for the supernode.

  Indeed, the user asked specifically for their work flow if configure could, 
based on additional configure argument, ensure that they did not get a threaded 
BLAS; they did not ask that configure never give a threaded BLAS or even that 
it give a non-threaded BLAS by default.


> We could test at runtime whether child threads exist/are created when calling 
> BLAS and deliver a warning. 

  How does one test for this? Some standard Unix API for checking this?

> 
> Barry Smith  writes:
> 
>>   There would need to be, for example, some symbol in all the threaded BLAS 
>> libraries that is not in the unthreaded libraries. Of at least in some of 
>> the threaded libraries but never in the unthreaded. 
>> 
>>   BlasLapack.py could check for the special symbol(s) to determine.
>> 
>>  Barry
>> 
>> 
>>> On Dec 7, 2022, at 4:47 PM, Mark Lohry  wrote:
>>> 
>>> Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and 
>>> the --download-fblaslapack way occurred to me. 
>>> 
>>> I was hoping for something that "just worked" (refuse to build in this 
>>> case) but I don't know if it's programmatically possible for petsc to tell 
>>> whether or not it's linking to a threaded BLAS?
>>> 
>>> On Wed, Dec 7, 2022 at 4:35 PM Satish Balay >> > wrote:
 If you don't specify a blas to use - petsc configure will guess and use 
 what it can find.
 
 So only way to force it use a particular blas is to specify one [one way 
 is --download-fblaslapack]
 
 Wrt multi-thread openblas -  you can force it run single threaded [by one 
 of these 2 env variables]
 
# Use single thread openblas
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
 
 Satish
 
 
 On Wed, 7 Dec 2022, Mark Lohry wrote:
 
> I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
> my application was launching NP threads, such that when running with
> multiple ranks the machine was quickly oversubscribed and performance
> tanked.
> 
> The root cause of this was petsc linking against the system-provided
> library (libopenblas0-pthread in this case) set by the update-alternatives
> in ubuntu. At some point this machine got updated to using the threaded
> blas implementation instead of serial; not sure how, and I wouldn't have
> noticed if I weren't running interactively.
> 
> Is there any mechanism in petsc or its build system to prevent linking
> against an inappropriate BLAS, or do I need to be diligent about manually
> setting the BLAS library in the configuration stage?
> 
> Thanks,
> Mark
> 
 

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Jed Brown]

It isn't always wrong to link threaded BLAS. For example, a user might need to 
call threaded BLAS on the side (but the application can only link one) or a 
sparse direct solver might want threading for the supernode. We could test at 
runtime whether child threads exist/are created when calling BLAS and deliver a 
warning. 

Barry Smith  writes:

>There would need to be, for example, some symbol in all the threaded BLAS 
> libraries that is not in the unthreaded libraries. Of at least in some of the 
> threaded libraries but never in the unthreaded. 
>
>BlasLapack.py could check for the special symbol(s) to determine.
>
>   Barry
>
>
>> On Dec 7, 2022, at 4:47 PM, Mark Lohry  wrote:
>> 
>> Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and 
>> the --download-fblaslapack way occurred to me. 
>> 
>> I was hoping for something that "just worked" (refuse to build in this case) 
>> but I don't know if it's programmatically possible for petsc to tell whether 
>> or not it's linking to a threaded BLAS?
>> 
>> On Wed, Dec 7, 2022 at 4:35 PM Satish Balay > > wrote:
>>> If you don't specify a blas to use - petsc configure will guess and use 
>>> what it can find.
>>> 
>>> So only way to force it use a particular blas is to specify one [one way is 
>>> --download-fblaslapack]
>>> 
>>> Wrt multi-thread openblas -  you can force it run single threaded [by one 
>>> of these 2 env variables]
>>> 
>>> # Use single thread openblas
>>> export OPENBLAS_NUM_THREADS=1
>>> export OMP_NUM_THREADS=1
>>> 
>>> Satish
>>> 
>>> 
>>> On Wed, 7 Dec 2022, Mark Lohry wrote:
>>> 
>>> > I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
>>> > my application was launching NP threads, such that when running with
>>> > multiple ranks the machine was quickly oversubscribed and performance
>>> > tanked.
>>> > 
>>> > The root cause of this was petsc linking against the system-provided
>>> > library (libopenblas0-pthread in this case) set by the update-alternatives
>>> > in ubuntu. At some point this machine got updated to using the threaded
>>> > blas implementation instead of serial; not sure how, and I wouldn't have
>>> > noticed if I weren't running interactively.
>>> > 
>>> > Is there any mechanism in petsc or its build system to prevent linking
>>> > against an inappropriate BLAS, or do I need to be diligent about manually
>>> > setting the BLAS library in the configuration stage?
>>> > 
>>> > Thanks,
>>> > Mark
>>> > 
>>> 

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Barry Smith]

   There would need to be, for example, some symbol in all the threaded BLAS 
libraries that is not in the unthreaded libraries. Of at least in some of the 
threaded libraries but never in the unthreaded. 

   BlasLapack.py could check for the special symbol(s) to determine.

  Barry


> On Dec 7, 2022, at 4:47 PM, Mark Lohry  wrote:
> 
> Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and 
> the --download-fblaslapack way occurred to me. 
> 
> I was hoping for something that "just worked" (refuse to build in this case) 
> but I don't know if it's programmatically possible for petsc to tell whether 
> or not it's linking to a threaded BLAS?
> 
> On Wed, Dec 7, 2022 at 4:35 PM Satish Balay  > wrote:
>> If you don't specify a blas to use - petsc configure will guess and use what 
>> it can find.
>> 
>> So only way to force it use a particular blas is to specify one [one way is 
>> --download-fblaslapack]
>> 
>> Wrt multi-thread openblas -  you can force it run single threaded [by one of 
>> these 2 env variables]
>> 
>> # Use single thread openblas
>> export OPENBLAS_NUM_THREADS=1
>> export OMP_NUM_THREADS=1
>> 
>> Satish
>> 
>> 
>> On Wed, 7 Dec 2022, Mark Lohry wrote:
>> 
>> > I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
>> > my application was launching NP threads, such that when running with
>> > multiple ranks the machine was quickly oversubscribed and performance
>> > tanked.
>> > 
>> > The root cause of this was petsc linking against the system-provided
>> > library (libopenblas0-pthread in this case) set by the update-alternatives
>> > in ubuntu. At some point this machine got updated to using the threaded
>> > blas implementation instead of serial; not sure how, and I wouldn't have
>> > noticed if I weren't running interactively.
>> > 
>> > Is there any mechanism in petsc or its build system to prevent linking
>> > against an inappropriate BLAS, or do I need to be diligent about manually
>> > setting the BLAS library in the configuration stage?
>> > 
>> > Thanks,
>> > Mark
>> > 
>> 

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Mark Lohry]

Thanks, yes, I figured out the OMP_NUM_THREADS=1 way while triaging it, and
the --download-fblaslapack way occurred to me.

I was hoping for something that "just worked" (refuse to build in this
case) but I don't know if it's programmatically possible for petsc to tell
whether or not it's linking to a threaded BLAS?

On Wed, Dec 7, 2022 at 4:35 PM Satish Balay  wrote:

> If you don't specify a blas to use - petsc configure will guess and use
> what it can find.
>
> So only way to force it use a particular blas is to specify one [one way
> is --download-fblaslapack]
>
> Wrt multi-thread openblas -  you can force it run single threaded [by one
> of these 2 env variables]
>
> # Use single thread openblas
> export OPENBLAS_NUM_THREADS=1
> export OMP_NUM_THREADS=1
>
> Satish
>
>
> On Wed, 7 Dec 2022, Mark Lohry wrote:
>
> > I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
> > my application was launching NP threads, such that when running with
> > multiple ranks the machine was quickly oversubscribed and performance
> > tanked.
> >
> > The root cause of this was petsc linking against the system-provided
> > library (libopenblas0-pthread in this case) set by the
> update-alternatives
> > in ubuntu. At some point this machine got updated to using the threaded
> > blas implementation instead of serial; not sure how, and I wouldn't have
> > noticed if I weren't running interactively.
> >
> > Is there any mechanism in petsc or its build system to prevent linking
> > against an inappropriate BLAS, or do I need to be diligent about manually
> > setting the BLAS library in the configuration stage?
> >
> > Thanks,
> > Mark
> >
>
>

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Satish Balay via petsc-users]

If you don't specify a blas to use - petsc configure will guess and use what it 
can find.

So only way to force it use a particular blas is to specify one [one way is 
--download-fblaslapack]

Wrt multi-thread openblas -  you can force it run single threaded [by one of 
these 2 env variables]

# Use single thread openblas
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1

Satish


On Wed, 7 Dec 2022, Mark Lohry wrote:

> I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
> my application was launching NP threads, such that when running with
> multiple ranks the machine was quickly oversubscribed and performance
> tanked.
> 
> The root cause of this was petsc linking against the system-provided
> library (libopenblas0-pthread in this case) set by the update-alternatives
> in ubuntu. At some point this machine got updated to using the threaded
> blas implementation instead of serial; not sure how, and I wouldn't have
> noticed if I weren't running interactively.
> 
> Is there any mechanism in petsc or its build system to prevent linking
> against an inappropriate BLAS, or do I need to be diligent about manually
> setting the BLAS library in the configuration stage?
> 
> Thanks,
> Mark
> 

Re: [petsc-users] prevent linking to multithreaded BLAS?

[Barry Smith]

  We don't have configure code to detect if the BLAS is thread parallel, nor do 
we have code to tell it not to use a thread parallel version. 

  Except if it is using MKL then we do force it to not use the threaded BLAS.

  A "cheat" would be for you to just set the environmental variable BLAS uses 
for number of threads to 1 always, then you would not need to worry about 
checking to avoid the "bad" library.

  Barry




> On Dec 7, 2022, at 4:21 PM, Mark Lohry  wrote:
> 
> I ran into an unexpected issue -- on an NP-core machine, each MPI rank of my 
> application was launching NP threads, such that when running with multiple 
> ranks the machine was quickly oversubscribed and performance tanked.
> 
> The root cause of this was petsc linking against the system-provided library 
> (libopenblas0-pthread in this case) set by the update-alternatives in ubuntu. 
> At some point this machine got updated to using the threaded blas 
> implementation instead of serial; not sure how, and I wouldn't have noticed 
> if I weren't running interactively.
> 
> Is there any mechanism in petsc or its build system to prevent linking 
> against an inappropriate BLAS, or do I need to be diligent about manually 
> setting the BLAS library in the configuration stage?
> 
> Thanks,
> Mark

[petsc-users] prevent linking to multithreaded BLAS?

[Mark Lohry]

I ran into an unexpected issue -- on an NP-core machine, each MPI rank of
my application was launching NP threads, such that when running with
multiple ranks the machine was quickly oversubscribed and performance
tanked.

The root cause of this was petsc linking against the system-provided
library (libopenblas0-pthread in this case) set by the update-alternatives
in ubuntu. At some point this machine got updated to using the threaded
blas implementation instead of serial; not sure how, and I wouldn't have
noticed if I weren't running interactively.

Is there any mechanism in petsc or its build system to prevent linking
against an inappropriate BLAS, or do I need to be diligent about manually
setting the BLAS library in the configuration stage?

Thanks,
Mark