Derek Gaston <fried...@gmail.com> writes: > Sorry, should read: "any one MPI process is not involved in more than ~2000 > *communicators*"
Yes, as intended. Only the ranks in a communicator's group need to know about the existence of that communicator. > Derek > > On Tue, Apr 3, 2018 at 11:47 AM Derek Gaston <fried...@gmail.com> wrote: > >> On Tue, Apr 3, 2018 at 10:31 AM Satish Balay <ba...@mcs.anl.gov> wrote: >> >>> On Tue, 3 Apr 2018, Derek Gaston wrote: >>> > Which does bring up a point: I have been able to do solves before with >>> > ~50,000 separate PETSc solves without issue. Is it because I was >>> working >>> > with MVAPICH on a cluster? Does it just have a higher limit? >>> >>> Don't know - but thats easy to find out with a simple test code.. >>> >> >> I get 2044 using mvapich on my cluster too. >> >> The only thing I can think of as to why those massive problems work for me >> is that any one MPI process is not involved in more than ~2000 processors >> (because the communicators are split as you go down the hierarchy). At >> most, a single MPI process will see ~hundreds of PETSc solves but not >> thousands. >> >> That said: it's just because of the current nature of the solves I'm doing >> - it's definitely possible to have that not be the case with MOOSE. >> >> Derek >>