On Wed, Feb 3, 2021 at 2:42 PM Luciano Siqueira <[email protected]> wrote:
> Hello, > > I'm evaluating the performance of an application in a distributed > environment and I notice that it's much slower when running in many > nodes/cores when compared to a single node with a fewer cores. > > When running the application in 20 nodes, the Main Stage time reported > in PETSc's log is up to 10 times slower than it is when running the same > application in only 1 node, even with fewer cores per node. > > The application I'm running is an example code provided by libmesh: > > http://libmesh.github.io/examples/introduction_ex4.html > > The application runs inside a Singularity container, with openmpi-4.0.3 > and PETSc 3.14.3. The distributed processes are managed by slurm > 17.02.11 and each node is equipped with two Intel CPU Xeon E5-2695v2 Ivy > Bridge (12c @2,4GHz) and 128Gb of RAM, all communications going through > infiniband. > > My questions are: Is the slowdown expected? Should the application be > specially tailored to work well in distributed environments? > > Also, where (maybe in PETSc documentation/source-code) can I find > information on how PETSc handles MPI communications? Do the KSP solvers > favor one-to-one process communication over broadcast messages or > vice-versa? I suspect inter-process communication must be the cause of > the poor performance when using many nodes, but not as much as I'm seeing. > > Thank you in advance! > We can't say anything about the performance without some data. Please send us the output of -log_view for both cases. Thanks, Matt > Luciano. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
