Hello Vaishak, hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)? Ideally if you could send the file configs/sim/config-info that would tell me. Yours, Roland > Dear Sir, > > I am a little worried about the performance because this is a new cluster > we have and it is supposed to be performing well. I am inclined to think > that some libraries/compiler options / settings might be the bottleneck. > > > I am presently running two simulations, both using the same parameter file > GW150914.rpar. > > The first one is using mpich-3.3.1, the same as in the simulation mentioned > in the previous thread. I am using one node consisting of 2*16 cores, and > 32 mpiprocs. > > The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, > distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I > have 32 PPN, it is launching 4 mpiprocs per node. > > I am herewith attaching the carpet-timing..asc file from both these runs. > > Thanking you > > Regards, > Vaishak > > > On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland <[email protected]> wrote: > > > Hello Vaishak, > > > > I do not see anything obviously wrong with the setup. > > > > It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 > > cores per node. > > > > Lookin at the timer tree output at iteration 1024 (search for > > "gettimeof " and you will find the spot) out of 5977s spend during > > Evolve about 2143s were spent in "syncs" which is communication and > > about the same amount of time in "thorns" that is doing computation. > > While this ratio is not great (spending more time sending data than > > doing computation) it is also not unheard of. > > > > Getting the original output files for the gallery data from Zenodo > > (link is on the gallery page): > > > > wget > > https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1 > > > > you can see (in GW150914_28/output-0000/GW150914_28.out) that that one > > took about 137s for syncs and 198s for thorns, so the same ratio but > > about a factor of 10 faster. > > > > I am reaching for straws here, but sometimes having too many MPI ranks > > can be detrimental if there is not enough work to split up (OpenMP can > > be a bit more forgiving in that respect, the original gallery run > > used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank). > > > > Since each node has lots of RAM (more than the 96GB required to run the > > simulation), can you try and see what would happen if you were to run > > on only a single node? > > > > Also if you could add the parameter: > > > > Carpet::output_timers_every = 1024 > > > > then provide the files carpet-timing-statistics*.asc that would let us > > know in even more detail where the time is spent. > > > > Running for a short time (2048 iterations) is enough to get data to > > compare. > > > > Yours, > > Roland > > > > > Dear All, > > > > > > I am running the simulation GW150914 using the parameter file available > > at > > > the ETK gallery at (GW150914-ETK gallery > > > <https://einsteintoolkit.org/gallery/bbh/index.html>) using 128 cores. > > > > > > Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R) > > Xeon(R) > > > Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running > > > Einstein Toolkit without OpenMP and using mpich-3.3.1. > > > > > > > > > The issue is that the simulation seems to be running at a very slow pace. > > > The number of physical time per hour that it is completing is only about > > > 1.3 units. At this rate to complete 1700 units, it would take about 54 > > > days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ > > > 2.40GHz) as per the details at the example run of GW150914 available at > > the > > > gallery (GW150914-ETK gallery > > > <https://einsteintoolkit.org/gallery/bbh/index.html>). > > > > > > I have also tried using intel mpi (impi) but with simular results. > > > > > > I am also attaching the out file from the simulation. > > > > > > Looking forward to your inputs. > > > > > > > > > Thanks and regards, > > > > > > > > > > > > > > > > > > Vaishak P > > > > > > PhD Scholar, > > > Shyama Prasad Mukherjee Fellow > > > Inter-University Center for Astronomy and Astrophysics (IUCAA) > > > Pune, India > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
pgpHyY3wG02Dt.pgp
Description: OpenPGP digital signature
_______________________________________________ Users mailing list [email protected] http://lists.einsteintoolkit.org/mailman/listinfo/users
