On 28/02/17 23:17, David Radice wrote: > Hello Eloisa, > > sorry for the delay in the reply. For the records I did manage to > compile and run ET on KNL (stampede), but I did not manage to run any > benchmark with it yet. The current status is: > > * intel-17: the compiler fails to compile Carpet and either gives an > internal error or segfaults. * gcc-6.3: used to compile and run with > Erik's spack installation (it is currently broken). I did not really > manage to benchmark it since even a low-resolution TOV test did not > run to completion (meaning less than 4 coarse grid steps) within 30 > minutes on 4 nodes. > > This was using the current stable release of the ET (2016-11) and > WhiskyTHC. You might have more luck with GRHydro / pure-vacuum runs.
Hi David and all, thanks for all the help. It turned out that consolidating my configuration made things significantly better: I was using Intel 16 (to avoid the Carpet problem with Intel 17) along with a strange mix of libraries (mostly compiled with Intel 17, and the only available on Marconi), and that seemed to impact the performance quite strongly. With everything Intel 17 (and using -no-vec on bbox.cc), I now obtain a runspeed on a Marconi KNL node which is around 80% of a Xeon E5 v4. There are still some puzzling features, though. One is that using -no-vec, along with the settings: VECTORISE = no VECTORISE_ALIGNED_ARRAYS = no VECTORISE_INLINE = no VECTORISE_ALIGN_FOR_CACHE = no VECTORISE_ALIGN_INTERIOR = no in my optionlist, I obtain essentially the same throughput. This is a vacuum McLachlan run with very little else turned on (but I can run a QC0 benchmark for definiteness, if people are interested). I too am using the November release. Second, hyperthreading decreases the runspeed significantly. I am using 272 threads on the 68-core KNL, and for what I can gather from the Carpet output, all of the cores are engaged. More cores are reported, however, than available on the node: INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 1 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 272 threads, this is thread 0 INFO (Carpet): There are 272 threads in total INFO (Carpet): There are 272 threads per process INFO (Carpet): This process runs on host r098c04s01, pid=2465840 INFO (Carpet): This process runs on 272 cores: 0-271 INFO (Carpet): Thread 0 runs on 1 core: 0 INFO (Carpet): Thread 1 runs on 1 core: 68 INFO (Carpet): Thread 2 runs on 1 core: 136 INFO (Carpet): Thread 3 runs on 1 core: 204 … Notice that I am requesting hyperthreading by using num-smt=4 and num-threads=272. Is this correct? Thanks again, Eloisa _______________________________________________ Users mailing list [email protected] http://lists.einsteintoolkit.org/mailman/listinfo/users
