Hi, There are a few things that you could test to see whether they make difference.
1. Try to modify the number of aggregators used in collective I/O (assuming that the code uses collective I/O). You could try e.g. to set it to the number of nodes used (the algorithm determining the number of aggregators automatically is sometimes overly aggressive). E.g. mpirun –mca io_ompio_num_aggregators 16 -np 256 ./executable name (assuming here that you run 256 processes distributed on 16 nodes). Based on our tests from a while back gpfs was not super sensitive to this, but you never know, its worth a try. 1. If your data is large and mostly contiguous, you could try to disable data sieving for write operations, e.g. mpirun --mca fbtl_posix_write_datasieving 0 -np 256 ./… Let me know if these make a difference. There are quite a couple of info objects that the gpfs fs component understands and that potentially could be used to tune the performance, but I do not have experience with them, they are based on code contributed by the HLRS a couple of years ago. You can still have a look at them and see whether some of them would make sense (source location: ompi/ompi/mca/fs/gpfs/fs_gpfs_file_set_info.c). Thanks Edgar From: users <users-boun...@lists.open-mpi.org> On Behalf Of Eric Chamberland via users Sent: Saturday, June 11, 2022 9:28 PM To: Open MPI Users <users@lists.open-mpi.org> Cc: Eric Chamberland <eric.chamberl...@giref.ulaval.ca>; Ramses van Zon <r...@scinet.utoronto.ca>; Vivien Clauzon <vivien.clau...@michelin.com>; dave.mar...@giref.ulaval.ca; Thomas Briffard <thomas.briff...@michelin.com> Subject: Re: [OMPI users] MPI I/O, Romio vs Ompio on GPFS Hi, just almost found what I wanted with "--mca io_base_verbose 100" Now I am looking at performances for GPFS and I must say OpenMPI 4.1.2 performs very poorly when it comes the time to write. I am launching a 512 processes, read+compute (ghosts components of a mesh), and then later write a 79Gb file. Here are the timings (all in seconds): -------------------- IO module ; reading+ghost computing ; writing ompio ; 24.9 ; 2040+ (job got killed before completion) romio321 ; 20.8 ; 15.6 -------------------- I have run many times the job with Ompio module (the default) and Romio and the timings are always similar to those given. I also activated maximum debug output with " --mca mca_base_verbose stdout,level:9 --mca mpi_show_mca_params all --mca io_base_verbose 100" and got a few lines but nothing relevant to debug: Sat Jun 11 20:08:28 2022<stdout>:chrono::ecritMaillageMPI::debut VmSize: 6530408 VmRSS: 5599604 VmPeak: 7706396 VmData: 5734408 VmHWM: 5699324 <etiq_143> Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:delete: deleting file: resultat01_-2.mail Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:delete: Checking all available modules Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:delete: component available: ompio, priority: 30 Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:delete: component available: romio321, priority: 10 Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:delete: Selected io component ompio Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:file_select: new file: resultat01_-2.mail Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:file_select: Checking all available modules Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:file_select: component available: ompio, priority: 30 Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:file_select: component available: romio321, priority: 10 Sat Jun 11 20:08:28 2022<stdout>:[nia0073.scinet.local:236683] io:base:file_select: Selected io module ompio What else can I do to dig into this? Are there parameters ompio is aware of with GPFS? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42 On 2022-06-10 16:23, Eric Chamberland via users wrote: Hi, I want to try romio with OpenMPI 4.1.2 because I am observing a big performance difference with IntelMPI on GPFS. I want to see, at *runtime*, all parameters (default values, names) used by MPI (at least for the "io" framework). I would like to have all the same output as "ompi_info --all" gives me... I have tried this: mpiexec --mca io romio321 --mca mca_verbose 1 --mca mpi_show_mca_params 1 --mca io_base_verbose 1 ... But I cannot see anything about io coming out... With "ompi_info" I do... Is it possible? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42
