We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having issues with lines scrambling together occasionally. This causes issues our convergence checking code since we put convergence data there. We are not using any mpirun options for stdout we just redirect stdout/stderr to a file before we run the mpirun command so all output goes there. We had similar issue with Intel MPI in the past and used the -ordered-output to fix it but I do not see any similar option for OpenMPI. See example below. Is there anyway to ensure a line from a process gets one line in the output file?
The data in red below is scrambled up and should look like the cleaned-up version. You can see it put a line from a different process inside a line from another processes and the rest of the line ended up a couple of lines down. ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 aerosurfs *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 aerosurfs *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 Aircraft-Total *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 1625 12 Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6 aft-Total *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06 Aircraft-OML *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 Aircraft-OML Cleaned up version: ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 1.004E-01 sec *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 aerosurfs *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 aerosurfs *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 -4.945E-06 Aircraft-Total *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 -2.785E-05 Aircraft-Total Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 1625 12 Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6 *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 -4.961E-06 Aircraft-OML *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 -2.791E-05 Aircraft-OML Thanks!
