This usually happens if we get a number of warning message from multiple processes. Seems like unbuffered is what we want but not sure how this interacts with MPI since stdout/stderr is pulled back from different hosts. Not sure how you are doing that.
-----Original Message----- From: Joachim Protze <pro...@itc.rwth-aachen.de> Sent: Monday, December 06, 2021 11:12 AM To: Fisher (US), Mark S <mark.s.fis...@boeing.com>; Open MPI Users <users@lists.open-mpi.org> Subject: Re: [OMPI users] stdout scrambled in file I would assume, that the buffering mode is compiler/runtime specific. At least for Intel compiler, the default seems to be/have been unbuffered for stdout, but there is a flag for buffered output: https://community.intel.com/t5/Intel-Fortran-Compiler/Enabling-buffered-I-O-to-stdout-with-Intel-ifort-compiler/td-p/993203 In the worst case, each character might be written individually. If the scrambling only happens from time to time, I guess you really just see the buffer flush when the buffer filled up. - Joachim Am 06.12.21 um 16:42 schrieb Fisher (US), Mark S: > All strings are writing as one output so that is not the issue. Adding in > some flushing is a good idea and we can try that. We do not open stdout just > write to unit 6, but we could open it if there is some un-buffered option > that could help. I will look into that also. Thanks! > > -----Original Message----- > From: Joachim Protze <pro...@itc.rwth-aachen.de> > Sent: Monday, December 6, 2021 9:24 AM > To: Open MPI Users <users@lists.open-mpi.org> > Cc: Fisher (US), Mark S <mark.s.fis...@boeing.com> > Subject: Re: [OMPI users] stdout scrambled in file > > Hi Mark, > > "[...] MPI makes neither requirements nor recommendations for the output > [...]" (MPI4.0, §2.9.1) > > From my experience, an application can avoid such scrambling (still no > guarantee), if the output of lines is written atomically. C++ streams > are worst for concurrent output, as every stream operator writes a > chunk. It can help to collect output into a stringstream and print out > at once. Using printf in C is typically least problematic. Flushing the > buffer (fflush) helpts to avoid that the output buffer fills up and is > flushed in the middle of printing. > > I'm not the Fortran expert. But, I think there are some options to > change to a buffered output mode (at least I found such options for file > I/O). Again, the goal should be that a write statement is printed at > once and the buffer doesn't fill up while printing. > > In any case, it could help to write warnings to stderr and separate the > stdout and stderr streams. > > Best > Joachim > > Am 02.12.21 um 16:48 schrieb Fisher (US), Mark S via users: >> We are using Mellanox HPC-X MPI based on OpenMPI 4.1.1RC1 and having >> issues with lines scrambling together occasionally. This causes issues >> our convergence checking code since we put convergence data there. We >> are not using any mpirun options for stdout we just redirect >> stdout/stderr to a file before we run the mpirun command so all output >> goes there. We had similar issue with Intel MPI in the past and used the >> -ordered-output to fix it but I do not see any similar option for >> OpenMPI. See example below. Is there anyway to ensure a line from a >> process gets one line in the output file? >> >> *The data in red below is scrambled up and should look like the >> cleaned-up version. You can see it put a line from a different process >> inside a line from another processes and the rest of the line ended up a >> couple of lines down.* >> >> ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 >> 1.004E-01 sec >> >> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 >> -4.945E-06 aerosurfs >> >> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 >> -2.785E-05 aerosurfs >> >> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 >> -4.945E-06 Aircraft-Total >> >> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 >> -2.785E-05 Aircr Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 >> 1625 12 >> >> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6 >> >> aft-Total >> >> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 >> -4.961E-06 Aircraft-OML >> >> *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 >> -2.791E-05 Aircraft-OML >> >> *Cleaned up version:* >> >> ZONE 0 : Min/Max CFL= 5.000E-01 1.500E+01 Min/Max DT= 8.411E-10 >> 1.004E-01 sec >> >> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 >> -4.945E-06 aerosurfs >> >> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 >> -2.785E-05 aerosurfs >> >> *IGSTAB* 1626 7.392E-02 2.470E-01 -9.075E-04 8.607E-03 -5.911E-04 >> -4.945E-06 Aircraft-Total >> >> *IGMNTAERO* 1626 -6.120E-04 1.406E-02 6.395E-04 4.473E-08 3.112E-04 >> -2.785E-05 Aircraft-Total >> >> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 699 1625 12 >> >> Warning: BCFD: US_UPDATEQ: izon, iter, nBadpmin: 111 1626 6 >> >> *IGSTAB* 1626 6.623E-02 2.137E-01 -9.063E-04 8.450E-03 -5.485E-04 >> -4.961E-06 Aircraft-OML >> >> *IGMNTAERO* 1626 -6.118E-04 -1.602E-02 6.404E-04 5.756E-08 3.341E-04 >> -2.791E-05 Aircraft-OML >> >> Thanks! >> > > -- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 pro...@itc.rwth-aachen.de www.itc.rwth-aachen.de
