Am 19.11.26 um 1:46 AM schrieb Suvarchal K Cheedela:
What the current asynchronous I/O allows (with the reported bugs fixed)
- WRITE(ASYNCHRONOUS='YES') returns at once, but the standard requires
that the data not change until the WAIT, and that every routine that
references it in the meantime gives it the ASYNCHRONOUS attribute.
A model step changes its whole state, so the WAIT has to come
before the next step and nothing overlaps. Even overlapping reads
would cost: gfortran compiles ASYNCHRONOUS variables as volatile,
which made a cache-resident stencil about 45% slower in my tests
(ifx +52%).
Indeed. The current Fortran standard is pretty explicit:
"Execution of an asynchronous data transfer statement ... initiates a
pending data transfer operation. There may be multiple pending data
transfer operations for the same or multiple units simultaneously.
A pending data transfer operation remains pending until a corresponding
wait operation is performed. ..."
So as long as you did not WAIT(), you can execute additional data
transfer statements. If you want a library to handle multiple buffers,
how does the library know which buffer you mean for a data transfer?
I don't see anything in the standard that provides a means here.
But do you really save the model state at every timestep? That would
make no sense to me. Isn't it sufficient to save every n'th step?
And if it's just the time limit of the queue: you could have a signal
handler that is informed by the queuing system in advance before
reaching the limit to dump the model state. IIRC slurm can do this.
Furthermore, the ASYNCHRONOUS attribute should not be needed in the
driver program, only in the routines doing the asynchronous I/O.
Maybe you could have two (or more) allocatable versions of your model
state, which you rotate when you write the next restart file.
So likely no performance loss in the code. Would that help?
- The unit buffer does not change this: with asynchronous I/O it is
filled by the I/O thread after the statement returns, and it holds
128 KB, far less than a typical restart.
That is a historical value for a non-HPC system.
For an HPC filesystem, like Lustre, you want really large buffers.
The value for the I/O blocksize you get from stat is only a lower bound.
IIRC you don't get decent performance with Lustre below 4M or so, and
if you use striping you will want to go higher, 8M or 16M, or more.
Have you tried that? Or is there an issue with the handling of the
buffer size?
- What remains is double buffering in the application, as some of you
proposed: copy the state,
into buffers of its own, write those asynchronously, and WAIT at the
next restart. It is conforming and gives the overlap, but:
- every model has to write it again, with a buffer and packing code
for each component, section, pointer and nested type, and extend it
whenever a component is added; a field that is forgotten is a
restart that silently differs;
- it costs the same file-system time as a copy in the runtime, and
the buffers stay allocated between restarts unless the code
manages that too.
The compiler and runtime already walk all of these structures for
every WRITE (descriptors, scalarized components, defined I/O), so
this is a pattern every model repeats for what the runtime already
knows.
If your model is designed such that a model state is an instance of
a derived type that holds everything, how can you lose a variable?
(I have seen poorly designed codes with global fields. IMHO they are
badly broken and need fixing. OK, old codes from the Fortran 77 era
couldn't do it right...)
What I propose
- Overlap needs two things: a copy made at the statement, so that the
variable is free when the WRITE returns, and a thread that waits for
the file system. Today a synchronous WRITE has the first (for the
pieces that go through the unit buffer) and asynchronous I/O has the
second. The proposal combines them: in effect, a synchronous WRITE
into a large buffer whose flush runs on the I/O thread.
- An opt-in runtime setting, next to GFORTRAN_UNFORMATTED_BUFFER_SIZE:
for an unformatted WRITE, synchronous or asynchronous, the runtime
copies the data at the statement and returns, and a background
thread writes it. For an asynchronous WRITE this means the data is
no longer pending when the statement returns, so the WAIT returns at
once.
- Nothing is pending after the WRITE, so the program needs no
ASYNCHRONOUS attribute, no volatile and no WAIT. It also works for
defined I/O, which the standard does not(?) allow to be asynchronous.
Conforming programs only, no new syntax, the same
results with the option off and with every compiler.
- READ, FLUSH, CLOSE, INQUIRE of the size, WAIT and
program end drain the queue first, so the file is consistent wherever
a program can observe it.
- Trade-off: an I/O error is reported at the next of those statements
rather than at the WRITE, as already happens for data that sits in
the unit buffer when a WRITE returns.
Why in libgfortran, and why it is one pass
- The runtime already copies most of this data: pieces of up to 64 KB
(half the unit buffer) go through it, i.e. every element of a
section, pointer array or array component, every column of a 2-D
array (PR128796), and everything under GFORTRAN_CONVERT_UNIT --
intrusive.
Only contiguous blocks over 64 KB go to write() directly. The option
adds a copy for
those, and makes the rest in one pass into its own buffer instead of
many small ones.
- It has the descriptors and sees the child WRITEs of defined I/O, so
one implementation serves every model, whatever its types.
- It also makes extensions simple. A plugin loaded under the program,
for example one that compresses, runs on the runtime's I/O thread
and reads the runtime's copy as it is passed to write(): it needs no
thread of its own and no copy of the data, only its output buffer.
Nothing of that needs to be in libgfortran.
What you propose is something that does not seem to fit into a general
Fortran runtime library.
But are you saying that there is something that does not scale well when
changing the GFORTRAN_UNFORMATTED_BUFFER_SIZE? Then this would be
really interesting to know, and we might need to look into it.
Testcases are always helpful.
(Note: when I mentioned NetCDF or HDF5 recently, I did not mean to use
the parallel features of these libraries. These might eat to much
performance for your use case. Just serial R/W on each rank, taking
care that HDF5 file locking and other nuisances don't hit you.)
I understand the bug fixes come first (PR128719-128722, PR128796). Would
such an
opt-in write-behind for unformatted output may be acceptable in
principle later or evaluated?
If so, I would prepare a new patch after those. I am happy to provide
measurements, test programs or more details for any of the points
above.
I think Jerry is already looking into your bug reports, and I dare say
we really appreciate them!
Regards,
Suvarchal
Thanks,
Harald