Hello again,
Looking back, I was carried away by what this could do for climate
model restarts, output and compression all at once, and I did not state
my intent clearly; the proposal may have read as a way to make
non-conforming programs work. That is not my intention. My intention
is to make it simple to overlap computation with I/O, to save compute
time and energy. Since the first mail I have made the copy opt-in only
and left compression out. So let me try again with one narrower but
important case as an example: restarts, and what it currently takes to
overlap them with computation in a conforming program. Some
background: I have worked on climate models for many years, but
compilers are new ground for me. Part of my motivation is to close the
gap between model developers and compiler developers, and to show that
such collaboration is possible with open-source compilers, now also
with the help of AI tools. Please excuse any ignorance on my side.
The use case
- Climate and weather models have many components, such as land,
ocean and atmosphere, each with sub-components, and they typically
keep each component's state in derived types. As an example, the
restart of the ocean model FESOM2 consists of about 50 fields from
tens of modules; atmosphere components often have many more modules
and fields. The fields include array sections such as UV(1,:,:),
pointers and nested types, in different precisions such as REAL(4)
and REAL(8).
- A restart dumps that state as a binary image. Runs must restart
because of queue limits; in km-scale runs restarts are expensive in
storage and compute, and critical for debugging. They are also
essential for reproducibility: a state that is not restored
completely, or a default applied on initialization from a restart,
can take the model onto a different trajectory.
- Today this is usually a synchronous unformatted WRITE, and the model
waits for it. The goal i am exploring is to overlap that write with
the computation
of the following steps, which do not need the file; in that case the
savings are significant.
What it costs, and why it depends on the file system
- On a local disk, write() copies into the page cache and returns: a
synchronous WRITE of 8 MB takes 1.3 ms. So there is little to gain
there.
- On a parallel file system such as Lustre, write() returns once the
client cache takes the data, or blocks on the servers when its limit
is reached, depending on other jobs.
- Measured: FESOM2 alone at about 100 km (1.07 GB per restart) on the
Levante supercomputer runs at 173 simulated years per day (SYPD) on
16 nodes without restarts. With a restart every 16 steps (frequent,
so that the cost shows in a short run) it drops to 81 SYPD with
synchronous writes, -53% (median of 5 runs; -32% to -73%), and to
118 SYPD, -32% (-26% to -43%), when the data is copied at the WRITE
and written by a background thread. On 32 nodes: 175 SYPD without
restarts, 51 SYPD synchronous (-71%) and 105 SYPD copied (-40%).
The copied case still waits here only because 16 steps take less
time than writing the restart; with a longer interval the write is
hidden.
What the current asynchronous I/O allows (with the reported bugs fixed)
- WRITE(ASYNCHRONOUS='YES') returns at once, but the standard requires
that the data not change until the WAIT, and that every routine that
references it in the meantime gives it the ASYNCHRONOUS attribute.
A model step changes its whole state, so the WAIT has to come
before the next step and nothing overlaps. Even overlapping reads
would cost: gfortran compiles ASYNCHRONOUS variables as volatile,
which made a cache-resident stencil about 45% slower in my tests
(ifx +52%).
- The unit buffer does not change this: with asynchronous I/O it is
filled by the I/O thread after the statement returns, and it holds
128 KB, far less than a typical restart.
- What remains is double buffering in the application, as some of you
proposed: copy the state,
into buffers of its own, write those asynchronously, and WAIT at the
next restart. It is conforming and gives the overlap, but:
- every model has to write it again, with a buffer and packing code
for each component, section, pointer and nested type, and extend it
whenever a component is added; a field that is forgotten is a
restart that silently differs;
- it costs the same file-system time as a copy in the runtime, and
the buffers stay allocated between restarts unless the code
manages that too.
The compiler and runtime already walk all of these structures for
every WRITE (descriptors, scalarized components, defined I/O), so
this is a pattern every model repeats for what the runtime already
knows.
What I propose
- Overlap needs two things: a copy made at the statement, so that the
variable is free when the WRITE returns, and a thread that waits for
the file system. Today a synchronous WRITE has the first (for the
pieces that go through the unit buffer) and asynchronous I/O has the
second. The proposal combines them: in effect, a synchronous WRITE
into a large buffer whose flush runs on the I/O thread.
- An opt-in runtime setting, next to GFORTRAN_UNFORMATTED_BUFFER_SIZE:
for an unformatted WRITE, synchronous or asynchronous, the runtime
copies the data at the statement and returns, and a background
thread writes it. For an asynchronous WRITE this means the data is
no longer pending when the statement returns, so the WAIT returns at
once.
- Nothing is pending after the WRITE, so the program needs no
ASYNCHRONOUS attribute, no volatile and no WAIT. It also works for
defined I/O, which the standard does not(?) allow to be asynchronous.
Conforming programs only, no new syntax, the same
results with the option off and with every compiler.
- READ, FLUSH, CLOSE, INQUIRE of the size, WAIT and
program end drain the queue first, so the file is consistent wherever
a program can observe it.
- Trade-off: an I/O error is reported at the next of those statements
rather than at the WRITE, as already happens for data that sits in
the unit buffer when a WRITE returns.
Why in libgfortran, and why it is one pass
- The runtime already copies most of this data: pieces of up to 64 KB
(half the unit buffer) go through it, i.e. every element of a
section, pointer array or array component, every column of a 2-D
array (PR128796), and everything under GFORTRAN_CONVERT_UNIT --intrusive.
Only contiguous blocks over 64 KB go to write() directly. The option
adds a copy for
those, and makes the rest in one pass into its own buffer instead of
many small ones.
- It has the descriptors and sees the child WRITEs of defined I/O, so
one implementation serves every model, whatever its types.
- It also makes extensions simple. A plugin loaded under the program,
for example one that compresses, runs on the runtime's I/O thread
and reads the runtime's copy as it is passed to write(): it needs no
thread of its own and no copy of the data, only its output buffer.
Nothing of that needs to be in libgfortran.
I understand the bug fixes come first (PR128719-128722, PR128796).
Would such an
opt-in write-behind for unformatted output may be acceptable in
principle later or evaluated?
If so, I would prepare a new patch after those. I am happy to provide
measurements, test programs or more details for any of the points
above.
Regards,
Suvarchal
On 11/17/26 10:47, Janne Blomqvist wrote:
On Tue, Nov 17, 2026 at 4:33 AM Suvarchal K Cheedela
<[email protected]> wrote:
Hi Janne, Harald,
One more thought on below part, which I think after thought I did not
answer properly:
The cost of an asynchronous WRITE for a conforming program already
depends on the file system, and a copy is usually made anyway, only
not by libgfortran:
- On a local file system, write() copies the data into the page cache
and returns, so even the current asynchronous WRITE (with the bugs
fixed) costs one copy and the same amount of memory, held by the
kernel. On a workstation (ext4), a synchronous WRITE of a 24 MB
array took about 4-5 ms, about the same as a copy in user space.
The opt-in copy adds a second copy of the same size: more, but of
the same order as what the program pays already.
- On a parallel file system such as Lustre, the client caches dirty
pages up to a per-target limit and the servers cache as well. When
a write() blocks, and for how long, depends on those limits and on
the load from other jobs, none of which the program can see. The
run-to-run spread (4.9-17.5 s synchronous) most likely comes from
that.
So in both cases the program pays for buffering already; the question
is only who holds the buffer and whether the program can control when
it waits.
No? If you add an extra layer of buffering, it's yet another layer of
buffers that have to be copied. It does not replace the page cache,
which is still there. Unless your patch uses O_DIRECT (I haven't
looked) which is its own huge can of worms and will likely not work
well as a general purpose solution.
I also assume that HPC is of interest to Fortran/gfortran, given the
work on
OpenMP, OpenACC and coarrays. For HPC codes an opt-in copy makes
buffering a performance trade-off the user can select for their file
system and memory budget, rather than a cost imposed on every program
or something each application has to build itself.
The extra buffering is something a conforming program must do anyway,
if they want to keep accessing data which is in-progress for
asynchronous I/O. Either by implementing it in the application itself
or by using some library.
In general Fortran has a long and sordid history of vendors trying to
entice users by adding all kinds of extensions to their compilers,
creating a legacy of portability quagmires. Luckily that is mostly
behind us, but let's not go there again please. The very small
development community has enough troubles in implementing the
standardized features as it is. (With the caveat that I haven't been
an active GFortran developer for several years, so my comments in this
thread are more from the perspective of an interested bystander than
an official position of the development team!)