On 11/16/26 7:45 AM, Suvarchal Kumar Cheedela wrote:
Hello there,

I would like to ask for comments on a change in how libgfortran performs
asynchronous WRITE: copy the output list into a runtime buffer when the
statement executes, so that the program can continue to modify its data
while the I/O thread writes it.  Patches exist and are tested; before
posting them I would like to agree on the user interface (environment
variable, option, or default) and on whether compression on the I/O
thread, which builds on it, would be welcome as a follow-up.

This is my first RFC to GCC, so I would also be grateful for advice on
how you would like the patches split and presented.

1. Use case: restarts and output of climate models
--------------------------------------------------

A climate model run of several centuries is a chain of batch jobs.  At the
end of every job, and often more frequently, each process writes its
complete state (a restart) so that the next job can continue.  At high
resolution a single restart is terabytes.  In between, the model writes
its science output (daily or monthly means).  What matters is throughput
in simulated years per day, and every second a process spends in a WRITE
comes straight out of it.

This is exactly what Fortran asynchronous I/O was designed for, and it has
been in the language since Fortran 2003.  In practice models do not use it.
FESOM2 (the AWI ocean model) and others carry their own threaded writers
in C or C++, or depend on libraries such as netCDF, PnetCDF, XIOS or the
HDF5 asynchronous VOL to get the write off the critical path.

The reason is the zero-copy implementation.  libgfortran queues a pointer
to the data, so the program must not change it until WAIT.  A model
changes its whole state in the next time step, so a conforming program
has to WAIT before that step, and nothing overlaps.

2. Measurements
---------------

FESOM2, CORE2 mesh (medium size short ocean simulations) , raw restart written with unformatted asynchronous
WRITE, measured on a workstation and on an HPC system.

a) Workstation (one node, local NVMe disk), time per process inside the
    restart routine:

      processes x threads    synchronous    asynchronous, copy
      2 x 4                  1.83 s         0.44 s
      4 x 2                  1.03 s         0.35 s
      8 x 1                  0.54 s         0.29 s

    The write lands in the page cache, so even the synchronous write is
    cheap and the absolute gain is fractions of a second.  This is
    probably why the limitation has not been an issue so far.

b) Levante (DKRZ, Lustre), 32 processes x 4 threads per node, restarts
    every 16 steps (much more often than in production, so that the cost
    is measurable in a short run).  Whole run loop in seconds, median of
    5 runs, with the range:

                             16 nodes                32 nodes
      no restart             3.65                    3.61
      synchronous WRITE      7.75  (5.37 - 13.70)   12.30  (4.50 - 14.22)
      asynchronous, copy     5.36  (4.90 -  6.40)    6.02  (5.55 -  8.96)

    There is no page cache to absorb the write: synchronous restarts
    doubled (16 nodes) or tripled (32 nodes) the run time, and varied by
    up to a factor of three between runs.  With the copy they cost
    47-67% and stay within a narrow range.

So the copy helps on both systems, but it matters on the parallel file
systems that large simulations run on.

Without the copy, FESOM2 overwrote its arrays while the I/O thread was
still writing them: with every intermediate restart kept, 146 of 896
restart files differed from those of the synchronous run.  With the copy
all of them are byte-identical.

3. Other compilers
------------------

On the same test programs (Intel ifx 2025.3, NVIDIA nvfortran 25.11):
ifx makes the same choice as gfortran, an I/O thread per unit and no
copy.  nvfortran completes the transfer inside the WRITE statement, which
is safe but not asynchronous.  The standard allows all of these, and a
conforming program cannot tell the difference.  So today no compiler
offers asynchronous output that a model can use without freezing its
state, and gfortran could be the first, it has a high impact in climate simulations in saving energy.

4. Prerequisite fixes
---------------------

Using asynchronous I/O in FESOM2 also exposed four bugs, now reported
with patches:

   PR128719  WAIT with ID= returns before the statement completes
   PR128720  defined I/O in an asynchronous transfer crashes
   PR128721  skipped items leak after an asynchronous I/O error
   PR128722  pointer arrays are transferred element by element
             (a FESOM2 restart took 120 s instead of 4 s)

   
https://inbox.sourceware.org/gcc-patches/[email protected]/

ifx crashes on the defined I/O case as well.

5. The proposal
---------------

Branch async-copy on https://github.com/suvarchal/gcc, on top of the fixes:

   - Copy on enqueue: an asynchronous WRITE copies its items into buffers
     from a pool owned by the unit and returns; the I/O thread writes and
     releases them.  The amount of outstanding copies per unit can be
     limited; above the limit the statement first waits for the unit.
   - Buffered asynchronous READ: data is read into a buffer and copied to
     the program's variables at WAIT.
   - -fno-async-volatile: ASYNCHRONOUS variables are currently compiled as
     VOLATILE, which stops vectorization of loops that use them (about 45%
     slower on a stencil kernel; ifx loses about 50% for the same reason).
     The option compiles them like other variables.  It also needs the
     READ transfer calls of an asynchronous statement to declare their data
     argument as escaping, which is part of the series.
   - Optionally, CPU placement of the I/O thread
     (GFORTRAN_ASYNC_IO_CPUS) and several threads for large copies.

These are separate patches and can be reviewed and taken one by one; the
copy for WRITE is the part that matters most.

A standard-conforming program behaves the same with and without the copy.
The cost is memory for the outstanding output and one memory copy in the
calling thread.

The question I would most like opinions on is how the copy is enabled:

   a) an environment variable (GFORTRAN_ASYNC_COPY=yes, as on the branch);
   b) a compiler option that sets a flag in the statement's parameter
      block, so that the compiled code and the runtime cannot disagree;
   c) by default for every asynchronous WRITE, with a memory limit and an
      environment variable to switch it off.

I would argue for (c).  Asynchronous WRITE is rarely used today, and the
copy turns its most common misuse (modifying the data before WAIT) from a
silent data race into the intended result.  The cost is one memory copy,
which is small compared with writing the data.

6. Next step: compression on the I/O thread
-------------------------------------------

Once the data is copied, the I/O thread can also compress it without
cost to the program.  On the same runs, zstd level 1 on the I/O thread
cost nothing extra at the median and made the restarts 37% smaller (1.07
to 0.67 GB per restart on 32 nodes); compressing synchronously made it
the slowest restart case.  Writing compressed data in fixed-size chunks
would also allow aligning the writes with the stripes or objects of
parallel file systems.  A prototype (zstd, zlib, lz4 through c-blosc2,
loaded with dlopen) is on the async-compress branch, but it involves a
file format, optional libraries and an interface, so I would send a
separate RFC for it after this one.

Questions
---------

   1. Is copying the data of asynchronous WRITE acceptable in libgfortran,
      and should it be the default (5c) or opt-in (5a/5b)?
   2. Is -fno-async-volatile acceptable, and should it be tied to the copy?
   3. Would you prefer one RFC for copy and compression together, or
      separate ones as above?
  4. As trunk is now in stage 3, as i see, should I aim as improvement for 17? or for GCC 18 stage 1. I think this path being only fortran has little risk? I am excited seeing potential computational savings and energy saving this enables.

Thanks,
Suvarchal

--
Suvarchal Kumar Cheedela
Alfred Wegener Institute (AWI)
[email protected]


I am looking at your patches via the links in the bug reports. Honestly we have never dealt with these volumes of data. Quite interesting.

If the patches were AI assisted they need to so state in the commit logs.

I will post further comments in the Bugzilla.

Regards,

Jerry

Reply via email to