Hi Jerry,  my comments below

On 11/16/26 19:50, Jerry D wrote:
On 11/16/26 7:45 AM, Suvarchal Kumar Cheedela wrote:
Hello there,

I would like to ask for comments on a change in how libgfortran performs
asynchronous WRITE: copy the output list into a runtime buffer when the
statement executes, so that the program can continue to modify its data
while the I/O thread writes it.  Patches exist and are tested; before
posting them I would like to agree on the user interface (environment
variable, option, or default) and on whether compression on the I/O
thread, which builds on it, would be welcome as a follow-up.

This is my first RFC to GCC, so I would also be grateful for advice on
how you would like the patches split and presented.

1. Use case: restarts and output of climate models
--------------------------------------------------

A climate model run of several centuries is a chain of batch jobs.  At the
end of every job, and often more frequently, each process writes its
complete state (a restart) so that the next job can continue. At high
resolution a single restart is terabytes.  In between, the model writes
its science output (daily or monthly means).  What matters is throughput
in simulated years per day, and every second a process spends in a WRITE
comes straight out of it.

This is exactly what Fortran asynchronous I/O was designed for, and it has been in the language since Fortran 2003.  In practice models do not use it.
FESOM2 (the AWI ocean model) and others carry their own threaded writers
in C or C++, or depend on libraries such as netCDF, PnetCDF, XIOS or the
HDF5 asynchronous VOL to get the write off the critical path.

The reason is the zero-copy implementation.  libgfortran queues a pointer
to the data, so the program must not change it until WAIT.  A model
changes its whole state in the next time step, so a conforming program
has to WAIT before that step, and nothing overlaps.

2. Measurements
---------------

FESOM2, CORE2 mesh (medium size short ocean simulations) , raw restart written with unformatted asynchronous
WRITE, measured on a workstation and on an HPC system.

a) Workstation (one node, local NVMe disk), time per process inside the
    restart routine:

      processes x threads    synchronous    asynchronous, copy
      2 x 4                  1.83 s         0.44 s
      4 x 2                  1.03 s         0.35 s
      8 x 1                  0.54 s         0.29 s

    The write lands in the page cache, so even the synchronous write is
    cheap and the absolute gain is fractions of a second.  This is
    probably why the limitation has not been an issue so far.

b) Levante (DKRZ, Lustre), 32 processes x 4 threads per node, restarts
    every 16 steps (much more often than in production, so that the cost
    is measurable in a short run).  Whole run loop in seconds, median of
    5 runs, with the range:

                             16 nodes                32 nodes
      no restart             3.65                    3.61
      synchronous WRITE      7.75  (5.37 - 13.70)   12.30  (4.50 - 14.22)       asynchronous, copy     5.36  (4.90 -  6.40)    6.02  (5.55 -  8.96)

    There is no page cache to absorb the write: synchronous restarts
    doubled (16 nodes) or tripled (32 nodes) the run time, and varied by
    up to a factor of three between runs.  With the copy they cost
    47-67% and stay within a narrow range.

So the copy helps on both systems, but it matters on the parallel file
systems that large simulations run on.

Without the copy, FESOM2 overwrote its arrays while the I/O thread was
still writing them: with every intermediate restart kept, 146 of 896
restart files differed from those of the synchronous run.  With the copy
all of them are byte-identical.

3. Other compilers
------------------

On the same test programs (Intel ifx 2025.3, NVIDIA nvfortran 25.11):
ifx makes the same choice as gfortran, an I/O thread per unit and no
copy.  nvfortran completes the transfer inside the WRITE statement, which
is safe but not asynchronous.  The standard allows all of these, and a
conforming program cannot tell the difference.  So today no compiler
offers asynchronous output that a model can use without freezing its
state, and gfortran could be the first, it has a high impact in climate simulations in saving energy.

4. Prerequisite fixes
---------------------

Using asynchronous I/O in FESOM2 also exposed four bugs, now reported
with patches:

   PR128719  WAIT with ID= returns before the statement completes
   PR128720  defined I/O in an asynchronous transfer crashes
   PR128721  skipped items leak after an asynchronous I/O error
   PR128722  pointer arrays are transferred element by element
             (a FESOM2 restart took 120 s instead of 4 s)

https://inbox.sourceware.org/gcc-patches/[email protected]/

ifx crashes on the defined I/O case as well.

5. The proposal
---------------

Branch async-copy on https://github.com/suvarchal/gcc, on top of the fixes:

   - Copy on enqueue: an asynchronous WRITE copies its items into buffers      from a pool owned by the unit and returns; the I/O thread writes and
     releases them.  The amount of outstanding copies per unit can be
     limited; above the limit the statement first waits for the unit.
   - Buffered asynchronous READ: data is read into a buffer and copied to
     the program's variables at WAIT.
   - -fno-async-volatile: ASYNCHRONOUS variables are currently compiled as      VOLATILE, which stops vectorization of loops that use them (about 45%      slower on a stencil kernel; ifx loses about 50% for the same reason).
     The option compiles them like other variables.  It also needs the
     READ transfer calls of an asynchronous statement to declare their data
     argument as escaping, which is part of the series.
   - Optionally, CPU placement of the I/O thread
     (GFORTRAN_ASYNC_IO_CPUS) and several threads for large copies.

These are separate patches and can be reviewed and taken one by one; the
copy for WRITE is the part that matters most.

A standard-conforming program behaves the same with and without the copy.
The cost is memory for the outstanding output and one memory copy in the
calling thread.

The question I would most like opinions on is how the copy is enabled:

   a) an environment variable (GFORTRAN_ASYNC_COPY=yes, as on the branch);
   b) a compiler option that sets a flag in the statement's parameter
      block, so that the compiled code and the runtime cannot disagree;
   c) by default for every asynchronous WRITE, with a memory limit and an
      environment variable to switch it off.

I would argue for (c).  Asynchronous WRITE is rarely used today, and the
copy turns its most common misuse (modifying the data before WAIT) from a
silent data race into the intended result.  The cost is one memory copy,
which is small compared with writing the data.

6. Next step: compression on the I/O thread
-------------------------------------------

Once the data is copied, the I/O thread can also compress it without
cost to the program.  On the same runs, zstd level 1 on the I/O thread
cost nothing extra at the median and made the restarts 37% smaller (1.07
to 0.67 GB per restart on 32 nodes); compressing synchronously made it
the slowest restart case.  Writing compressed data in fixed-size chunks
would also allow aligning the writes with the stripes or objects of
parallel file systems.  A prototype (zstd, zlib, lz4 through c-blosc2,
loaded with dlopen) is on the async-compress branch, but it involves a
file format, optional libraries and an interface, so I would send a
separate RFC for it after this one.

Questions
---------

   1. Is copying the data of asynchronous WRITE acceptable in libgfortran,
      and should it be the default (5c) or opt-in (5a/5b)?
   2. Is -fno-async-volatile acceptable, and should it be tied to the copy?
   3. Would you prefer one RFC for copy and compression together, or
      separate ones as above?
   4. As trunk is now in stage 3, as i see, should I aim as improvement for 17? or for GCC 18 stage 1. I think this path being only fortran has little risk? I am excited seeing potential computational savings and energy saving this enables.

Thanks,
Suvarchal

--
Suvarchal Kumar Cheedela
Alfred Wegener Institute (AWI)
[email protected]


I am looking at your patches via the links in the bug reports. Honestly we have never dealt with these volumes of data. Quite interesting.

Thank you for looking at the patches and for confirming the bugs.

The numbers in the RFC actually understate the problem.  They come from
a relatively coarse, ocean-only FESOM2 configuration used in last climate model intercomparision 10 years ago. A climate model couples several such components: atmosphere, ocean, sea ice, land, and often
waves and biogeochemistry.  Each writes restarts and output in the same
way, and each has the same I/O problem.

At the kilometre scale that the community is now moving to (and for high societal impact), a rough estimate for the atmosphere alone, at 1.25 km (about 335 million cells, 90 levels, around 10 prognostic 3-D fields in double precision), is about 2.4 TB per restart.  The ocean adds a similar amount.  For output, a single 3-D field in single precision is about 120 GB per output time, so ten 3-D fields once a day already give about 1.2 TB per simulated day, or more than 400 TB per simulated year.  While these are written, often the cores of the machine partition wait for different reasons, so the I/O directly limits throughput and wastes compute time on the HPC system.

It also affects how hardware is chosen.  Some centres move these models to GPUs partly for throughput, typically with OpenACC and a commercial compiler.  There the model state lives on the device, and for output it is copied to the host arrays (UPDATE HOST), which then act as the buffer: the device computes the next steps while the host writes. So GPU runs get buffered asynchronous output almost by construction.  The same code on CPUs cannot overlap its output with Fortran asynchronous I/O as implemented today, so part of the difference in throughput comes from I/O rather than from the CPUs or the language.  It bothers me that such a simple choice makes CPUs, and Fortran, look less scalable than they are.

Part of the community is also exploring other languages, such as Julia and Python, and machine-learning models, in part because of limitations like these.  Fortran already has the feature that is needed, asynchronous I/O.  An optional copy in the runtime is a small compromise, off by default, and it would let existing codes save both compute time and, with compression, disk space.
If the patches were AI assisted they need to so state in the commit logs.

Thanks for pointing this out.  Yes, AI tools were used.  For these patches the lines are blurry for me: I explored the problem a while ago and recently brought it to a working state with help from AI, I missed it as developers of FESOM ocean model have taken a stance against giving authorship to AI agents, as it claims part of copyright and that gets entangled with contributor licensing. I realized the guideline used by gcc may be reasonable compromise to indicate assisted-by, will use it in the future and propagate to our ocean model. I mainly used Qwen and Claude Opus for this set of patches.  I will add it in next version of patches.

I will post further comments in the Bugzilla.

Regards,

Jerry

Cheers,
Suvarchal

Reply via email to