https://gcc.gnu.org/bugzilla/show_bug.cgi?id=127893
Bug ID: 127893
Summary: [14 regression] Severe libgfortran parallel-I/O
performance regression on macOS/Apple Silicon between
GCC 13 and GCC 14
Product: gcc
Version: 14.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: libfortran
Assignee: unassigned at gcc dot gnu.org
Reporter: [email protected]
Target Milestone: ---
Created attachment 65842
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=65842&action=edit
sample profile
I see a severe performance regression in a large Fortran/OpenMP application on
Apple Silicon macOS when several threads perform independent Fortran I/O on
different units/files.
The regression appears between GCC 13 and GCC 14 and is still present in GCC
16. Profiling points to contention in libgfortran `get_gfc_unit()` and the
pthread rwlock introduced in GCC 14.
Unfortunately I cannot provide the application source code. However, the
behavior is very reproducible, and I can provide timing results and complete
macOS `sample` profiles.
**System**
- Apple Silicon MacBook Pro, ARM64
- macOS 26.7.1
- Homebrew GCC
- OpenMP
- 14 hardware cores
- GCC 13.4.0: Homebrew gcc@13
- GCC 14.4.0: Homebrew gcc@14
- GCC 16.2.0
The application processes many independent image files. OpenMP parallelism is
at the file/image level. Each OpenMP worker uses its own Fortran unit numbers
and independently opens, reads, writes, and closes its files. There is no
OpenMP CRITICAL section around the I/O.
The important parallel region effectively has one independent group of files
per OpenMP thread.
**Performance**
For the same test processing 100 images with 14 OpenMP threads:
real user sys
gfortran 13 2.84 s 9.27 s 3.93 s
gfortran 14 11.43 s 10.41 s 9.40 s
gfortran 16 10.50 s 9.52 s 9.45 s
Thus GCC 14 is approximately 4 times slower in wall-clock time than GCC 13.
The most striking change is system CPU time, which rises from 3.93 s with GCC
13 to about 9.4 s with GCC 14/16.
With GCC 16, performance as a function of OpenMP thread count is also unusual.
The application scales well up to approximately 5-6 threads:
4 threads ~2.2 s
5 threads ~2.0 s
6 threads ~2.2 s
Above 6 threads performance degrades steeply, reaching approximately 10.5 s
with 14 threads.
The individual worker timings show that this is not load imbalance. With 14
threads all workers take approximately 8.5-9.5 s, while with 5 threads all five
workers take approximately 1.3-1.35 s despite each processing many more images.
**OpenMP runtime itself does not appear to be the problem**
I tested a CPU-only OpenMP benchmark using gfortran, with no file I/O. It
scales normally:
threads elapsed
1 3.374 s
2 1.722
3 1.219
4 0.899
5 0.730
6 0.623
7 0.533
8 0.475
9 0.432
10 0.367
11 0.391
12 0.369
13 0.347
14 0.356
Therefore libgomp/thread scheduling on this machine appears to work well in the
absence of Fortran I/O.
LLVM Flang also runs the real application efficiently with high thread counts,
so the problem appears specific to libgfortran rather than to the application,
filesystem, or macOS scheduler in general.
**Compiler options**
The problem is reproducible with a minimal compilation:
gfortran -O2 -march=native -fopenmp \
-o xds_test \
file1.f90 file2.f90 file3.f90
For GCC 16 at 14 threads this gives:
real 11.14
user 10.40
sys 9.52
This is essentially identical to the result obtained with the application's
normal, more complicated optimization/link options.
Changing `GFORTRAN_UNFORMATTED_BUFFER_SIZE`, `OMP_WAIT_POLICY`,
`GOMP_SPINCOUNT`, and `OMP_PROC_BIND` does not materially change the result.
**Profiling**
macOS `sample` shows that with GCC 16 most threads spend a very large fraction
of their time in libgfortran unit lookup and pthread rwlock operations.
Typical stacks are:
scanimg
data_transfer_init
get_gfc_unit
_pthread_rwlock_lock_slow
_pthread_rwlock_lock_wait
__psynch_rw_rdlock
and
scanimg
data_transfer_init
get_gfc_unit
_pthread_rwlock_lock_slow
_pthread_rwlock_lock_wait
__psynch_rw_wrlock
The profile's collapsed top-of-stack counts are dominated by:
__psynch_rw_wrlock 23516
__psynch_rw_rdlock 23513
__psynch_rw_unlock 999
whereas the actual kernel `read` and `write` calls appear only rarely.
The profile also shows this path occurring during unformatted reads:
fcb_read
fcb_read_byte
data_transfer_init
get_gfc_unit
pthread_rwlock...
Each OpenMP thread uses different Fortran units, so these are not multiple
threads accessing the same unit.
**Possible regression point**
The regression appears consistent with r14-6425-gb806c88fab3f9c (‘libgfortran:
Replace mutex with rwlock’), because the regression begins with GCC 14 and the
macOS sample profile is dominated by get_gfc_unit() → pthread rwlock calls.
The timing bisect is consistent with that possibility:
GCC 13: fast
GCC 14: slow
GCC 16: slow
On macOS the pthread rwlock implementation ultimately enters the
`__psynch_rw_*` kernel functions that dominate the sample profile.
I realize that the rwlock change was intended to improve parallel I/O
scalability, but on this macOS/Apple Silicon workload it appears to have the
opposite effect once roughly 7 or more threads perform concurrent Fortran I/O.
I would be happy to attach the complete `sample` output and provide additional
profiling or run patched libgfortran versions if that would help. I cannot
redistribute the application source code.
