https://gcc.gnu.org/bugzilla/show_bug.cgi?id=127973
Bug ID: 127973
Summary: caf_shmem: collectives do not report an exhausted
shared memory pool through STAT= and ERRMSG=
Product: gcc
Version: 17.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: libfortran
Assignee: unassigned at gcc dot gnu.org
Reporter: jvdelisle at gcc dot gnu.org
Target Milestone: ---
With -fcoarray=shared, CO_SUM, CO_MIN, CO_MAX, CO_REDUCE and CO_BROADCAST
need a buffer in the shared memory pool that is larger than the data
(for the reductions, the data times the number of images). When the
pool cannot hold it, the collective neither sets STAT= nor ERRMSG=. One
image error terminates with
Fortran runtime error: Shared memory exhausted; increase
GFORTRAN_SHARED_MEMORY_SIZE
and the other images hang forever.
Reproducer:
program co_sum_pool
implicit none
real(8), allocatable :: a(:)
integer :: st
character(80) :: msg
allocate (a(4*1024*1024)) ! 32 MiB per image
a = this_image ()
st = 0
msg = ''
call co_sum (a, stat=st, errmsg=msg)
print '(a,i0,a,i0,2a)', 'image ', this_image (), ': stat=', st, &
' errmsg=', trim (msg)
end program co_sum_pool
$ gfortran -fcoarray=shared co_sum_pool.f90
$ GFORTRAN_SHARED_MEMORY_SIZE=64M GFORTRAN_NUM_IMAGES=4 ./a.out
Fortran runtime error: Shared memory exhausted; increase
GFORTRAN_SHARED_MEMORY_SIZE
ERROR: Image 4(pid: 2772604) failed with signal 0, exitstatus 1.
(hangs)
CO_BROADCAST of a 128 MiB array with the same settings behaves the same.
Expected: every image returns from CO_SUM with a nonzero STAT and an
ERRMSG saying the shared memory is exhausted, as ALLOCATE of a coarray
does since PR127751.
Cause: get_collsub_buf in libgfortran/caf/shmem/collective_subroutine.c
grows the team's collective buffer with allocator_shared_malloc_or_die
while it holds the buffer's mutex. The image that grows it terminates
with the mutex still locked, so the others block in
pthread_mutex_lock in get_collsub_buf:
#1 pthread_mutex_lock
#2 get_collsub_buf (size=134217728) collective_subroutine.c:201
#3 collsub_reduce_array collective_subroutine.c:346
#4 _gfortran_caf_co_sum shmem.c:956
get_collsub_buf also calls the allocator without allocator_lock, so it
races with ALLOCATE or FORM TEAM running in another team.
A fix needs get_collsub_buf to report the failure instead of terminating,
in a way that every image of the collective sees, and
collsub_reduce_array and collsub_broadcast_array need a result besides
"an image terminated" so that the _gfortran_caf_co_* entry points can set
STAT= and ERRMSG=.
Raised by Mikael Morin in the review of PR127751.
