Re: [patch, fortran] PR25829 Asynchronous I/O (patch version 2.0)
Hi Nicolas, > Here is the next version of the async I/O patch. It adds the documentation, > renames the testcases, uses "gthr.h", follows the style guidelines and has > been regression tested cleanly. > > As for adding additional flags, I think it would be better to follow ifort > to minimize complexity. > > The benchmark (not for the test suite) should also run on systems with > small stack sizes. I didn't look at the code this time, just bootstrapped with the patch included on i386-pc-solaris2.11 and sparc-sun-solaris2.11. There are still problems: +FAIL: gfortran.dg/f2003_io_1.f03 -O0 execution test +FAIL: gfortran.dg/f2003_io_1.f03 -O1 execution test +FAIL: gfortran.dg/f2003_io_1.f03 -O2 execution test +FAIL: gfortran.dg/f2003_io_1.f03 -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions execution test +FAIL: gfortran.dg/f2003_io_1.f03 -O3 -g execution test +FAIL: gfortran.dg/f2003_io_1.f03 -Os execution test on both 32 and 64-bit sparc and x86. The test always fails with STOP 3 FAIL: libgomp.fortran/async_io_1.f90 -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions execution test on 64-bit Solaris/x86 only (STOP 2). Also on 32-bit Solaris/x86, I had an instance of f2003_inquire_1.exe hang for 1 1/2 hours, although it should have been aborted by runtest after the usual 300 s timeout: +FAIL: gfortran.dg/f2003_inquire_1.f03 -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions execution test +WARNING: program timed out. Looks like a deadlock: 4036: ./f2003_inquire_1.exe lwp# 1 / thread# 1 --- fd9b9db9 lwp_park (0, 0, 0) fd9b2f05 cond_wait_queue (80656a0, 8065688, 0, fd9b3433) + 60 fd9b34b4 __cond_wait (80656a0, 8065688, feff4e58, fd9b34f6) + 8f fd9b3504 cond_wait (80656a0, 8065688, feff4eb8, fd9b3534) + 24 fd9b3549 pthread_cond_wait (80656a0, 8065688, 583e6a5, fd9a4655) + 21 fddf3355 _gfortrani_async_wait (feff4fac, 8065640, feff4f2f, feff4fac) + 125 fddd0e38 _gfortran_st_close (feff4fac, feff4f61, 5, 8050f0b, feff50a8, feff5050) + 58 080514db MAIN__ (20202020, 20202020, 20202020, 20202020, 4f525020, 53534543) + 2cb 20202038 () lwp# 2 / thread# 2 --- fd9b9db9 lwp_park (0, 0, 0) fd9b2f05 cond_wait_queue (8065674, 806565c, 0, fd9b3433) + 60 fd9b34b4 __cond_wait (8065674, 806565c, fd7fef38, fd9b34f6) + 8f fd9b3504 cond_wait (8065674, 806565c, fd7fef58, fd9b3534) + 24 fd9b3549 pthread_cond_wait (8065674, 806565c, fd7fefa8, fddf0d96) + 21 fddf131d async_io (80631d0, fda08000, fd7fefc8, fd9b9a27) + 5cd (async.c:102) fd9b9a79 _thrp_setup (fdab0240) + 99 fd9b9d60 _lwp_start (0, 0, 0, 0, 0, 0) Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
[patch, fortran] PR25829 Asynchronous I/O (patch version 2.0)
Hey everyone,
Here is the next version of the async I/O patch. It adds the
documentation, renames the testcases, uses "gthr.h", follows the style
guidelines and has been regression tested cleanly.
As for adding additional flags, I think it would be better to follow
ifort to minimize complexity.
The benchmark (not for the test suite) should also run on systems with
small stack sizes.
I hope I forgot nothing.
Nicolas
2018-06-16 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* gfortran.texi: Add description of asynchronous I/O.
* trans-decl.c (gfc_finish_var_decl): Treat asynchronous variables
as volatile.
* trans-io.c (gfc_build_io_library_fndecls): Rename st_wait to
st_wait_async and change argument spec from ".X" to ".w".
(gfc_trans_wait): Pass ID argument via reference.
2018-06-16 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* gfortran.dg/f2003_inquire_1.f03: Add write statement.
* gfortran.dg/f2003_io_1.f03: Add wait statement.
2018-06-16 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* Makefile.am: Add async.c to gfor_io_src.
Add async.h to gfor_io_headers.
* Makefile.in: Regenerated.
* gfortran.map: Add _gfortran_st_wait_async.
* io/async.c: New file.
* io/async.h: New file.
* io/close.c: Include async.h.
(st_close): Call async_wait for an asynchronous unit.
* io/file_pos.c (st_backspace): Likewise.
(st_endfile): Likewise.
(st_rewind): Likewise.
(st_flush): Likewise.
* io/inquire.c: Add handling for asynchronous PENDING
and ID arguments.
* io/io.h (st_parameter_dt): Add async bit.
(st_parameter_wait): Correct.
(gfc_unit): Add au pointer.
(st_wait_async): Add prototype.
(transfer_array_inner): Likewise.
(st_write_done_worker): Likewise.
* io/open.c: Include async.h.
(new_unit): Initialize asynchronous unit.
* io/transfer.c (async_opt): New struct.
(wrap_scalar_transfer): New function.
(transfer_integer): Call wrap_scalar_transfer to do the work.
(transfer_real): Likewise.
(transfer_real_write): Likewise.
(transfer_character): Likewise.
(transfer_character_wide): Likewise.
(transfer_complex): Likewise.
(transfer_array_inner): New function.
(transfer_array): Call transfer_array_inner.
(transfer_derived): Call wrap_scalar_transfer.
(data_transfer_init): Check for asynchronous I/O.
Perform a wait operation on any pending asynchronous I/O
if the data transfer is synchronous. Copy PDT and enqueue
thread for data transfer.
(st_read_done_worker): New function.
(st_read_done): Enqueue transfer or call st_read_done_worker.
(st_write_done_worker): New function.
(st_write_done): Enqueue transfer or call st_read_done_worker.
(st_wait): Document as no-op for compatibility reasons.
(st_wait_async): New function.
* io/unit.c (insert_unit): Use macros LOCK, UNLOCK and TRYLOCK;
add NOTE where necessary.
(get_gfc_unit): Likewise.
(init_units): Likewise.
(close_unit_1): Likewise. Call async_close if asynchronous.
(close_unit): Use macros LOCK and UNLOCK.
(finish_last_advance_record): Likewise.
(newunit_alloc): Likewise.
* io/unix.c (find_file): Likewise.
(flush_all_units_1): Likewise.
(flush_all_units): Likewise.
* libgfortran.h (generate_error_common): Add prototype.
* runtime/error.c: Include io.h and async.h.
(generate_error_common): New function.
2018-06-16 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* testsuite/libgfomp.fortran/async_io_1.f90: New test.
* testsuite/libgfomp.fortran/async_io_2.f90: New test.
* testsuite/libgfomp.fortran/async_io_3.f90: New test.
program main
implicit none
integer, parameter :: n = 10**7
character(3), parameter :: yes = "yes"
real, dimension(:), allocatable :: a,b,c
allocate (a(n), b(n), c(n))
call random_number(a)
call random_number(b)
call random_number(c)
open (10, file="a.dat",asynchronous=yes)
open (20, file="b.dat",asynchronous=yes)
open (30, file="c.dat",asynchronous=yes)
write (10,*,asynchronous=yes) a
write (20,*,asynchronous=yes) b
write (30,*,asynchronous=yes) c
wait (10)
wait (20)
wait (30)
end program main
! { dg-do run }
! Check basic functionality of async I/O
program main
implicit none
integer:: i=1, j=2, k, l
real :: a, b, c, d
character(3), parameter:: yes="yes"
character(4) :: str
complex :: cc, dd
integer, dimension(4):: is = [0, 1, 2, 3]
integer, dimension(4):: res
character(10) :: inq
open (10, file='a.dat', asynchronous=yes)
cc = (1.5, 0.5)
inquire
Re: [Patch, Fortran] PR25829: Asynchronous I/O
On 2018-06-11 11:24 AM, Rainer Orth wrote: Hi John, On 2018-06-03 2:59 PM, Nicolas Koenig wrote: Since the implementation relies on pthreads, it would be great if somebody could try the patch on non-linux targets, to see whether it causes any problems there. I tried it on hppa64-hp-hpux11.11. The gomp support mostly works on this target. I didn't see any regressions but asynchronous_9 fails: FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test on Solaris, this is due to the test's stack usage, which exeeds the 8 MB default. Maybe the same problem on HP-UX? Yes. Program received signal SIGSEGV, Segmentation fault. 0x40002c7c in MAIN__ () (gdb) disass Dump of assembler code for function MAIN__: 0x40002c70 <+0>: std rp,-10(sp) 0x40002c74 <+4>: addil L%7271000,sp,r1 0x40002c78 <+8>: ldo 160(r1),sp => 0x40002c7c <+12>: std r18,-f8(sp) Store fails after creating 12 MB frame. Dave -- John David Anglin dave.ang...@bell.net
Re: [Patch, Fortran] PR25829: Asynchronous I/O
Hi John, > On 2018-06-03 2:59 PM, Nicolas Koenig wrote: >> Since the implementation relies on pthreads, it would be great if >> somebody could try the patch on non-linux targets, to see whether it >> causes any problems there. > I tried it on hppa64-hp-hpux11.11. The gomp support mostly works on this > target. > > I didn't see any regressions but asynchronous_9 fails: > FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test on Solaris, this is due to the test's stack usage, which exeeds the 8 MB default. Maybe the same problem on HP-UX? Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O
On 2018-06-03 2:59 PM, Nicolas Koenig wrote: Since the implementation relies on pthreads, it would be great if somebody could try the patch on non-linux targets, to see whether it causes any problems there. I tried it on hppa64-hp-hpux11.11. The gomp support mostly works on this target. I didn't see any regressions but asynchronous_9 fails: FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test Dave -- John David Anglin dave.ang...@bell.net
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
>>> FAIL: gfortran.dg/f2003_inquire_1.f03 -O1 execution test
>
> This seems to be a bug in the test suite. It tries to find out whether an id
> is pending that is never initialized.
>
>>> FAIL: gfortran.dg/f2003_io_1.f03 -O*
>
> And another bug in the test suite. This time the wait after the read is
> missing.
Do you have any fix for them ?
> asynchronous_7.f90 is a test for an error, but dg-shouldfail is not working
> in libgomp. Dominique is looking into this.
I have this in my working tree
! { dg-do run }
program main
integer :: i
open (10,file="tst.dat")
write (10,'(A4)') 'asdf'
close(10)
i = 234
open(10,file="tst.dat", asynchronous="yes")
read (10,'(I4)',asynchronous="yes") i
wait(10)
end program main
! { dg-output "Fortran runtime error: Bad value during integer read" }
! { dg-final { remote_file build delete "tst.dat" } }
>
>> Besides, I see
>> +FAIL: libgomp.fortran/asynchronous_6.f90 -O1 execution test
>> STOP 2
>> 32-bit i386 only.
>
> I have trouble replicating this bug even with -m32. Could you get some more
> debugging info for the test on your machine?
I have copied the asynchronous tests from libgomp.fortran to gfortran.dg and
ran both
make -k check-gfortran RUNTESTFLAGS="dg.exp=asynchronous_*
--target_board=unix'{-m32,-m64}’"
in gcc and
make -k check RUNTESTFLAGS="fortran.exp=asynchronous_*.f90
--target_board=unix'{-m32,-m64}’"
in x86_64-apple-darwin17.6.0/libgomp/testsuite 10 times and I see ~10 failures
in each cases (mostly STOP 2, but also STOP 4). I some cases I had to kill the
process.
Note that these tests should probably protected by something such as
! { dg-require-effective-target pthread }
or
! { dg-require-effective-target tls }
Dominique
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Nicolas,
a few other nits:
* The current patch has a large number of GNU Coding Style violations,
many catched by contrib/check_GNU_style.{sh,py}.
* Others are partially pre-existing (additional blank before formal
paramater name as in
+destroy_adv_cond (struct adv_cond * ac)
and many many more.
* Many of the new functions lack comments.
* Hardcoded escape codes in async.h.
* Several indentation and line length errors, like
+ if (au->tail)
+ internal_error(NULL, "Trying to free nonempty unit");
TAB instead of 4 spaces.
* Stuff like this in async.c at the very least needs a comment
explaining what's going on:
+#ifndef GTHREAD_USE_WEAK
+#ifdef SUPPORTS_WEAK
+#define GTHREAD_USE_WEAK 1
+#endif
+#endif
This should already be handled in gthr.h. Why not use that?
+#define _GTHREAD_USE_COND_INIT_FUNC
+#include "../../libgcc/gthr-posix.h"
Again: gthr.h would include that (via gthr-default.h) already.
As I stated, those are mostly nits, but should be fixed before this
patch goes in.
Thanks.
Rainer
--
-
Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Jakub, >> I do note that if one has, say, 8 files and only one >> file uses async IO, then during linking of the 8 *.o >> files to make the final execute -lpthread must be added. >> How doesn't gfortran communicate that to the loader? > > ELF doesn't a way to do this, you'd need to add a special linker plugin for > that. But is that really necessary? > Simply let the users link with -lpthread or -pthread if they want real > asynchronous IO, or without it if they are ok with the gcc <= 8 > behavior and document it. wouldn't it be better to use a special gfortran flag for that? -pthread/-lpthread is an implementation detail on some platforms, but others may not need it (Solaris has folded libpthread into libc and there are no more single-threaded processes, for example) or need something different for async io (Windows, ...). IMO it would be better to abstract this low-level detail away and let the driver DTRT, under the control of a flag if need be. Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
On Tue, Jun 05, 2018 at 11:47:21PM -0700, Steve Kargl wrote: > I have not looked at the source code, but do have a question. > With -fopenmp, gfortran automatically adds -lpthread to > the command line. Is it possible during the parsing Even if it wouldn't, libgomp.so.1 depends on libpthread.so.0 and thus it will be around. > phase to detect an async structure and on the fly add > -lpthread to the loader options? > > I do note that if one has, say, 8 files and only one > file uses async IO, then during linking of the 8 *.o > files to make the final execute -lpthread must be added. > How doesn't gfortran communicate that to the loader? ELF doesn't a way to do this, you'd need to add a special linker plugin for that. But is that really necessary? Simply let the users link with -lpthread or -pthread if they want real asynchronous IO, or without it if they are ok with the gcc <= 8 behavior and document it. Jakub
Re: [Patch, Fortran] PR25829: Asynchronous I/O
On 06/04/2018 11:21 PM, JonY wrote: > On 06/03/2018 06:59 PM, Nicolas Koenig wrote: >> Hello everyone, >> >> this patch adds asynchronous I/O support. Thomas and I finally finished >> a feature-complete and debugged version, so here it is. In order to use >> asynchronous I/O, it is still necessary to link against libpthread, >> libgomp or another library linked against any of the aforementioned two. >> While it might not be the nicest way, it at least keeps in line with the >> likes of ifort. Two of the test I send deal with asynchronous error >> handling, so they will fail if not linked accordingly. >> >> Since the implementation relies on pthreads, it would be great if >> somebody could try the patch on non-linux targets, to see whether it >> causes any problems there. >> >> Let the rain of regressions begin ;) >> >> Nicolas >> >> P.S.: I would very much recommend removing the #undef DEBUG in async.h. >> I have to admit, I am quite proud of the debug printouts. They even >> build a data structure in the background telling you were a locked mutex >> was locked. > > I'm not too familiar with Fortran, but I'll test it out over the > weekends with the asynchronous_9.f90 program. > > Looks like v1 is working with x86_64-w64-mingw32 and winpthreads, I don't have the mail for v2, strangely. signature.asc Description: OpenPGP digital signature
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
On Wed, Jun 06, 2018 at 08:34:57AM +0300, Janne Blomqvist wrote: > On Wed, Jun 6, 2018 at 3:43 AM, Jerry DeLisle wrote: > > > On 06/05/2018 06:58 AM, Rainer Orth wrote: > > > >> Hi Nicolas, > >> > >> Because they were originally intended for the gfortran test suite, but I > >>> couldn't run it there because of libpthread. I will change the numbering > >>> scheme. > >>> > >> > >> the way that libpthread dependency is currently handled seems weird to > >> me: right now it is only dragged in via -fopenmp, although libgomp isn't > >> otherwise used AFAICS. Is this really supposed to work this way? And > >> what about targets that don't have pthreads? Isn't supposed to > >> abstract away from the details of the underlying threading library? > >> > > > > From my perspective, since async is a feature of the language it should > > not require any special flags, just link to pthread always. > > > > If a user does not use it, it will most likely be optimized out. > > > > This has the disadvantage that then the library will always use pthread > stuff also for single-threaded programs, where we now use the stubbed-out > dummy implementations which are faster. > I have not looked at the source code, but do have a question. With -fopenmp, gfortran automatically adds -lpthread to the command line. Is it possible during the parsing phase to detect an async structure and on the fly add -lpthread to the loader options? I do note that if one has, say, 8 files and only one file uses async IO, then during linking of the 8 *.o files to make the final execute -lpthread must be added. How doesn't gfortran communicate that to the loader? PS: Nice work Nicolas. We'll figure out the details. -- Steve
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
On Wed, Jun 6, 2018 at 3:43 AM, Jerry DeLisle wrote: > On 06/05/2018 06:58 AM, Rainer Orth wrote: > >> Hi Nicolas, >> >> Because they were originally intended for the gfortran test suite, but I >>> couldn't run it there because of libpthread. I will change the numbering >>> scheme. >>> >> >> the way that libpthread dependency is currently handled seems weird to >> me: right now it is only dragged in via -fopenmp, although libgomp isn't >> otherwise used AFAICS. Is this really supposed to work this way? And >> what about targets that don't have pthreads? Isn't supposed to >> abstract away from the details of the underlying threading library? >> > > From my perspective, since async is a feature of the language it should > not require any special flags, just link to pthread always. > > If a user does not use it, it will most likely be optimized out. > > Jerry > This has the disadvantage that then the library will always use pthread stuff also for single-threaded programs, where we now use the stubbed-out dummy implementations which are faster. -- Janne Blomqvist
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
On 06/05/2018 06:58 AM, Rainer Orth wrote: Hi Nicolas, Because they were originally intended for the gfortran test suite, but I couldn't run it there because of libpthread. I will change the numbering scheme. the way that libpthread dependency is currently handled seems weird to me: right now it is only dragged in via -fopenmp, although libgomp isn't otherwise used AFAICS. Is this really supposed to work this way? And what about targets that don't have pthreads? Isn't supposed to abstract away from the details of the underlying threading library? From my perspective, since async is a feature of the language it should not require any special flags, just link to pthread always. If a user does not use it, it will most likely be optimized out. Jerry
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Nicolas, > Because they were originally intended for the gfortran test suite, but I > couldn't run it there because of libpthread. I will change the numbering > scheme. the way that libpthread dependency is currently handled seems weird to me: right now it is only dragged in via -fopenmp, although libgomp isn't otherwise used AFAICS. Is this really supposed to work this way? And what about targets that don't have pthreads? Isn't supposed to abstract away from the details of the underlying threading library? If async io needs libpthread (or whatever else), the gfortran driver needs to drag that in, maybe under the control of some option, instead of relying on libgomp that happens to use it already. Maybe have a look at what libobjc and libstdc++ (the other consumers of gthr.h) do here. >> ... and asynchronous_9.f90 is missing from the ChangeLog, which >> ..._7.f90 is missing from the sequence. > > asynchronous_7.f90 is a test for an error, but dg-shouldfail is not working > in libgomp. Dominique is looking into this. Weird: it's already being used in many places in libgomp.oacc*. >> Besides, I see >> >> +FAIL: libgomp.fortran/asynchronous_6.f90 -O1 execution test >> >> STOP 2 >> >> 32-bit i386 only. >> > > I have trouble replicating this bug even with -m32. Could you get some more > debugging info for the test on your machine? I'll try. It seems pretty random, though: I've just run another test where only the 32-bit i386 FAIL: libgomp.fortran/asynchronous_6.f90 -O3 -fomit-frame-pointer -funroll-loops -fpeel-loops -ftracer -finline-functions execution test FAILs. Maybe different runs of the test with different options (under make -j testing) tread on each other's toes? >> +FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test >> >> 32 and 64-bit i386 and sparc, no error message. >> > > This file wasn't supposed to be a test case, that's why it is not in the > ChangeLog. It is a benchmark program, so it takes some time. Maybe a time > out? Could you maybe try running it outside the test suite? Sure: when rebuilt with -g3, I get Thread 2 received signal SIGSEGV, Segmentation fault. [Switching to Thread 1 (LWP 1)] MAIN__ () at /vol/gcc/src/hg/trunk/solaris/libgomp/testsuite/libgomp.fortran/asynchronous_9.f90:7 7 call random_number(a) (gdb) where #0 MAIN__ () at /vol/gcc/src/hg/trunk/solaris/libgomp/testsuite/libgomp.fortran/asynchronous_9.f90:7 (gdb) display/i $pc 1: x/i $pc => 0x8051679 : movl $0x0,-0x7270f60(%ebp) (gdb) p/x $ebp $1 = 0xfeffda78 That address (FEFFDA78-7270F60 = F7D8CB18) isn't even mapped: 9698: /var/gcc/gcc-9.0.0-20180604/11.4-gcc/i386-pc-solaris2.11/libgomp/tests Address Kbytes RSS Anon Lock Mode Mapped File 0805 40 40-- r-x [ text ] asynchronous_9.exe 08069000 888- rwx [ data ] asynchronous_9.exe 0806B000 888- rwx [ heap ] FDB9352 288-- r-x [ text ] libquadmath.so.0.0.0 FDBF7000 444- rwx [ data ] libquadmath.so.0.0. Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O
On 06/03/2018 06:59 PM, Nicolas Koenig wrote: > Hello everyone, > > this patch adds asynchronous I/O support. Thomas and I finally finished > a feature-complete and debugged version, so here it is. In order to use > asynchronous I/O, it is still necessary to link against libpthread, > libgomp or another library linked against any of the aforementioned two. > While it might not be the nicest way, it at least keeps in line with the > likes of ifort. Two of the test I send deal with asynchronous error > handling, so they will fail if not linked accordingly. > > Since the implementation relies on pthreads, it would be great if > somebody could try the patch on non-linux targets, to see whether it > causes any problems there. > > Let the rain of regressions begin ;) > > Nicolas > > P.S.: I would very much recommend removing the #undef DEBUG in async.h. > I have to admit, I am quite proud of the debug printouts. They even > build a data structure in the background telling you were a locked mutex > was locked. I'm not too familiar with Fortran, but I'll test it out over the weekends with the asynchronous_9.f90 program. signature.asc Description: OpenPGP digital signature
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Dominique and Rainer, First of all thanks for testing! Hi Dominique, Nicolas, I have applied your patch on top of revision r261130 on x86_64-apple-darwin17 (SSD with APFS file system). I've tried it on i386-pc-solaris2.11 and sparc-sun-solaris2.11. I also see two regressions FAIL: gfortran.dg/f2003_inquire_1.f03 -O1 execution test only with -m32 and -O1 (STOP 5), and It fails for me at -O[0s] (i386) resp. -O[01] (sparc), 64-bit only. This seems to be a bug in the test suite. It tries to find out whether an id is pending that is never initialized. FAIL: gfortran.dg/f2003_io_1.f03 -O* with both -m32 and -m64 (STOP 1). Same here: FAILs at -O[0-3s] for both 32 and 64-bit. And another bug in the test suite. This time the wait after the read is missing. The is also typos for the added tests s/libgfomp/libgomp/ Will fix. Why do the tests start at asynchronous_6.f90? Because they were originally intended for the gfortran test suite, but I couldn't run it there because of libpthread. I will change the numbering scheme. ... and asynchronous_9.f90 is missing from the ChangeLog, which ..._7.f90 is missing from the sequence. asynchronous_7.f90 is a test for an error, but dg-shouldfail is not working in libgomp. Dominique is looking into this. Besides, I see +FAIL: libgomp.fortran/asynchronous_6.f90 -O1 execution test STOP 2 32-bit i386 only. I have trouble replicating this bug even with -m32. Could you get some more debugging info for the test on your machine? +FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test 32 and 64-bit i386 and sparc, no error message. This file wasn't supposed to be a test case, that's why it is not in the ChangeLog. It is a benchmark program, so it takes some time. Maybe a time out? Could you maybe try running it outside the test suite? Rainer Dominique wrote: > "Treat asynchronous variables the same as volatile, for now." could probably simplified as > "Treat asynchronous variables as volatile, for now." Will do. > > I also wonder if > > +wrap_scalar_transfer (dtp, BT_INTEGER, p, kind, kind, 1); > > is correct without a cast to size_t for the last two arguments (and for the last argument in other instances). Note that I am C challenged, so forgive the question if it is stupid. It atomatically casts based on the type information in the prototype in io.h. > > Thanks for the nice work. With pleasure! :) > > Dominique
Re: [Patch, Fortran] PR25829: Asynchronous I/O
Hi Nicolas, > P.S.: I would very much recommend removing the #undef DEBUG in async.h. I > have to admit, I am quite proud of the debug printouts. They even build a > data structure in the background telling you were a locked mutex was > locked. however, doing so breaks quite a number of tests in gfortran.dg where the additional output is unexpected. Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Dominique, Nicolas, > I have applied your patch on top of revision r261130 on > x86_64-apple-darwin17 (SSD with APFS file system). I've tried it on i386-pc-solaris2.11 and sparc-sun-solaris2.11. > I also see two regressions > > FAIL: gfortran.dg/f2003_inquire_1.f03 -O1 execution test > > only with -m32 and -O1 (STOP 5), and It fails for me at -O[0s] (i386) resp. -O[01] (sparc), 64-bit only. > FAIL: gfortran.dg/f2003_io_1.f03 -O* > > with both -m32 and -m64 (STOP 1). Same here: FAILs at -O[0-3s] for both 32 and 64-bit. > The is also typos for the added tests > > s/libgfomp/libgomp/ > > Why do the tests start at asynchronous_6.f90? ... and asynchronous_9.f90 is missing from the ChangeLog, which ..._7.f90 is missing from the sequence. Besides, I see +FAIL: libgomp.fortran/asynchronous_6.f90 -O1 execution test STOP 2 32-bit i386 only. +FAIL: libgomp.fortran/asynchronous_9.f90 -O execution test 32 and 64-bit i386 and sparc, no error message. Rainer -- - Rainer Orth, Center for Biotechnology, Bielefeld University
Re: [Patch, Fortran] PR25829: Asynchronous I/O (v2)
Hi Nicolas,
I have applied your patch on top of revision r261130 on x86_64-apple-darwin17
(SSD with APFS file system).
The only remaining failure on my own tests is for the test (pr35840)
write(10,*, asynchronous="Y"//"E"//trim("S "))
end
giving at run time
At line 1 of file pr35840.f90 (unit = 10, file = 'fort.10')
Fortran runtime error: ASYNCHRONOUS transfer without ASYHCRONOUS='YES' in OPEN
I also see two regressions
FAIL: gfortran.dg/f2003_inquire_1.f03 -O1 execution test
only with -m32 and -O1 (STOP 5), and
FAIL: gfortran.dg/f2003_io_1.f03 -O*
with both -m32 and -m64 (STOP 1).
The is also typos for the added tests
s/libgfomp/libgomp/
Why do the tests start at asynchronous_6.f90?
"Treat asynchronous variables the same as volatile, for now." could probably
simplified as
"Treat asynchronous variables as volatile, for now."
I also wonder if
+wrap_scalar_transfer (dtp, BT_INTEGER, p, kind, kind, 1);
is correct without a cast to size_t for the last two arguments (and for the
last argument in other instances). Note that I am C challenged, so forgive the
question if it is stupid.
Thanks for the nice work.
Dominique
[Patch, Fortran] PR25829: Asynchronous I/O
Hello everyone,
this patch adds asynchronous I/O support. Thomas and I finally finished
a feature-complete and debugged version, so here it is. In order to use
asynchronous I/O, it is still necessary to link against libpthread,
libgomp or another library linked against any of the aforementioned two.
While it might not be the nicest way, it at least keeps in line with the
likes of ifort. Two of the test I send deal with asynchronous error
handling, so they will fail if not linked accordingly.
Since the implementation relies on pthreads, it would be great if
somebody could try the patch on non-linux targets, to see whether it
causes any problems there.
Let the rain of regressions begin ;)
Nicolas
P.S.: I would very much recommend removing the #undef DEBUG in async.h.
I have to admit, I am quite proud of the debug printouts. They even
build a data structure in the background telling you were a locked mutex
was locked.
Regression tested cleanly on x86_64-pc-linux-gnu.
2018-06-03 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* trans-decl.c (gfc_finish_var_decl): Treat asynchronous variables
as volatile.
* trans-io.c (gfc_build_io_library_fndecls): Rename st_wait to
st_wait_async and change argument spec from ".X" to ".w".
(gfc_trans_wait): Pass ID argument via reference.
2018-06-03 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* Makefile.am: Add async.c to gfor_io_src.
Add async.h to gfor_io_headers.
* Makefile.in: Regenerated.
* gfortran.map: Add _gfortran_st_wait_async.
* io/async.c: New file.
* io/async.h: New file.
* io/close.c: Include async.h.
(st_close): Call async_wait for an asynchronous unit.
* io/file_pos.c (st_backspace): Likewise.
(st_endfile): Likewise.
(st_rewind): Likewise.
(st_flush): Likewise.
* io/inquire.c: Add handling for asynchronous PENDING
and ID arguments.
* io/io.h (st_parameter_dt): Add async bit.
(st_parameter_wait): Correct.
(gfc_unit): Add au pointer.
(st_wait_async): Add prototype.
(transfer_array_inner): Likewise.
(st_write_done_worker): Likewise.
* io/open.c: Include async.h.
(new_unit): Initialize asynchronous unit.
* io/transfer.c (async_opt): New struct.
(wrap_scalar_transfer): New function.
(transfer_integer): Call wrap_scalar_transfer to do the work.
(transfer_real): Likewise.
(transfer_real_write): Likewise.
(transfer_character): Likewise.
(transfer_character_wide): Likewise.
(transfer_complex): Likewise.
(transfer_array_inner): New function.
(transfer_array): Call transfer_array_inner.
(transfer_derived): Call wrap_scalar_transfer.
(data_transfer_init): Check for asynchronous I/O.
Perform a wait operation on any pending asynchronous I/O
if the data transfer is synchronous. Copy PDT and enqueue
thread for data transfer.
(st_read_done_worker): New function.
(st_read_done): Enqueue transfer or call st_read_done_worker.
(st_write_done_worker): New function.
(st_write_done): Enqueue transfer or call st_read_done_worker.
(st_wait): Document as no-op for compatibility reasons.
(st_wait_async): New function.
* io/unit.c (insert_unit): Use macros LOCK, UNLOCK and TRYLOCK;
add NOTE where necessary.
(get_gfc_unit): Likewise.
(init_units): Likewise.
(close_unit_1): Likewise. Call async_close if asynchronous.
(close_unit): Use macros LOCK and UNLOCK.
(finish_last_advance_record): Likewise.
(newunit_alloc): Likewise.
* io/unix.c (find_file): Likewise.
(flush_all_units_1): Likewise.
(flush_all_units): Likewise.
* libgfortran.h (generate_error_common): Add prototype.
* runtime/error.c: Include io.h and async.h.
(generate_error_common): New function.
2018-06-03 Nicolas Koenig
Thomas Koenig
PR fortran/25829
* testsuite/libgfomp.fortran/asynchronous_6.f90: New test.
* testsuite/libgfomp.fortran/asynchronous_8.f90: New test.
Index: gcc/fortran/trans-decl.c
===
--- gcc/fortran/trans-decl.c (Revision 259739)
+++ gcc/fortran/trans-decl.c (Arbeitskopie)
@@ -699,7 +699,8 @@ gfc_finish_var_decl (tree decl, gfc_symbol * sym)
&& CLASS_DATA (sym)->ts.u.derived->attr.has_dtio_procs)))
TREE_STATIC (decl) = 1;
- if (sym->attr.volatile_)
+ /* Treat asynchronous variables the same as volatile, for now. */
+ if (sym->attr.volatile_ || sym->attr.asynchronous)
{
TREE_THIS_VOLATILE (decl) = 1;
TREE_SIDE_EFFECTS (decl) = 1;
Index: gcc/fortran/trans-io.c
