[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #37 from burnus at gcc dot gnu dot org 2006-12-19 17:15 --- Subject: Bug 29568 Author: burnus Date: Tue Dec 19 17:14:22 2006 New Revision: 120053 URL: http://gcc.gnu.org/viewcvs?root=gccview=revrev=120053 Log: Merged revisions 119412-119459 via svnmerge from svn+ssh://gcc.gnu.org/svn/gcc/trunk r119412 | tkoenig | 2006-12-01 22:04:38 +0100 (Fr, 01 Dez 2006) | 70 lines 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * gfortran.dg/convert_implied_open.f90: Change to new default record length. * gfortran.dg/unf_short_record_1.f90: Adapt to new error message. * gfortran.dg/unformatted_subrecords_1.f90: New test. 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * gfortran.h (gfc_option_t): Add max_subrecord_length. (top level): Define MAX_SUBRECORD_LENGTH. * lang.opt: Add option -fmax-subrecord-length=. * trans-decl.c: Add new function set_max_subrecord_length. (gfc_generate_function_code): If we are within the main program and max_subrecord_length has been set, call set_max_subrecord_length. * options.c (gfc_init_options): Add defaults for max_subrecord_lenght, convert and record_marker. (gfc_handle_option): Add handling for -fmax_subrecord_length. * invoke.texi: Document the new default for -frecord-marker=n. 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * libgfortran/libgfortran.h (compile_options_t): Add record_marker. (top level): Define GFC_MAX_SUBRECORD_LENGTH. * runtime/compile_options.c (set_record_marker): Change default to four-byte record marker. (set_max_subrecord_length): New function. * runtime/error.c (translate_error): Change error message for short record on unformatted read. * io/io.h (gfc_unit): Add recl_subrecord, bytes_left_subrecord and continued. * io/file_pos.c (unformatted_backspace): Change default of record marker size to four bytes. Loop over subrecords. * io/open.c: Default recl is max_offset. If compile_options.max_subrecord_length has been set, set set u-recl_subrecord to its value, to the maximum value otherwise. * io/transfer.c (top level): Add prototypes for us_read, us_write, next_record_r_unf and next_record_w_unf. (read_block_direct): Separate codepaths for unformatted direct and unformatted sequential. If a recl has been set by the user, use the number of bytes left for the record if it is smaller than the read request. Loop over subrecords. Set an error if the user has set a recl and the read was short. (write_buf): Separate codepaths for unformatted direct and unformatted sequential. If a recl has been set by the user, use the number of bytes left for the record if it is smaller than the read request. Loop over subrecords. Set an error if the user has set a recl and the read was short. (us_read): Add parameter continued (to indicate that bytes_left should not be intialized). Change default of record marker size to four bytes. Use subrecord. If the subrecord length is smaller than zero, this indicates a continuation. (us_write): Add parameter continued (to indicate that the continued flag should be set). Use subrecord. (pre_position): Use 0 for continued on us_write and us_read calls. (skip_record): New function. (next_record_r_unf): New function. (next_record_r): Use next_record_r_unf. (write_us_marker): Default size for record markers is four bytes. (next_record_w_unf): New function. (next_record_w): Use next_record_w_unf. r119415 | reichelt | 2006-12-01 22:28:35 +0100 (Fr, 01 Dez 2006) | 5 lines PR c++/30021 * c-common.c (check_main_parameter_types): Check for error_mark_node. * g++.dg/other/main1.C: New test. r119416 | reichelt | 2006-12-01 22:35:25 +0100 (Fr, 01 Dez 2006) | 7 lines PR c++/30022 * typeck.c (type_after_usual_arithmetic_conversions): Fix assertion for vector types. (build_binary_op): Use temporary for inner type of vector types. * g++.dg/ext/vector5.C: New test. r119421 | tsmigiel | 2006-12-01 23:43:18 +0100 (Fr, 01 Dez 2006) | 8 lines * config/spu/predicates.md (spu_mov_operand): Add. * config/spu/spu.c (spu_expand_extv): Remove unused code. (print_operand_address, print_operand): Handle addresses containing AND. (spu_split_load, spu_split_store): Use updated movti pattern. * config/spu/spu.md: (_movmode, _movdi, _movti): Handle loads and stores in mov patterns for correct operation
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #36 from tkoenig at gcc dot gnu dot org 2006-12-13 21:37 --- Fixed for trunk and 4.2. Will not backport to 4.1. Closing. -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #32 from tkoenig at gcc dot gnu dot org 2006-12-01 21:04 --- Subject: Bug 29568 Author: tkoenig Date: Fri Dec 1 21:04:38 2006 New Revision: 119412 URL: http://gcc.gnu.org/viewcvs?root=gccview=revrev=119412 Log: 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * gfortran.dg/convert_implied_open.f90: Change to new default record length. * gfortran.dg/unf_short_record_1.f90: Adapt to new error message. * gfortran.dg/unformatted_subrecords_1.f90: New test. 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * gfortran.h (gfc_option_t): Add max_subrecord_length. (top level): Define MAX_SUBRECORD_LENGTH. * lang.opt: Add option -fmax-subrecord-length=. * trans-decl.c: Add new function set_max_subrecord_length. (gfc_generate_function_code): If we are within the main program and max_subrecord_length has been set, call set_max_subrecord_length. * options.c (gfc_init_options): Add defaults for max_subrecord_lenght, convert and record_marker. (gfc_handle_option): Add handling for -fmax_subrecord_length. * invoke.texi: Document the new default for -frecord-marker=n. 2006-12-01 Thomas Koenig [EMAIL PROTECTED] PR libfortran/29568 * libgfortran/libgfortran.h (compile_options_t): Add record_marker. (top level): Define GFC_MAX_SUBRECORD_LENGTH. * runtime/compile_options.c (set_record_marker): Change default to four-byte record marker. (set_max_subrecord_length): New function. * runtime/error.c (translate_error): Change error message for short record on unformatted read. * io/io.h (gfc_unit): Add recl_subrecord, bytes_left_subrecord and continued. * io/file_pos.c (unformatted_backspace): Change default of record marker size to four bytes. Loop over subrecords. * io/open.c: Default recl is max_offset. If compile_options.max_subrecord_length has been set, set set u-recl_subrecord to its value, to the maximum value otherwise. * io/transfer.c (top level): Add prototypes for us_read, us_write, next_record_r_unf and next_record_w_unf. (read_block_direct): Separate codepaths for unformatted direct and unformatted sequential. If a recl has been set by the user, use the number of bytes left for the record if it is smaller than the read request. Loop over subrecords. Set an error if the user has set a recl and the read was short. (write_buf): Separate codepaths for unformatted direct and unformatted sequential. If a recl has been set by the user, use the number of bytes left for the record if it is smaller than the read request. Loop over subrecords. Set an error if the user has set a recl and the read was short. (us_read): Add parameter continued (to indicate that bytes_left should not be intialized). Change default of record marker size to four bytes. Use subrecord. If the subrecord length is smaller than zero, this indicates a continuation. (us_write): Add parameter continued (to indicate that the continued flag should be set). Use subrecord. (pre_position): Use 0 for continued on us_write and us_read calls. (skip_record): New function. (next_record_r_unf): New function. (next_record_r): Use next_record_r_unf. (write_us_marker): Default size for record markers is four bytes. (next_record_w_unf): New function. (next_record_w): Use next_record_w_unf. Added: trunk/gcc/testsuite/gfortran.dg/unformatted_subrecord_1.f90 Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/gfortran.h trunk/gcc/fortran/invoke.texi trunk/gcc/fortran/lang.opt trunk/gcc/fortran/options.c trunk/gcc/fortran/trans-decl.c trunk/gcc/testsuite/ChangeLog trunk/gcc/testsuite/gfortran.dg/convert_implied_open.f90 trunk/gcc/testsuite/gfortran.dg/unf_short_record_1.f90 trunk/libgfortran/ChangeLog trunk/libgfortran/io/file_pos.c trunk/libgfortran/io/io.h trunk/libgfortran/io/open.c trunk/libgfortran/io/transfer.c trunk/libgfortran/libgfortran.h trunk/libgfortran/runtime/compile_options.c trunk/libgfortran/runtime/error.c -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #33 from tkoenig at gcc dot gnu dot org 2006-12-01 21:18 --- Fixed on trunk. I'll be waiting for some time for any regressions before backporting to 4.2. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #31 from patchapp at dberlin dot org 2006-11-30 20:10 --- Subject: Bug number PR libfortran/29568 A patch for this bug has been added to the patch tracker. The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2006-11/msg02060.html -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #28 from tkoenig at gcc dot gnu dot org 2006-11-29 22:12 --- (In reply to comment #27) Hi Jerry, The program fails on x86-64-freebsd and never completes the first write. It just keeps going, and going, and going This is a target specific issue. My guess is that it has to do with alignment or types. Maybe mixed type arithmetic, size_t vs gfc_offset. Is size_t 32 bits on freebsd and gfc_offset 64? I will attempt to change all new variables to gfc_offset and see if that fixes it. Signedness could indeed be the problem. If that doesn't solve the problem, could you print out the values of - nbytes - to_write_record - to_write_subrecord - dtp-u.p.current_unit-bytes_left - dtp-u.p.current_unit-bytes_left_subrecord at the start of the while loop in write_buf() (but please cast them to long as appropriate :-) This should help to see where things go wrong. Possibly, there is an initialization problem in some other place. Thomas -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #29 from jvdelisle at gcc dot gnu dot org 2006-11-30 03:32 --- to_write_subrecord = 0 have_written = 0 to_write_subrecord = 0 have_written = 0 to_write_subrecord = 0 have_written = 0 to_write_subrecord = 0 have_written = 0 ad infinitum Now I wonder if the patch applied correctly. I will be checking that, but we definitely are broken here. I will keep at it. I did find some gcc warnings on comparing signed and unsigned integers so I cast those. To no effect. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #30 from jvdelisle at gcc dot gnu dot org 2006-11-30 06:25 --- Good News. I did not have a clean apply of the patch. I reverted everything and started over. There was one part of the patch I had to manually apply before and I must have messed it up. Now everything works great on X86-64 FreeBSD tests. There are two warnings in transfer.c for comparison of signed and unsigned types that need to get cleaned up. Right around line 649 IIRC. With that, this patch is ready to submit. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #27 from jvdelisle at gcc dot gnu dot org 2006-11-29 03:57 --- Created an attachment (id=12704) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12704action=view) Huge+ record size unformatted write then read test This test program exercizes patch 10 with very large record sizes. The program executes correctly (except the recl value printed, noted elsewhere) on x86-64-linux-gnu. The program fails on x86-64-freebsd and never completes the first write. It just keeps going, and going, and going This is a target specific issue. My guess is that it has to do with alignment or types. Maybe mixed type arithmetic, size_t vs gfc_offset. Is size_t 32 bits on freebsd and gfc_offset 64? I will attempt to change all new variables to gfc_offset and see if that fixes it. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #25 from jvdelisle at gcc dot gnu dot org 2006-11-27 08:24 --- I have successfully completed testing to the extent I can including large numbers of I/O operations and huge records ( 5 gigabyte) and large file (50 gigabyte). All looks Good. I have performance checked against ifort and gfortran is comparable at least for my simple tests. With my minor comments offlist I think this is ready to submit for approval. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #26 from tkoenig at gcc dot gnu dot org 2006-11-27 20:43 --- (In reply to comment #24) Subject: Re: implement unformatted files with subrecords (Intel style) I have reviewed the patch and I have one minor comment. I suggest that the continued flag be placed in with the unit flags similar to has_recl. Only for consistency. I am in the process of testing still. Hi Jerry, looking at the unit_flags struct, I notice that all components are determined by flags set on the I/O statement. Putting the continued flag doesn't seem right to me. Apart from that, I have put in a better error message and am currently in the process of bootstrapping and regtesting. Hopefully, I'll be able to submit tomorrow. Thanks for your help! Thomas -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #22 from tkoenig at gcc dot gnu dot org 2006-11-26 20:39 --- Created an attachment (id=12696) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12696action=view) Serious attempt Hi folks, here is a serious attempt at the patch. Jerry, if you could give it a spin, I'd be grateful. At least it passes write_check3.f90. I'm currently regression-testing this. Thomas -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Attachment #12691|0 |1 is obsolete|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #23 from tkoenig at gcc dot gnu dot org 2006-11-26 20:43 --- Created an attachment (id=12697) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12697action=view) Test case Here's a test case for reading and writing with a restricted subrecord length. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #24 from jvdelisle at verizon dot net 2006-11-26 22:27 --- Subject: Re: implement unformatted files with subrecords (Intel style) I have reviewed the patch and I have one minor comment. I suggest that the continued flag be placed in with the unit flags similar to has_recl. Only for consistency. I am in the process of testing still. Jerry -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #20 from tkoenig at gcc dot gnu dot org 2006-11-25 22:58 --- Created an attachment (id=12691) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12691action=view) Latest update Here's the latest update. This is fairly complete, but still lacks testing on exceeding the write record length. Also not yet regression-tested. Thomas -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Attachment #12646|0 |1 is obsolete|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #21 from jvdelisle at gcc dot gnu dot org 2006-11-26 04:10 --- Patch 9 regression test are OK on x86-64. Just need the RECL= limit checks and error messages. I will start on some nore complex tests now. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #18 from jvdelisle at gcc dot gnu dot org 2006-11-22 01:36 --- Thats what I get for late nite fun! :) Initializing x gives a clean valgrind check. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #19 from jvdelisle at gcc dot gnu dot org 2006-11-22 01:55 --- I noticed with ifort that with recl=25, a severe run time error is given for exceeding the record length. gfc happily proceeds and this is not related to the patch. Hmm, have we dropped an error check somewhere on recl? -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #9 from tkoenig at gcc dot gnu dot org 2006-11-19 20:42 --- Created an attachment (id=12646) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12646action=view) Latest update Here's the latest update of the patch, for reading, writing and backspace. In order to be able to test this at all, I set a maximums subrecord size of 16 (see the #ifdef). This is getting to the state where the patch could use testing by people who actually write 2GB records (I can't do that on my home system). Volunteers, anybody? :-) Thomas -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Attachment #12609|0 |1 is obsolete|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #8 from tkoenig at gcc dot gnu dot org 2006-11-15 21:05 --- (In reply to comment #7) Unfortunately, there was one thinko in the approach I took with reading. Even for subrecords, we need to be able to spot if we exceed recl. Back to the drawing board... Actually, we currently don't check wether we exceed recl on unformatted write or read, and neither ifort or g77 do so: $ cat recl.f program main character*10 a,b open (9, file=foo.dat, form=unformatted, access=sequential, recl=80) write (9) '1234567890','1234567890' close (9) open (10, file=foo.dat, form=unformatted, access=sequential,recl=10) read (10) a,b print *,a,b end $ gfortran recl.f ./a.out 12345678901234567890 $ ifort recl.f ./a.out 12345678901234567890 $ g77 recl.f ./a.out 12345678901234567890 If there's a bug to fix, it doesn't concern subrecords, at least. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #7 from tkoenig at gcc dot gnu dot org 2006-11-13 19:11 --- Created an attachment (id=12609) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12609action=view) newest version Here's the newest version of the patch, which does reading and backspace, plus defaults to four-byte record markers, with requried corrections in the testsuite. Unfortunately, there was one thinko in the approach I took with reading. Even for subrecords, we need to be able to spot if we exceed recl. Back to the drawing board... -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Attachment #12581|0 |1 is obsolete|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #6 from tkoenig at gcc dot gnu dot org 2006-11-09 20:43 --- Created an attachment (id=12581) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12581action=view) patch for reading and backspace Here's the next installment of the patch, which seems to work OK for backspace, too. Some more testing is still required. Thomas -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Attachment #12550|0 |1 is obsolete|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #5 from tkoenig at gcc dot gnu dot org 2006-11-05 21:48 --- Created an attachment (id=12550) -- (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12550action=view) Patch for reading only This is a partial patch, for reading only. I have taken the approach that we should allow any record size 0 to mean continued records (regardless of whether we have four- or eight-byte record markers). This doesn't break anything, simplifies codepaths and clears the way for records longer than 2**63 bytes :-) -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #4 from jb at gcc dot gnu dot org 2006-11-04 14:28 --- There is also some documentation in the ifort manual: http://www.intel.com/software/products/compilers/flin/docs/main_for/mergedProjects/bldaps_for/format_of_record_types_.htm -- jb at gcc dot gnu dot org changed: What|Removed |Added CC||jb at gcc dot gnu dot org http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #3 from tobias dot burnus at physik dot fu-berlin dot de 2006-10-31 21:37 --- Steve Lionel from Intel wrote http://groups.google.com/group/comp.lang.fortran/tree/browse_frm/thread/062ce3447e5ef570/7e2c6b5723c3b228#doc_a7f0b804f755e27b For a record length greater than 2,147,483,639 bytes, the record is divided into subrecords. The subrecord can be of any length from 1 to 2,147,483,639, inclusive. The sign bit of the leading length field indicates whether the record is continued or not. The sign bit of the trailing length field indicates the presence of a preceding subrecord. The position of the sign bit is determined by the endian format of the file. A subrecord that is continued has a leading length field with a sign bit value of 1. The last subrecord that makes up a record has a leading length field with a sign bit value of 0. A subrecord that has a preceding subrecord has a trailing length field with a sign bit value of 1. The first subrecord that makes up a record has a trailing length field with a sign bit value of 0. -- tobias dot burnus at physik dot fu-berlin dot de changed: What|Removed |Added CC||tobias dot burnus at physik ||dot fu-berlin dot de http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #2 from tkoenig at gcc dot gnu dot org 2006-10-25 20:11 --- (In reply to comment #1) Thomas, Have you written Adrain about his plans concerning his patch? Not yet (I tried CC'ing him with this, but apparently this doesn't work). IIRC (and Adrian, please correct me) his patch concerned the per-unit setting of four-byte markers. Here, I'm trying to implement the Intel subrecord markers. BTW, I think the Intel subrecord approach is probably the best solution to the large record problem. I'm glad we agree on this. -- tkoenig at gcc dot gnu dot org changed: What|Removed |Added Status|UNCONFIRMED |NEW Ever Confirmed|0 |1 Last reconfirmed|-00-00 00:00:00 |2006-10-25 20:11:19 date|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
[Bug libfortran/29568] implement unformatted files with subrecords (Intel style)
--- Comment #1 from kargl at gcc dot gnu dot org 2006-10-23 19:49 --- Thomas, Have you written Adrain about his plans concerning his patch? BTW, I think the Intel subrecord approach is probably the best solution to the large record problem. -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29568
