Hi Jerry, all,
Am 04.11.26 um 1:08 AM schrieb Jerry D:
On 11/3/26 2:39 PM, Harald Anlauf wrote:
Dear all,
this patch seems to be a very special challenge:
The testcase has an UTF-8 byte order mark, which I do not see in the
patch unless I use "od". If you don't see it, then please see the
attachment in the PR for reference.
We used a too short line buffer when reading the testcase, leading
to valgrind invalid read/writes. The solution here is to extend
the line buffer, taking care of line length limits. I do hope I
got it right.
Regtested on x86_64-pc-linux-gnu. OK for mainline?
Thanks,
Harald
Yes, OK Harald.
I did a read of the standard and this is acceptable.
thanks, this is what I hoped for.
Since this thread is growing, I would like to remind all that
my patch does not change the behavior of gfortran other than
to prevent a buffer overrun.
The current behavior of gfortran (scanner.cc:load_file) was implemented
by FX in 2007/2008. His PR mentions
https://en.wikipedia.org/wiki/Byte_order_mark
commit caef7872f0dc4a62dfc449785974eaa179b0a449
Author: Francois-Xavier Coudert <[email protected]>
Date: Sun Apr 29 10:45:57 2007 +0000
re PR fortran/31645 (Error on reading Byte Order Mark)
PR fortran/31645
* scanner.c (load_file): Discard the byte order mark if one is
found on the first non-preprocessor line of a file.
/* If this is the first line of the file, it can contain a byte
order mark (BOM), which we will ignore:
FF FE is UTF-16 little endian,
FE FF is UTF-16 big endian,
EF BB BF is UTF-8. */
if (first_line
&& ((line_len >= 2 && line[0] == (unsigned char) '\xFF'
&& line[1] == (unsigned char) '\xFE')
|| (line_len >= 2 && line[0] == (unsigned char) '\xFE'
&& line[1] == (unsigned char) '\xFF')
|| (line_len >= 3 && line[0] == (unsigned char) '\xEF'
&& line[1] == (unsigned char) '\xBB'
&& line[2] == (unsigned char) '\xBF')))
{
...
This behavior is tested by a couple of testcases, so I am not too happy
if there is the requirement to change the default.
Note the BOM will be removed only when it is at the beginning of the
first line, and only then the column count should be adjusted.
Then I did not take care of UTF-16 or UTF-32 in particular. I just
ran the testcases under valgrind but did not touch anything here.
A slightly modified version of the testcase (see attached) still gives:
% gfc-17 pr128343-v2.f -w
pr128343-v2.f:24:2:
24 | ! <- here's an unsolicited BOM { dg-error "Invalid character
0xBB" }
| 1
Error: Non-numeric character in statement label at (1)
pr128343-v2.f:24:3:
24 | ! <- here's an unsolicited BOM { dg-error "Invalid character
0xBB" }
|
Error: Invalid character 0xBB at (1)
I get the same as with 16 modulo the (previous) risk of memory
corruption.
If there are strong opinions, I can add a warning under a non-default
-W option if a BOM is found so that existing testcases remain happy.
Harald
Jerry
#define a(i) b(i)
!
! PR fortran/128343 - UTF-8 byte order mark and internal line buffer
!
! { dg-additional-options "-w" }
!
! Note: this file has the following characters at the beginning:
!
! 0000000 357 273 277 043 144 145 146 151 156 145 040 141 050 151 051 040
! 357 273 277 # d e f i n e a ( i )
!
! which lead to invalid write/read errors under valgrind.
!
! Derived from 1066_0007.f from the Fujitsu test suite on Github.
#if 0
#else ! a
! a
write(10,*) a
print *,'pass'
#endif ! a a
end
! <- here's an unsolicited BOM { dg-error "Invalid character 0xBB" }
! { dg-prune-output "Non-numeric character in statement label" }
