On Thu, 13 Mar 2025, Simon Sobisch wrote:
>
> Am 13.03.2025 um 12:49 schrieb Richard Biener:
> > On Thu, 13 Mar 2025, Sam James wrote:
> >
> >> Simon Sobisch <simonsobi...@gnu.org> writes:
> >>
> >>> Thanks for your work on adding a testsuite. Can you please explain why
> >>> you do this when a complete testsuite exists in autoconf (autotest)
> >>> format (which roots back to decade of work in GnuCOBOL, with all
> >>> copyrights for that already with the FSF)?
> >>>
> >>
> >> I don't think any of us were aware of it ("we" being "the general GCC
> >> developer community", not the COBOL folks, for the purposes of this
> >> email) until yesterday when richi mused about it on IRC maybe existing
> >> and we went looking out of curiosity.
> >>
> >> I agree that having that testsuite integrated would be fantastic.
> >>
> >>> Is the existence of this in upstream [1] just unknown (because it was
> >>> not part of the initial patches [for reasons I not understood])?
> >>>
> >>
> >> I would've personally liked to see the NIST testsuite integration at
> >> least in the initial patches, but it is what it is. I don't think the
> >> GnuCOBOL testsuite was brought up at all (and I think most of us weren't
> >> aware of it) in the patch upstreaming discussions.
> >>
> >> Now that we *are* aware of it, it seems desirable to have for sure.
> >>
> >>> Is the format such a big issue (note: previous discussions elaborated
> >>> "a test suite is very important and other frontends also use a
> >>> framework other than dejagnu)?
> >>>
> >>> If dejagnu is the way to go:
> >>>
> >>> * Shouldn't there be deprecation of autotest in autoconf (of course
> >>> only if that preference is also outside of gcc)?
> >>
> >> It's a GCC / GNU toolchain-only preference because it allows easily
> >> doing cross + simulator testing, and all of our tools are used to its
> >> format.
> >
> > That's indeed the main reason.
>
> Thanks for the explanation. That's totally fine.
>
> >
> >> It's definitely not perfect. Years ago (way before I followed GCC),
> >> there was talk of replacing dejagnu, just efforts failed.
> >>
> >>>
> >>> * Shouldn't there be a (at least semi automated) script / migration
> >>> tool (at least for this specific time in place to convert the "UAT"
> >>> once into dejagnu format)?
> >>
> >> Yes. Having testsuite integration is seen as critical at this
> >> point. richi just wanted to present this as a non-COBOL person to give
> >> us something to play with.
> >
> > Yes, and to give people familiar with how GCC tests are done a place
> > to put regression tests going forward.
> >
> > I do think that integrating the testsuites the COBOLworx folks have
> > is important and of course integrating tests from GNU Cobol is desirable
> > as well. Whether we can or want to integrate tests based on autotest
> > is another question - I'd probably avoid that, even as short-term
> > solution, as such tend to stay forever.
>
> I agree. Note: COBOLworx started by using the GnuCOBOL testsuite; even with
> the current UAT's state it would be a lot of manual work to re-synchronize
> them, so going one step further to dejagnu seems to not make it much harder
> either.
> It will definitely be useful if the "original test file names" (like
> run_subscripts.at, or at least run_subscripts) are kept somewhere - a comment
> like "auto-translated from run_subscripts.at" is enough - and maybe they can
> stay in one file each (I don't know enough about dejagnu to comment on that).
>
> The main point is that it seems most reasonable to convert those files into
> dejagnu format once (so obviously a "script working good enough, not
> installed" comes into mind), instead of writing it from scratch.
Definitely.
> > What would be nice is to have a common separate test harness you can
> > test an installed compiler against - I'm not sure whether the GNU
> > Cobol test harness or the COBOLworx one qualifies here. The NIST
> > one probably does, but it seems to require "plumbing" that's not
> > part of NIST and that, in implementation, might differ from GNU Cobol
> > to COBOLworx.
>
> That's a good opportunity to be picky: it is GnuCOBOL (one word, COBOL in
> upper-case) :-)
>
> And yes: a common separate test harness is most reasonable and that's exactly
> what the idea of NIST was.
> If you ever wonder: GnuCOBOL uses make (with one sub-directory per "Module")
> along with perl [2].
> This allows to not only do testing (or just extraction of the files) along
> with counting and tracking time, but also to automate some of the required
> "needs manual inspection".
>
> And given gcobc, I'd argue that gcobol should not fail the following (and
> ideally show its superior compile and run time):
>
> $> tar -xvf gnucobol-3.*.tar.*
> $> cd gnucobol-3.*/
> $> ./configure # for automake and autoconf doing the setup
> $> cd tests/cobol85
> $> make test COBC=gcobc-15
>
> ... just tried that:
> gcobol: error: unrecognized command-line option ‘-std=cobol85’
>
> --> seems like the gcobc should drop that and set the right flags for
> gcobol here (I know, should be on bugzilla, or just fixed)
>
>
> $> make test COBC=gcobc-15 COBC_FLAGS=--debug
> Compiling EXEC85 program
> warning: --debug implies -fstack-check: ignored
> EXEC85.cob:1:73: error: syntax error, unexpected NAME, expecting FUNCTION or
> PROGRAM-ID
> 1 | 000100 IDENTIFICATION DIVISION.
> EXEC84.2
> |
> ^
> cobol1: error: failed compiling EXEC85.cob
>
> --> seems the recognition in cobol1 for the reference-format can be
> improved (that's no *problem* in general)
>
>
> $> make test COBC=gcobc-15 COBC_FLAGS="-fec=all -fformat=fixed -g"
> Compiling EXEC85 program
> Building module directory NC ...
> /home/build/gcobol-test/gnucobol-3.2/pre-inst-env: 69: exec: ../EXEC85: not
> found
>
>
> --> that is definitely another problem of the gcobc driver script - if
> no output file is given this must be deduced from the first input file to
> match cobc behavior
Thanks for doing this! It's exactly what we want at this point - people
trying and finding bugs. It would be super useful if you'd record
bugzilla issues for each of the above on https://gcc.gnu.org/bugzilla
Compatibility of gcobc is likely quite important.
>
> $> mv a.out EXEC85
> $> make test COBC=gcobc-15 COBC_FLAGS="-fec=all -fformat=fixed -g"
> Building module directory NC ...
>
>
> ... that runs until crash.
>
> /bin/bash: Line 6: 6503 Killed ( cd NC && COB_UNIX_LF=Y
> "/home/build/gcobol-test/gnucobol-3.2/pre-inst-env" ../EXEC85 )
>
>
> because of some looping (single core completely used, memory fast increasing,
> here's the output from "top" shortly before the system-internal kill
>
> simon 20 0 19,1g 15,1g 960 R 15,0 97,0 1:52.12 EXEC85
>
>
> Side note: even when the system kills the process, it should have some time to
> write a reasonable stacktrace or something to know where the error happened.
Unfortunatelly a KILL from the system is fatal, no way to record anything.
>
> I'm quite sure the looping part is related to file handling (rooting in
> configuration that is different for gcobol and GnuCOBOL [which it may
> shouldn't]) but that's for Bob to inspect/fix (I've used the COBOLworx debian
> package generated yesterday, BTW).
>
> ... ok, re-executed and attached with GDB - it loops around the READ at
> EXEC.cob:2202, presumably for the non-open file.
>
>
> With the guessed configuration issue:
>
> * the program should not hog memory during looping (Bob, please inspect
> this one *first*)
> * the program should abort somewhere, likely at the OPEN long before it
> gets to the READ where it loops, but that should be fixed after this
> specific "loop creates a memory problem" is fixed (end then maybe in
> reverse order, first let the READ [directly after the OPEN] of the
> not-opened file abort, then the OPEN itself) [note: there is neither
> a USE section nor an io-status for the file defined and even the
> -fec-io runtime checks were requested to be enabled, so ISO says that
> this READ (and the OPEN before) should result in a fatal runtime
> abort]
>
> Thanks,
> Simon
>
>
> >
> > Richard.
> >
> >>>
> >>> [1]:
> >>> https://gitlab.cobolworx.com/COBOLworx/gcc-cobol/-/tree/master+cobol/gcc/cobol/UAT
>
> [2]:
> https://sourceforge.net/p/gnucobol/code/HEAD/tree/branches/gnucobol-3.x/tests/cobol85/
>
--
Richard Biener <rguent...@suse.de>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Ivo Totev, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
