Re: How does GHC's testsuite work?
Hi Sebastien, I’m looking forward to your report, surely there will be some interesting inspirations for us. Am Montag, den 30.10.2017, 11:25 -0400 schrieb Edward Z. Yang: > Actually, it's the reverse of what you said: like OCaml, GHC essentially > has ~no unit tests; it's entirely Haskell programs which we compile > (and sometimes run; a lot of tests are for the typechecker only so > we don't bother running those.) The .T file is just a way of letting > the Python driver know what tests exist. let me add that these tests rarely check the actual output of the compiler (i.e. the program, or even the simplified code). Often it is enough to check * whether the compile succeeds or fails as expected, or maybe * what messages the compiler prints. In a few cases we do dump the complete intermediate code (-ddump- simpl), but then the test case specifies a “normalization function” that checks the output for a certain property, e.g. by grepping for certain patterns. The only real unit tests that I know of are these: http://git.haskell.org/ghc.git/tree/HEAD:/testsuite/tests/callarity/unittest These are effectively programs using “GHC-the-library” Joachim -- Joachim Breitner m...@joachim-breitner.de http://www.joachim-breitner.de/ signature.asc Description: This is a digitally signed message part ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Re: How does GHC's testsuite work?
Excerpts from Sébastien Hinderer's message of 2017-10-30 16:39:24 +0100: > Dear Edward, > > Many thanks for your prompt response! > > Edward Z. Yang (2017/10/30 11:25 -0400): > > Actually, it's the reverse of what you said: like OCaml, GHC essentially > > has ~no unit tests; it's entirely Haskell programs which we compile > > (and sometimes run; a lot of tests are for the typechecker only so > > we don't bother running those.) The .T file is just a way of letting > > the Python driver know what tests exist. > > Oh okay! Would you be able to point me to just a few tests to get an > idea of a few typical situations, please? For example: The metadata https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/all.T The source file https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.hs The expected error output https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.stderr > One other question I forgot to ask: how do you deal with conditional > tests? For instance, if a test should be run only on some platforms? Or, > in OCaml we have tests for Fortran bindings that should be run only if a > Fortran compiler is available. How would you deal with such tests? All managed inside the Python driver code. Example: https://github.com/ghc/ghc/blob/master/testsuite/tests/rts/all.T#L32 Edward ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Re: How does GHC's testsuite work?
Dear Edward, Many thanks for your prompt response! Edward Z. Yang (2017/10/30 11:25 -0400): > Actually, it's the reverse of what you said: like OCaml, GHC essentially > has ~no unit tests; it's entirely Haskell programs which we compile > (and sometimes run; a lot of tests are for the typechecker only so > we don't bother running those.) The .T file is just a way of letting > the Python driver know what tests exist. Oh okay! Would you be able to point me to just a few tests to get an idea of a few typical situations, please? One other question I forgot to ask: how do you deal with conditional tests? For instance, if a test should be run only on some platforms? Or, in OCaml we have tests for Fortran bindings that should be run only if a Fortran compiler is available. How would you deal with such tests? Thanks! Sébastien. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Re: How does GHC's testsuite work?
Actually, it's the reverse of what you said: like OCaml, GHC essentially has ~no unit tests; it's entirely Haskell programs which we compile (and sometimes run; a lot of tests are for the typechecker only so we don't bother running those.) The .T file is just a way of letting the Python driver know what tests exist. Edward Excerpts from Sébastien Hinderer's message of 2017-10-30 16:17:38 +0100: > Dear all, > > I am a member of OCaml's developement team. More specifically, I am > working on a test-driver for the OCaml compiler, which will be part of > OCaml's 4.06 release. > > I am currently writing an article to describe the tool and its > principles. In this article, I would like to also talk about how other > compilers' testsuites are handled and loking how things are done in GHC > is natural. > > In OCaml, our testsuite essentially consist in whole programs that > we compile and run, checking that the compilation and execution results > match the expected ones. > > From what I could see from GHC's testsuite, it seemed to me that it uses > Python to drive the tests. I also understood that the testsuite has > tests that are more kind of unit-tests, in the .T file. Am I correct > here? Or do you guys also have whole program tests? > If you do, how do you compile and run them? > > Any comment / hint on this aspect of the test harness' design would be > really helpful. > > Many thanks in advance, > > Sébastien. > ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
How does GHC's testsuite work?
Dear all, I am a member of OCaml's developement team. More specifically, I am working on a test-driver for the OCaml compiler, which will be part of OCaml's 4.06 release. I am currently writing an article to describe the tool and its principles. In this article, I would like to also talk about how other compilers' testsuites are handled and loking how things are done in GHC is natural. In OCaml, our testsuite essentially consist in whole programs that we compile and run, checking that the compilation and execution results match the expected ones. From what I could see from GHC's testsuite, it seemed to me that it uses Python to drive the tests. I also understood that the testsuite has tests that are more kind of unit-tests, in the .T file. Am I correct here? Or do you guys also have whole program tests? If you do, how do you compile and run them? Any comment / hint on this aspect of the test harness' design would be really helpful. Many thanks in advance, Sébastien. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
