Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
On 11 August 2011 05:17, John Millikin jmilli...@gmail.com wrote: This is just a quick package I whipped up out of frustration with test-framework scrolling an error message out of sight, for the millionth time. Patches to make test-framework less noisy (either by default or with a flag) will be gratefully accepted, if anyone wants to give it a go :-) Max ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
Hi John, I am wondering if you have seen the hspec package? [1] It seems to solve all the problems you are with chell, including that it silences Hunit output. We are using it for all the Yesod tests now. Thanks, Greg Weber [1]: http://hackage.haskell.org/packages/archive/hspec/0.6.1/doc/html/Test-Hspec.html ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
I have, but it's not quite what I'm looking for: - I don't want to silence HUnit's output, I just don't want anything to show on the console when a test *passes*. Showing output on a failure is good. - I'm not interested in BDD. Not to say it's not useful, but it doesn't match my style of testing (which uses mostly pass/fail assertions and properties). On Thu, Aug 11, 2011 at 07:18, Greg Weber g...@gregweber.info wrote: Hi John, I am wondering if you have seen the hspec package? [1] It seems to solve all the problems you are with chell, including that it silences Hunit output. We are using it for all the Yesod tests now. Thanks, Greg Weber [1]: http://hackage.haskell.org/packages/archive/hspec/0.6.1/doc/html/Test-Hspec.html ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
I tried, actually, but couldn't figure out how to separate running the test from printing its output. All the attempted patches turned into huge refactoring marathons. When given the choice between sending a huge replace all your code with my code patch, and just releasing a separate package, I prefer to do the second. There's usually a reason a library behaves as it does, and this way both behaviors are available to users (even if I find one frustrating). On Wed, Aug 10, 2011 at 23:51, Max Bolingbroke batterseapo...@hotmail.com wrote: On 11 August 2011 05:17, John Millikin jmilli...@gmail.com wrote: This is just a quick package I whipped up out of frustration with test-framework scrolling an error message out of sight, for the millionth time. Patches to make test-framework less noisy (either by default or with a flag) will be gratefully accepted, if anyone wants to give it a go :-) Max ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
It silences HUnit's output, but will tell you what happens when there is a failure- which I think is what you want. There are a few available output formatters if you don't like the default output, or you can write your own output formatter. BDD is really a red herring. Instead of using function names to name tests you can use strings, which are inherently more descriptive. In chell you already have `assertions numbers`, in hspec it would be `it numbers`. The preferred style it to remove `test test_Numbers and the test_Numbers definition` which are redundant in this case, and instead place that inline where you define the suite, although that is optional. So I really can't tell any difference betwee BDD and pass/fail assertions. You still just use assertions in hspec. On Thu, Aug 11, 2011 at 7:36 AM, John Millikin jmilli...@gmail.com wrote: I have, but it's not quite what I'm looking for: - I don't want to silence HUnit's output, I just don't want anything to show on the console when a test *passes*. Showing output on a failure is good. - I'm not interested in BDD. Not to say it's not useful, but it doesn't match my style of testing (which uses mostly pass/fail assertions and properties). On Thu, Aug 11, 2011 at 07:18, Greg Weber g...@gregweber.info wrote: Hi John, I am wondering if you have seen the hspec package? [1] It seems to solve all the problems you are with chell, including that it silences Hunit output. We are using it for all the Yesod tests now. Thanks, Greg Weber [1]: http://hackage.haskell.org/packages/archive/hspec/0.6.1/doc/html/Test-Hspec.html ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
On Thu, Aug 11, 2011 at 07:52, Greg Weber g...@gregweber.info wrote: It silences HUnit's output, but will tell you what happens when there is a failure- which I think is what you want. There are a few available output formatters if you don't like the default output, or you can write your own output formatter. I'm a bit confused. From what I can tell, HUnit does not output *anything* just from running a test -- the result has to be printed manually. What are you silencing? BDD is really a red herring. Instead of using function names to name tests you can use strings, which are inherently more descriptive. In chell you already have `assertions numbers`, in hspec it would be `it numbers`. The preferred style it to remove `test test_Numbers and the test_Numbers definition` which are redundant in this case, and instead place that inline where you define the suite, although that is optional. So I really can't tell any difference betwee BDD and pass/fail assertions. You still just use assertions in hspec. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
I am confused also, as to both what output you don't like that motivated chell and what exactly hspec silences :) Suffice to say I am able to get a small relevant error message on failure with hspec. I am adding the hspec maintainer to this e-mail- he can answer any of your questions. On Thu, Aug 11, 2011 at 8:03 AM, John Millikin jmilli...@gmail.com wrote: On Thu, Aug 11, 2011 at 07:52, Greg Weber g...@gregweber.info wrote: It silences HUnit's output, but will tell you what happens when there is a failure- which I think is what you want. There are a few available output formatters if you don't like the default output, or you can write your own output formatter. I'm a bit confused. From what I can tell, HUnit does not output *anything* just from running a test -- the result has to be printed manually. What are you silencing? BDD is really a red herring. Instead of using function names to name tests you can use strings, which are inherently more descriptive. In chell you already have `assertions numbers`, in hspec it would be `it numbers`. The preferred style it to remove `test test_Numbers and the test_Numbers definition` which are redundant in this case, and instead place that inline where you define the suite, although that is optional. So I really can't tell any difference betwee BDD and pass/fail assertions. You still just use assertions in hspec. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
On Thu, Aug 11, 2011 at 08:17, Greg Weber g...@gregweber.info wrote: I am confused also, as to both what output you don't like that motivated chell and what exactly hspec silences :) Suffice to say I am able to get a small relevant error message on failure with hspec. I am adding the hspec maintainer to this e-mail- he can answer any of your questions. The output I didn't like wasn't coming from HUnit, it was coming from the test aggregator I used (test-framework). It prints one line per test case run, whether it passed or failed. That means every time I ran my test suite, it would print *thousands* of lines to the terminal. Any failure immediately scrolled up and out of sight, so I'd have to either Ctrl-C and hunt it down, or wait for the final report when all the tests had finished running. Chell does the same thing as test-framework (aggregates tests into suites, runs them, reports results), but does so quietly. It only reports failed and aborted tests. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
Is this different than the --hide-successes flag for test-framework? Looks like it was added a few months back: https://github.com/batterseapower/test-framework/commit/afd7eeced9a4777293af1e17eadab4bf485fd98f -n On Thu, Aug 11, 2011 at 8:21 AM, John Millikin jmilli...@gmail.com wrote: The output I didn't like wasn't coming from HUnit, it was coming from the test aggregator I used (test-framework). It prints one line per test case run, whether it passed or failed. That means every time I ran my test suite, it would print *thousands* of lines to the terminal. Any failure immediately scrolled up and out of sight, so I'd have to either Ctrl-C and hunt it down, or wait for the final report when all the tests had finished running. Chell does the same thing as test-framework (aggregates tests into suites, runs them, reports results), but does so quietly. It only reports failed and aborted tests. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
Possible -- I ran into dependency conflicts between t-f/t-f-q/quickcheck when trying to migrate to test-framework 0.4, so I clamped all my test subprojects to 0.3. On Thu, Aug 11, 2011 at 09:09, Nathan Howell nathan.d.how...@gmail.com wrote: Is this different than the --hide-successes flag for test-framework? Looks like it was added a few months back: https://github.com/batterseapower/test-framework/commit/afd7eeced9a4777293af1e17eadab4bf485fd98f ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
As Greg pointed out, HSpec does have an option to output just the failed tests. I looked at the example on the Chell project home page and converted the example tests into these hspec style specs: import Test.Hspec (Specs, descriptions, describe, it) import Test.Hspec.Runner (hHspecWithFormat) import Test.Hspec.Formatters (failed_examples) import Test.Hspec.HUnit import Test.HUnit import System.IO (stdout) -- some functions to test equal = (==) greater = () equalWithin = undefined equalLines = (==) specs :: IO Specs specs = descriptions [ describe number comparison module [ it can check for equality (assertBool 1 should equal 1 $ equal 1 1), it can compare order (assertBool 2 should be greater than 1 $ greater 2 1), it can compare eqauality with floating point numbers (assertBool 1.0001 should be close enough to 1.0 $ equalWithin 1.0001 1.0 0.01) ], describe text comparison module [ it can compare strings for equality (let str1 = foo\nbar\nbaz :: String str2 = foo\nbar\nqux :: String in assertBool foo\\nbar\\nbaz shouldn't equal foo\\nbar\\nqux $ equalLines str1 str2) ]] main = hHspecWithFormat (failed_examples True) stdout specs And when run, got the following output in red text since it's only reporting failures: ] x can compare eqauality with floating point numbers FAILED [1] ] x can compare strings for equality FAILED [2] ] ] 1) number comparison module can compare eqauality with floating point numbers FAILED ] Prelude.undefined ] ] 2) text comparison module can compare strings for equality FAILED ] foo\nbar\nbaz shouldn't equal foo\nbar\nqux ] ] Finished in 0. seconds ] ] 4 examples, 2 failures You can write provide your own formatter if that's not what you'd like to see. You also don't have to use the HUnit assertion text either; you could use the following function to make your specs even more like your Chell example, at the cost of losing the extra output description: assert = assertBool Hspec uses HUnit TestCases and assertions but also supports QuickCheck properties almost exactly the same way Chell does. The hspec project homepage (https://github.com/trystan/hspec) has more examples, including the specs for hspec itself. Trystan Spangler From: John Millikin jmilli...@gmail.com To: Greg Weber g...@gregweber.info Cc: trystan s trysta...@comcast.net, haskell-cafe@haskell.org Sent: Thursday, August 11, 2011 8:21:52 AM Subject: Re: ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework) On Thu, Aug 11, 2011 at 08:17, Greg Weber g...@gregweber.info wrote: I am confused also, as to both what output you don't like that motivated chell and what exactly hspec silences :) Suffice to say I am able to get a small relevant error message on failure with hspec. I am adding the hspec maintainer to this e-mail- he can answer any of your questions. The output I didn't like wasn't coming from HUnit, it was coming from the test aggregator I used (test-framework). It prints one line per test case run, whether it passed or failed. That means every time I ran my test suite, it would print *thousands* of lines to the terminal. Any failure immediately scrolled up and out of sight, so I'd have to either Ctrl-C and hunt it down, or wait for the final report when all the tests had finished running. Chell does the same thing as test-framework (aggregates tests into suites, runs them, reports results), but does so quietly. It only reports failed and aborted tests. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
On 11 August 2011 15:49, John Millikin jmilli...@gmail.com wrote:
I tried, actually, but couldn't figure out how to separate running the
test from printing its output. All the attempted patches turned into
huge refactoring marathons.
Just FYI test-framework already has exactly this split between running
tests and printing their results. If you had wanted to change this you
could have modified showImprovingTestResult in
https://github.com/batterseapower/test-framework/blob/master/core/Test/Framework/Runners/Console/Run.hs.
However, as someone else has already pointed out, the --hide-successes
flag does what you want, and you can even make it the default for your
particular testsuite by making your main be (do { args - getArgs;
defaultMainWithArgs tests ([--hide-successes] ++ args) })
Cheers,
Max
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] ANNOUNCE: Chell: A quiet test runner (low-output alternative to test-framework)
Homepage: https://john-millikin.com/software/chell/ Hackage: http://hackage.haskell.org/package/chell This is just a quick package I whipped up out of frustration with test-framework scrolling an error message out of sight, for the millionth time. Chell has the same general purpose (aggregate your assertions + properties + whatever into a single executable), but only prints when a test fails or aborts. It also has a small built-in test library, similar to HUnit, so you don't need to depend on 2-3 separate libraries if you're just doing simple tests. Cool features thereof: * it reports the line number of failed assertions * you can use $expect instead of $assert, so even if it fails, the test keeps going (all the failures are printed at the end) * you can add notes to a test, which are saved in logs and reports. you can put in any sort of metadata you want (nice for figuring out why a test is failing) * assertions for text diffs, so if you're testing two big chunks of text for equality you don't have to copy+paste to see what's different. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
