In general, you should look at what DrDr does to decide how to run some tests. For instance, with Rackunit, you can look here:
http://drdr.racket-lang.org/27688/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/ and here in particular: http://drdr.racket-lang.org/27688/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt One thing that you will see is that as part of testing there are some failing tests to see what happens, but the tests deliberately switch current-error-port with current-input-port so that it does not result in STDERR output that DrDr would count as a failure. Jay On Sat, Nov 2, 2013 at 10:21 PM, David T. Pierson <d...@mindstory.com> wrote: > What is the correct way to run the tests which test rackunit? > > I am running the following after building with a clean tree: > > $ racket/bin/raco test pkgs/rackunit-pkgs/rackunit-test/tests/rackunit > > This seems to show several "failures", though since it is testing rackunit > I'm pretty sure most if not all are intentional (to verify that things > which should fail do fail.) The full output is below [*]. > > What is not clear is how do people run this and determine whether they > have any unintentional failures? Is there a separate command I should > be using? Perhaps people just diff the output from a clean tree vs > the output with their changes? > > BTW, I have read: > > http://blog.racket-lang.org/2012/11/tutorial-contributing-to-racket.html > > which has been very helpful since Joe's example was a change to > rackunit, but I didn't see anything addressing this problem. > > For additional context: my motivation for running these tests is to improve > check-exn so it no longer passes when given a non-thunk. I think it > should raise an error instead. I already have a patch for this but > running the tests has me thinking I'm not using the correct command to > test rackunit. > > David > > [*] Here is the output from the above 'raco test ...' command, with stdout > and stderr redirected to a file. Note the redirection changes the > ordering of some lines compared to the terminal output. > > > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/base-test.rkt" > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/check-info-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/check-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/counter-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/format-test.rkt" > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/hash-monad-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/location-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/log.rkt" > -------------------- > FAILURE > name: check-true > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/log.rkt> > 38 2 1185 15) > expression: (check-true #f) > params: (#f) > > Check failure > -------------------- > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/monad-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/pr10950.rkt" > 1 test-------------------- > ERROR > Outta here! > context...: > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:133:29 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:58:0: > top-level-check-around > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:119:21: > check-pred492750 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt: > [running body] > f88 > loop > loop > loop > f88 > > /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: > [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body] > > -------------------- > -------------------- > FAILURE > name: check > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt> > 44 0 1344 17) > expression: (check = 1 2) > params: (#<procedure:=> 1 2) > message: 0.0 > > Check failure > -------------------- > -------------------- > ERROR > First Outta here! > context...: > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt: > [running body] > f88 > loop > loop > loop > f88 > > /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: > [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body] > > -------------------- > -------------------- > error > ERROR > Second Outta here! > context...: > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt: > [running body] > f88 > loop > loop > loop > f88 > > /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: > [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body] > > -------------------- > -------------------- > FAILURE > actual: 1 > expected: 2 > name: check-eq? > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt> > 19 12 520 15) > expression: (check-eq? 1 2) > > Check failure > -------------------- > -------------------- > failure > FAILURE > actual: 1 > expected: 2 > name: check-eq? > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt> > 20 21 558 15) > expression: (check-eq? 1 2) > > Check failure > -------------------- > -------------------- > FAILURE > actual: #"" > expected: #"Oh HAI!\nI didn't run\n" > name: check-equal? > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> > 29 4 956 23) > expression: (check-equal? cout #"Oh HAI!\nI didn't run\n") > > Check failure > -------------------- > -------------------- > FAILURE > actual: #"" > expected: #"--------------------\nERROR\nOutta > here!\n\n--------------------\n--------------------\nFAILURE\nname: > check\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-check-test.rkt> 44 0 1344 > 17)\nexpression: (check = 1 2)\nparams: (#<procedure:=> 1 2)\nmessage: > 0.0\n\nCheck failure\n--------------------\n" > name: check-equal? > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> > 30 4 984 23) > expression: (check-equal? cerr #"--------------------\nERROR\nOutta > here!\n\n--------------------\n--------------------\nFAILURE\nname: > check\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-check-test.rkt> 44 0 1344 > 17)\nexpression: (check = 1 2)\nparams: (#<procedure:=> 1 2)\nmessage: > 0.0\n\nCheck failure\n--------------------\n") > > Check failure > -------------------- > -------------------- > FAILURE > actual: #"" > expected: #"--------------------\nERROR\nFirst Outta > here!\n\n--------------------\n--------------------\nerror\nERROR\nSecond > Outta here!\n\n--------------------\n--------------------\nFAILURE\nactual: > 1\nexpected: 2\nname: check-eq?\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 19 12 > 520 15)\nexpression: (check-eq? 1 2)\n\nCheck > failure\n--------------------\n--------------------\nfailure\nFAILURE\nactual: > 1\nexpected: 2\nname: check-eq?\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 20 21 > 558 15)\nexpression: (check-eq? 1 2)\n\nCheck failure\n--------------------\n" > name: check-equal? > location: > (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> > 30 4 984 23) > expression: (check-equal? cerr #"--------------------\nERROR\nFirst Outta > here!\n\n--------------------\n--------------------\nerror\nERROR\nSecond > Outta here!\n\n--------------------\n--------------------\nFAILURE\nactual: > 1\nexpected: 2\nname: check-eq?\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 19 12 > 520 15)\nexpression: (check-eq? 1 2)\n\nCheck > failure\n--------------------\n--------------------\nfailure\nFAILURE\nactual: > 1\nexpected: 2\nname: check-eq?\nlocation: > (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 20 21 > 558 15)\nexpression: (check-eq? 1 2)\n\nCheck > failure\n--------------------\n") > > Check failure > -------------------- > passed > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/result-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt" > 127 success(es) 0 failure(s) 0 error(s) 127 test(s) run > 0 > -------------------- > Failures > Intended to fail > Intended to fail > FAILURE > name: fail > location: > pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:47:33 > params: Check failure > -------------------- > -------------------- > Failures > Also intended to fail > Also intended to fail > FAILURE > name: check-eq? > location: > pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:48:38 > actual: 'apples > expected: 'orange > Check failure > -------------------- > -------------------- > Failures > Yet again intended to fail > Yet again intended to fail > FAILURE > name: check-equal? > location: > pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:49:3 > actual: "apples" > expected: "oranges" > Check failure > -------------------- > -------------------- > Failures > Intended to throw error > Intended to throw error > ERROR > testing: <<This is an error message>> > context...: > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/result.rkt:99:3 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:28:2 > the-tests > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:60:0: > apply-test-suite > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/text-ui.rkt:238:0: > run-tests14 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt: > [running body] > f88 > loop > loop > loop > f88 > > /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: > [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body] > > -------------------- > -------------------- > Failures > Error within a check > Error within a check > ERROR > name: check > location: > pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:51:37 > params: #<procedure:error> > 'foo > 'bar > error: contract violation > expected: string? > given: 'bar > argument position: 2nd > other arguments...: > 'foo > context...: > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:133:29 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:119:21: > check491437 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/result.rkt:99:3 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:28:2 > the-tests > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:60:0: > apply-test-suite > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/text-ui.rkt:238:0: > run-tests14 > > /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt: > [running body] > f88 > loop > loop > loop > f88 > > /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: > [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body] > /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body] > > -------------------- > 0 success(es) 3 failure(s) 2 error(s) 5 test(s) run > 5 > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt" > Oh HAI! > I didn't run > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt" > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-case-test.rkt" > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-suite-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-test.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/text-ui-test.rkt" > raco test: > "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/text-ui-util-test.rkt" > 55/340 test failures > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/tl.rkt" > raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/util-test.rkt" > _________________________ > Racket Developers list: > http://lists.racket-lang.org/dev -- Jay McCarthy <j...@cs.byu.edu> Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93 _________________________ Racket Developers list: http://lists.racket-lang.org/dev
