David Storrs wrote on 11/8/18 11:10 AM:
Will test successes show output as well?  For me that's the hard stop on using RackUnit -- the fact that it shows nothing on success means that there's no way to tell the difference between "everything ran and passed", "some things ran and passed but some things that should have run did not", and "nothing ran".

I've felt the same about this -- it still bothers me a little. (Examples: In DrRacket, there's always a possibility that test-running has been disabled in a preference setting that's buried in a nonintuitive place, several hops deep in UI navigation, not a top-level indicator, but that someone might turn off accidentally when trying to adjust a different behavior.  Or, on the command line, you might not realize you're in the wrong directory to pick up the tests you're expecting.  Or maybe you'd like at least a crude idea of how much tests are running for a bit of unfamiliar code.)

Perhaps the difference between "test failures" and "code errors" here (neither of which you'll see in DrRacket, if there's not a problem) is that, when you run code as a program, there is usually some other behavior you can see, to know that at least some code has been processed.  Not so for passing tests.

My original portable Scheme test framework always printed a full log, including ending with a summary like:

    ;;; END "Foo Station" TESTS: FAILED
    ;;;     (Total: 4  Passed: 3  Failed: 1)

My later Racket-specific test framework, Overeasy, used the logger for non-test-failure messages.  But probably the Racket logs are not normally visible to most programmers, so this isn't great.

Absent a richer test metadata mechanism in DrRacket and `raco test`, I'm not sure how to do this better for both DrRacket and command line.

So... Would the following simple two-part approach (special-casing DrRacket) work, or does someone have a better idea?

* When tests are under current DrRacket, we could use stdout for a one-line successful tests summary message, and stderr for a one-line tests failure summary message of the same format.  The different streams are so that we avoid misusing the HCI effect that DrRacket achieves when it colors errors in angry red.  (I think this doesn't break much, since programs under DrRacket won't be using stdout and stderr for old Unix "software tools" composition anyway.)

* When tests are run under `raco test`, we could use either stdout or stderr for the one-line summary message.  The question then is whether any CI-like frameworks that run tests like `raco test` currently treating any output as failure (I'm not certain, offhand).  We could always kludge an environment variable to inhibit the summary for unusual situations, but I'd like to keep this kind of thing simple and clean.

An alternative to the above would be to expose test metadata from the `test` submodules somehow, such as with "push" calls out akin to how `error-display-handler` works, or by a convention of `test` submodules providing a "pull" interface for programs like DrRacket to query.  Making that work requires changes to core Racket code. (And, as usual with this kind of facility, probably half of everyone will think it's being done wrong.)  The simple stdout/stderr convention above, special-casing DrRacket, OTOH, can be done easily, in a distributed and non-disruptive way.

You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to