chromatic wrote:
However, trying to design in sufficient flexibility that the toolset never
needs to be upgraded despite everyone everywhere throwing in new key/value
pairs willy-nilly misses, I think, one very important point:
All* of the TAP I've ever seen has been transient data generated by one tool,
processed by another tool, and almost immediately discarded.
This is a situation which is changing out from under us, and we're just doing
what we can to support it. New TAP consumers are being written all the time
by non-Perl people (Python, MySQL and Mediawiki that I know of) and it's only
a matter of time before somebody writes another parser. So the idea that
there's going to be one canonical producer and one canonical parser is going
away.
It's also clear that there's value in storing TAP to examine and replay it
later. Smolder is just that. Also things like the pugssmoke HTML test
matrix. So the idea that TAP is transient is also going away.
Let's make that two points:
TAP and Test::Builder succeed by making tests unambiguous. They passed or
they didn't. True or false; that's it.
This is still an absolute priority. It wasn't made clear that the YAML keys
are all entirely optional and (so far) are purely informational and have no
effect on the outcome of the test.
If somebody wants to customize their parser to make diagnostic keys
significant, they're free to do that but they've wandered outside the box of
what is generally safe.
Universal and pain-free extensibility is a nice goal, I suppose, but I've
never actually seen it in practice. It usually fails because of explosive
ambiguity.
YAML parsers right now *should* ignore extra fields. If people want to add
extra fields, that's fine. They can write their own parsers. If TAP-diag
needs to add fields in the future, the existing tools can do so as well.
I think we're having a violent agreement. The idea is to ignore extra fields,
otherwise we can't even add official ones. We simply want to reserve a set of
keys so we don't conflict with future user defined fields.
(If you really want to start a free-for-all, throw in an officially
recommended "metadata" hash and laissez votre mains under that.)
Again, we're talking about processing documents generated and almost always
immediately thrown away after processing. Any organization which can't fix
the handful of pieces which generate TAP-diag and the one or two pieces of
code which consume TAP-diag in an afternoon is doing something wrong with
regard to duplication and maintainability, and it'll take quite a few
libations to make me believe that such organizations are actually testing
their code in any rigorous sense.
I think this jives with my problems with the X-foo user keys. The emphasis is
all about making sure the TAP is valid. If you're spending your time double
checking that your TAP producer outputs valid TAP, rather than reading the
test results to see what's caused the test to fail, something is Very Wrong.
--
124. Two drink limit does not mean first and last.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
