A. Pagaltzis wrote:
I agree that this would be good. The test suite seems quite
awkward so far. I just can’t think of any ways to automate tests
to the extent you propose. Have you got any suggestions for a
format to use?
I know not everyone is going to agree with me on this, but my first
suggestion would be to cut down on the number of tests. Mark Pilgrim has
something like 3000+ tests in his UFP test suite. That's great; that's what
aggregator developers should be doing, since those kind of tests can easily
be automated. But that's not the level of testing that we can put up on a
wiki and hope people will manually contribute results.
What we need is fewer tests with more coverage, and that involves tradeoffs.
Take relative URIs as an example. I probably have 50 odd unit tests, but you
don't need all of those in a conformance test. You certainly don't need each
of those tested on each element that uses a URI. Pick one element that's
easy to test (say html content using img tags) and include a small subset of
the trickier URIs.
You create "good" images at the destinations that would be reached if the
parsing has been performed correctly and, where possible, "bad" images at
the destination that an aggregator might reach if they're doing something
wrong (often this won't be possible, but in many cases aggregators get
things predictably wrong). From this you get fairly comprehensive coverage
of the different forms of relative URI in a simple set of tests that can be
evaluated at a glance.
Now to test what level of support there is for relative URIs in other
elements, you just pick one reasonably complicated (but not the worst) test
case and test each element just once. In many cases this will involve
following a link to see whether it's going to the right place, but you can
make this more pleasant by including a "good" page at the expected
destination and, as with the images, a couple of "bad" pages in all the
places you think an aggregator might go if it's getting it wrong.
The tradeoff here is that you're assuming an aggregator that can handle all
the variations you've thrown at it in html content would also be able handle
those variations in other elements as long as it can pass the single test
that you've tested everywhere else. That may not always be the case, but
your other choice is an exponentially larger test suite that just isn't
manageable.
For most tests it's actually not that complicated though. If you're testing
something that can be seen somewhere in the UI, like say a title, date or
author, simple include exactly what the value should be right there in the
message content for that entry. When you're doing a set of tests along the
same lines (like say titles encoded in different ways) try and make sure
every test produces the same results so you can see at a glance if they're
all correct.
When testing things like the different forms of escaping which can have a
huge number of permutations, it's often easier to test several different
combinations at the same time in one test (similar to the html acid tests).
When you fail a test you can't easily tell the exact cause of the failure,
but it does cut down the number of tests that have to be checked.
And on the subject of huge sets of tests, it's obviously a lot easier to
test a single feed with 20 tests than it is to have to subscribe to 20
different feeds to run each test separately. However, it's sometimes a good
idea to also have each test separately in case you have an aggregator that
flat out refuses to subscribe to a feed because of one test, but would
otherwise be perfectly capable of passing all the others. I find it easier
to do this with a PHP script and some mod_rewriteing to make the URLs
looking prettier.
Also, when you need a lot of separate feed tests (either for the reasons
mentioned above, or for tests that can't be grouped) it can be helpful if
you include an opml file that can import all the tests in one go rather than
having to add them individually.
Some other suggestions...
Make sure your ids are always unique (I know this goes without saying, but
it's easy to overlook when you're throwing together a test). Some
aggregators will refuse to display entries with ids they've already seen
before even if the feed from which those ids came has long since been
deleted. It's also advisable to include an alternate link on all elements
unless that would interfere with your test since I think there are some
aggregators that fail to subscribe to feeds without element links
Avoid using xhtml content or weird escaping techniques unless you're
actually testing those features. I should probably double check this, but I
think the safest content type is html with named entity escaping (lt, amp) -
no numeric entities or cdata escaping. It's also advisable to keep your
dates very recent (this is easy if tests are generated via php and you can
make them up on the fly) since some aggregators won't display entries that
are older than a certain date.
That's about all I can think of for the moment.
Regards
James
