Niklas Gustavsson wrote:
> On Sun, Jun 21, 2009 at 11:14 AM, Bernd Fondermann<[email protected]>
> wrote:
>> For testing new functionality (mostly handlers), starting the server is
>> not neccessary (and that's a big plus). Just testing the handler in
>> isolation is usually sufficient.
>
> I should have pointed out that I did not suggest to in any way replace
> the current testing (nor step away from the design that allows
> handlers and other classes to be tested in isloation), that should
> still be done!
And I didn't understand it like you were suggesting this.
>> At some point, we might need that integration tests, to make sure
>> everything works properly together, though.
>
> Yeah, that's kind of the experience we've had with FtpServer. The
> integration tests has located tons of bugs, and allows us easily to
> write tests for cases reported by users ("server does this when my
> client does that").
>
>> What do you intend to use this end-to-end testing for?
>
> I would say that we should intend to test most, if not all, of the
> client facing functionality this way (as well as with more
> fine-grained test on each class).
Mmhh. don't know. I agree with the "users reported odd behavior"
approach. I agree to test those stuff that can go wrong on a socket with
a real socket.
But keep in mind: XMPP has a pretty expensive initial handshaking protocol.
>> Integration tests for XMPP may prove difficult, because
>> + XMPP is XML and the way elements are closed or allowance of arbitrary
>> whitespaces can add for a lot of noise, elements order may be arbitrary etc.
>> + stanzas may come out of order, too
>> + extensions are manyfold
>> + tracking session state is easier with our own testing facilities
>
> I probably (surely) don't understand all the details here. But, my
> idea would be to use Smack as the client which at least should isolate
> us as much as we want from the protocol (but still allow us to create
> arbitrary packages for testing odd cases). I also think this would
> handle out of order stanzas and extensions. Not sure about session
> state.
You could be wrong here. There are no out-of-order stanzas in XMPP which
you'd have to test for. This is completely ok for this protocol. But
receiving and sending things *will* make end-to-end tests harder to succeed.
Some stanzas, though, are prohibited/valid at certain session states.
There is a separate layer taking care of this (see ProtocolWorker and
SessionState). So there is no need to test this on a socket, because the
socket does not add anything useful to the test (except for odd cases
maybe, but not in general).
Session states are very important in XMPP. Ca. 50% of RFC3920 deals with
the states the session goes through on initial handshake.
(see http://tools.ietf.org/html/draft-saintandre-rfc3920bis-09 chapters
4, 5, 6, 7 and 8)
So, Vysper uses a state machine to track this. Which works independent
of a socket. It should just work for any stream of stanzas and is
testable on this level. See for example all subclasses of
AbstractProtocolStateTestCase.
XMPP is different to line-oriented protocols like FTP, SMTP, Telnet etc.
XMPP is using a subset of XML, so it is less strict about syntax (the
set of wellformedness is larger).
<iq id="ab1" result="cancel" />
is semantically the same as
<iq
result='cancel'
iq='ab1'
>
</iq>
That's why I saw very early on that it's very painful to test against
the character stream. Now I prefer to test everything that deals with
stanzas against Stanza objects.
XMPP is not only specified for socket transport. It's defined BOSH (over
HTTP) as well. I'm sure you wouldn't want to double every e2e test for
both transports?
>
> Again, let me point out that these tests should never replace the
> current tests but merely test at a different abstraction.
Yes, I agree this could prove useful. But in most scenarios I can think
of at the moment, more unit-like tests would be
+ simpler
+ easier to write and maintain
+ running much faster
+ succeed more often ;-)
But we're kind of talking very theoretical here (otherwise my answer
could have been shorter and there'd be somebody still listening down
here). Let's talk about a concrete example. For the moment, I'm -0 and
done issueing my little warning ;-)
Bernd
