On Thu, Sep 15, 2016 at 7:42 PM, Alan Field <afi...@redhat.com> wrote: > I also like this idea for a Unit-Based TCK for all clients, if this is > possible. > >> - we identify and group the tests depending on their scope (basic >> protocol ops, bulk ops, topology/failover, security, etc). A client >> which implements the functionality of a group MUST pass all of the tests >> in that group with NO exceptions > > This makes sense to me, but I also agree that the hard part will be in > categorizing the tests into these buckets. Should the groups be divided by > intelligence as well? I'm just wondering about "dumb" clients like REST and > Memcached. > >> - we assign a unique identifier to each group/test combination (e.g. >> HR.BASIC.PUT, HR.BASIC.PUT_FLAGS_SKIP_LOAD, etc). These should be >> collected in a "test book" (some kind of structured file) for comparison >> with client test runs > > Are these identifiers just used as the JUNit test group names? > >> - we refactor the Java client testsuite according to the above grouping >> / naming strategy so that testsuite which use the wrapping approach >> (i.e. C++ with SWIG) can consume it by directly specifying the supported >> groups > > This makes sense to me as well. > > I think the other requirements here are that the client tests must use a real > server distribution and not the embedded server. Any non-duplicated tests > from the server integration test suite have to be migrated to the client test > suite as well. I think this also is an opportunity to inventory the client > test suite and reduce it to the most minimal number of tests that verify the > adherence to the protocol and expected behavior beyond the protocol. >
Reducing the number of tests may not be so easy... remember that we need to test all versions of the protocol, not just the latest one. And we still need to test stuff that's not explicitly in the protocol, especially around state transfer/server crashes and around query (which the protocol says almost nothing about). More importantly, if I have to rebuild the entire server distribution every time I make a change in the HR server, then I'm pretty sure I won't touch the HR server again :) Cheers Dan _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
