Mark Hindess wrote: > I think it might help this disagreement if we step back and decide > what scenarios for running tests we are trying to optimise. > > Personally, whenever I write tests I'm doing it to optimise the > scenario where a new users comes to the project and does: > > 1$ svn co ... classlib # or wget/tar if you prefer > 2$ cd classlib/make > 3$ ant test > > I do this because I want the tests to be easy to run by new users > particularly on new platforms. I hope making it this easy means we > get more people running tests and that we get a broader set of results > than we could acheive ourselves. > > This scenario becomes slightly less pleasant if, between steps 2 and > 3, [*] you have to find your nearest Linux machine and install Apache > httpd, your-favourite-ftp-server, Dante socks server, etc. This > typically means that I'd be inclined to write stub servers to test > against. > > But this doesn't mean George is wrong! Because *if* there was a > publically accessible Internet server that already had Apache httpd, > twoftpd (my favourite ftp server this week), Dante socks, etc, then > the scenario I like to optimise becomes possible. The effort of > setting up one hosting server is definitely cheaper than the effort of > implementing stubs - I know because I set up the server George tests > against and it didn't take much time at all.
While I don't object to having such publically accessible services (though I would understand if ASF did), that cannot be the required set-up for the tests. We need to maintain the ability to test locally (can you imagine Geir's Skyfone charges otherwise ;-) ) > Having one server means we wont get 1000's of users asking Apache > httpd, twoftpd, Dante socks configuration questions on our mailing > lists. It also means we have a way to see the other half of the > results - that is, the server logs. (Stubs should make this easier > and this cost should be considered too but if we ran the server we > could make this easier.) > > George's approach continues to be cheaper even if a few groups have to > set up there own servers - though I think the set of users who "don't > have Internet access but do have a mechanism for getting up to date > Harmony code" should be small and getting smaller. Of course, it > becomes much more expensive if everyone has to do it. The size of development 'group' sharing the test server resource can be 1 upwards, so while sharing makes sense it should not be required. Regards, Tim > I've never tried using George's approach on past projects because I've > either also been writing server code, that can/should be used by > tests, or because the server was massive and I didn't have the > experience or resources to run a real server. Because of this, > initially, I was inclined to support the implemention of stubs, but > now I'm not so certain. > > I think it basically comes down to whether or not we can provide a > central server. I'm not sure it's a simple question but for me this > question is key to this issue. > > Or perhaps I'm optimising for the wrong scenario? > > Regards, > Mark. > > [*] More likely people would run step 3 find it goes horribly wrong > then read the README they ignored initially and discover why. ;-) > -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
