Re: quality assurance (was: Re: GNUstep Base 1.11.0)
On 2005-07-27 02:02:56 +0100 Alex Perez [EMAIL PROTECTED] wrote: Lars Sonchocky-Helldorf wrote: How about using the donation money GNUstep got for buying some testing machines which would go into the same place as the GNUstep web servers are and could then be used for testing/developing GNUstep onto some more exotic platforms: Some *BSD, some Solaris boxen (be it x86 or SPARC), a Mac (Mac mini should be enough), maybe some HPUX box should be enough for the start. Those machines don't necessarily need to be new hardware, I guess we could get them used somewhere for cheap. Then put them online with special ssh accounts (maybe some VNC too) for the core devs and there you go: compilation and testing is possible remotely for those who need it. In a word, no. I think that would be a huge waste of the money, because we can get access to these kinds of exotic machines through places like sourceforge et al. who have large compile farms for EXACTLY THIS REASON. Why is their offering insufficient for us? (Seriously) I know nothing about what sourceforge offer, so I'm talking in pretty general terms here ... but IMO lack of man(woman)power is our biggest problem ... and to make good use of more machines/operating systems we would need to be able to work with them in a very automated way. We don't have the time to manually log on to lots of machines and build/check every code change on all systems/configurations even if we have the hardware and operating systems available to us. Of course, they would be useful for people to log on to and debug system specific problems, but in order to screen for problems in the first place I think, we would need to develop scripts which would run automatically/periodically to 1. retrieve the latest CVS code. 2. build it and run tests on it in various configurations. 3. in the event of a success, log it somewhere so that our website can display status. 4. in the event of a failure of a build or a test failure in any configuration, wait a while and try CVS again 5. after some period in which failures are consistent, email out alerts. I imagine this could keep the a machine occupied a lot of the time (if we are going to have it testing a lot of different configuration options). My guess is that it would not be acceptable usage for a public shared resource at sourceforge. ___ Discuss-gnustep mailing list Discuss-gnustep@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnustep
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
On Jul 27, 2005, at 1:04 AM, Richard Frith-Macdonald wrote: Of course, they would be useful for people to log on to and debug system specific problems, but in order to screen for problems in the first place I think, we would need to develop scripts which would run automatically/periodically to 1. retrieve the latest CVS code. 2. build it and run tests on it in various configurations. 3. in the event of a success, log it somewhere so that our website can display status. 4. in the event of a failure of a build or a test failure in any configuration, wait a while and try CVS again 5. after some period in which failures are consistent, email out alerts. Actually, I already have a script set up to do 1-3 (logs either success or failure - then it's up to the developers to sees what is wrong). The logs get posted daily to [EMAIL PROTECTED] While I'm at it, I'll reiterate my call for anyone who has a machine with a few spare cycles to help test GNUstep by running this script. You need a CVS checkout and the script in CVS: gnustep/Startup/scripts/test-gnustep Instuctions included. You don't need root permission to run the script and you don't need to tell me - it just works! ___ Discuss-gnustep mailing list Discuss-gnustep@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnustep
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
Am 24.07.2005 um 08:08 schrieb Richard Frith-Macdonald: Actually, I'm more concerned with getting bug reports at all ... I occasionally hear tales of MacOS-X users having tried GNUstep and given up because it's buggy ... but I don't see corresponding bug reports on savannah (few appear to come from MacOS-X users), so I suspect that most people can't be bothered to file a bug report, let alone provide a testcase. Some sort of an automated bug reporting system might help here. If GNUstep can write a ... please send this to bug- [EMAIL PROTECTED] ..., it could prepare a standard collection of files (config.log, ...) as well and just ask yes/no wether the report/ collection should be created and sent. For GUI-oriented systems like Mac OS X or Windows, there might be no working CLI-mailer. A prepared package in $HOME (/tmp is hidden in Mac OS X) to be sent as an attachment should be fine, then. Markus - - - - - - - - - - - - - - - - - - - Dipl. Ing. Markus Hitter http://www.jump-ing.de/ ___ Discuss-gnustep mailing list Discuss-gnustep@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnustep
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
On 2005-07-23 20:32:54 +0100 Lars Sonchocky-Helldorf [EMAIL PROTECTED] wrote: Branching is good in the way that we can point users of GNUstep to a bugfix release if they happen to encounter a bug in a GNUstep release instead of saying: use the CVS version which turned into some sort of standard reply in the case of a bug. That would give us greater freedom on changing things in the CVS while still having something useable for the users. This would also result in applications developed against a certain release instead of have an application requiring the CVS because a necessary bugfix is in there. Well, creating the branch is trivial ... but I wonder how many bugfixes would get into it ... having to fix both the head and the latest branch (possibly with different fixes) could chew up quite a bit of time. Mostly, when I have reservations about funadmentally good ideas, it's because I know I don't even have the time to do things I want to do and suspect that's true of everyone else too. That being said ... if we created a branch from each release, perhaps there is someone out there who would like to maintain the latest branch by copying back any fixes made to head? Think about your use-pattern of GCC: do you use the CVS or weekly snapshot version of GCC or do you use GCC 3.3.6 or GCC 3.4.4 because the later are a lot more mature than the first two? People don't want always live on the bleeding edge. My pattern is erratic ... sometimes cvs for a few weeks, more often whatever's in debian/unstable, depending on whether activity in ObjC support in the compiler has turned interesting. I think in general though, I just use what works ... and if it doesn't work, I generally update to latest release or cvs version I can find. And in turn this could improve the perception of GNUstep. I spoke to several Cocoa-developers who tried to use GNUstep (after I am told them to do so). The common response was that every time something else is broken, including things which already were working, which annoyed them a lot and was leading to the conclusion: GNUstep is just a hobbyists project I think the only thing which would help with that at all is a *lot* more manpower. I don't think things *often* get broken in obvious ways ... but in the gui/backend in particular there is a lot of stuff where interactions with external systems, timing of events etc can mean that small changes produce problems that don't show up under testing by the developer because the app(s) they are using for testing or the environment (window manager, speed of processor etc) doesn't trigger them. These sort of issues appear in the cvs code and get into the next release, only being detected *after* release ... because we don't have neough people using the cvs code. So to that extent, I think we must try to *encourage* people to use cvs frequently somehow (though with a clear understanding of the downsides of doing so). Hmm ... but how can we encourage lots of people to use/test CVS or prerelese snapshots? How about using the donation money GNUstep got for buying some testing machines which would go into the same place as the GNUstep web servers are and could then be used for testing/developing GNUstep onto some more exotic platforms: Some *BSD, some Solaris boxen (be it x86 or SPARC), a Mac (Mac mini should be enough), maybe some HPUX box should be enough for the start. Those machines don't necessarily need to be new hardware, I guess we could get them used somewhere for cheap. Then put them online with special ssh accounts (maybe some VNC too) for the core devs and there you go: compilation and testing is possible remotely for those who need it. That sounds good ... but I don't know if we have anything like enough money. There is the cost of the machines ... but there is also the issue of running them ... where would we put them so that they can be kept running continuously with a good network link ... what would the running costs be? Adam might know if this is practical... but it does seem a good use for any money we get. Even better than donating a machine to developer X because this way all can participate and share. 2. keep asking people to contribute regression testcases if/when they have the time. The fact that people had to install guile to run the old regression tests was an excuse for them not to help ... with the new testsuite that excuse no longer exists. maybe we should ask for testcases/bug isolations on the bug reporting page? I don't know if/how the savannah bug reporting page can be modified, but we did (do?) have that sort of request on the website. Actually, I'm more concerned with getting bug reports at all ... I occasionally hear tales of MacOS-X users having tried GNUstep and given up because it's buggy ... but I don't see corresponding bug reports on savannah (few appear to come from MacOS-X users), so I suspect that most people can't
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
On Jul 23, 2005, at 9:11 AM, Lars Sonchocky-Helldorf wrote: IIRC the odd minor release number (11 in 1.11.0) points out that this is an unstable release of GNUstep. No it's not. I don't use stable/unstable anymore. I don't really even know how to define the difference between the two. ___ Discuss-gnustep mailing list Discuss-gnustep@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnustep
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
On 2005-07-23 16:11:30 +0100 Lars Sonchocky-Helldorf [EMAIL PROTECTED] wrote: Am Samstag, 23.07.05 um 13:47 Uhr schrieb Benoit Astruc: Hello, I just tried to compil GNUstep Base 1.11.0, freshly download from GNUstep http) and I found that there is an error GSXML.m at line 5 142 : request = [tequest initWithURL: [NSURL URLWithString: connectionURL]]; should be request = [request initWithURL: [NSURL URLWithString: connectionURL]]; I suppose. Maybe it's corrected in CVS but it seems quite bad to have such error in the last stable version. IIRC the odd minor release number (11 in 1.11.0) points out that this is an unstable release of GNUstep. But nevertheless I think it is a bad sign that this kind of a bug (doesn't compile) could creep into a *release* of GNUstep. And even worse, it would not have been detected if Benoit would not have built this special configuration. The bug has gone unnoticed for a long time, it has been introduced with: http://savannah.gnu.org/cgi-bin/viewcvs/gnustep/gnustep/core/base/ Source/Additions/GSXML.m.diff?r1=1.78r2=1.79 on Sat Mar 12 17:38:18 2005 annotated with Patch to port GSXMLRPC to MacOSX which tells me the submitter did *not* test his patch before committing. Otherwise he would have experienced what Benoit experienced. Please don't get me wrong here. I don't want to do finger pointing because of a small bug that only occurs in border cases. My issue is something else: While we keep fixing things here while inadvertently destroying things there. This shows that we could need some sort of quality assurance in our development process. Otherwise GNUstep will never become really mature. There are several things we could do in this regard (all stolen from the GCC team) CVS check-ins: - require at least a compile test before things get checked in (this would catch this sort of bugs) - peer review of patches to be checked in - do a regression test before checking in things (needs test cases, see below) bug fixes: - write test cases that prove a bug is fixed releases: - create a CVS branch (not just a tag) for each release (this helps to insure that no new bugs are introduced in bugfix releases (caused by new features added meanwhile)) - maintain release branches for a while, at least until the next stable release (and port bugfixes back to mainline of course) - create a series of release candidates before actually doing the release some of those are very complex (peer review comes to my mind here) others not. I think we should at least require some of them What do you think? Well ... all those things are nice to have, but they are obviously of varying cost/benefit ratios. None of them caught this bug (and most of them are in routine use in varying degrees) ... because the sad fact is that nobody had this particular setup ... realistically there is no way to catch bugs in unusual configurations because people won't have the machines/operating systems or resources to change machine/OS/configuration etc, so you have to adopt one of two approaches ... 1. reduce all the options/configurations to a minimal set such that they can be properly tested/supported by the core developers.. 2. accept that the odd options/configurations won't get tested except when people who need to use them try them out. I think we generally prefer the second approach, but there is always a balance struck between the two extremes. Personally, I think that branching is more overhead than it's worth, but that regression testing is good, which is why that I wrote the regression testing framework we've been using up till now, and support the move to the latest one (because it's easier for ObjC programmers to write tests for). I don't think that a test on all platforms/configurations is reasonable before a checkin ... that's just not ever going to be practical. I do agree that a test of the standard gnu/linux build is needed ... but I think everyone does that. You say 'which tells me the submitter did *not* test his patch before committing' If I remember correctly, I committed that patch on behalf of the submitter, and I'm sure he tested the code before he sent it to us ... as he was using it himself. The fact is that accidental alterations to files/patches do occur between testing and being submitted/committed ... this particular single character change probably was a result of viewing in an editor for a visual check. I don't think it suggests that the submitter was bad/sloppy in the processes he used. So ... my vote for priorities would be .. 1. ask people to volunteer to test the odd configurations they have the machines for. If we had a list of people who actuyally used the less popular hardware, operating systems, and configuration/build options, we could do an email reminder asking them to test the latest cvs code just before a release (even better if they could do
Re: quality assurance (was: Re: GNUstep Base 1.11.0)
Am Samstag, 23.07.05 um 21:32 Uhr schrieb Lars Sonchocky-Helldorf: How about using the donation money GNUstep got for buying some testing machines which would go into the same place as the GNUstep web servers are and could then be used for testing/developing GNUstep onto some more exotic platforms: Some *BSD, some Solaris boxen (be it x86 or SPARC), a Mac (Mac mini should be enough), maybe some HPUX box should be enough for the start. Those machines don't necessarily need to be new hardware, I guess we could get them used somewhere for cheap. Then put them online with special ssh accounts (maybe some VNC too) for the core devs and there you go: compilation and testing is possible remotely for those who need it. Something else came to my mind: We even would need only one machine per architecture and could than do the rest with virtualization solutions like f.i. Mac on Linux for PPC ( http://www.maconlinux.org/ ): Install Linux on the Box as host system and then run Mac OS X, pure Darwin, Linux, the *BSDs in a virtual environment. This way we could kill (not just) two birds with one stone. regards, Lars ___ Discuss-gnustep mailing list Discuss-gnustep@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnustep
