On Wed, Sep 25, 2002 at 07:35:24PM +0200, Warly wrote: > Thanks to you all for your precious help.
And thank you to all of the Mandrake employees for their hard work in producing 9.0. > During last 6 months period, and especially in the last beta period, > some of you give some advice/critic/flame regarding Mandrakesoft > development process. I'll admit I'm one of those advisors, critics and flamers. Before I go into what I think will be my analysis of the development process I want to make it clear this isn't personal. I really do appreciate the hard work that the Mandrake crew puts in. My criticism is aimed at an imperfect process which I think could be improved. And the improvements really should help the developers, Mandrake as a company, contributors and IMHO end product quality. However, I don't think that there is a perfect process. Only a possibly better process. :) > * send a mail to the changelog disk when packages are removed with > the reason This is an excellent suggestion. I couldn't agree more. > * improve the cooker cooker FAQ pages, about cooker etiquette and > everything when reporting a bug > (http://www.mandrakelinux.com/en/cookerfaq.php3) Agreed 100%. > * improved bugzilla to have a easy mail interaction system, and a more > friendly interface. And to have a last known problems page. This demands some flushing out. While the above issues I think pretty much everyone will agree with, people seem to be on various sides of the fence on this and it seems (at least from your comments in the past) that Mandrake's employees are dead set against this. So to flush it out in what will likely be a very long response - A response which I hope will lay out the problems with the current situation, the strengths with the current situation and perhaps a way to get the best of both worlds. I want to start with looking at the status quo before I go into how to solve the problem (an obvious starting place, but nonetheless lots of people seem to be spewing solutions without considering the problems). Problems: a) Various places have inconsistent reporting instructions and requirements. Some people/sites tell people to post to mandrake expert, others to bugzilla or even the cooker mailing list. There abounds confusion as to where to report and what to include in your report. b) The mailing list is not easily searchable for people who aren't maintaining their own archives. Before everyone starts pointing me off to the web archives at theaimsgroup hear me out here. I often have difficulty finding messages there that I know were posted. Frankly the search on that site is not all that effective. But even considering that, Mandrake doesn't even link to it anywhere that I know of. No mention of online archives are made on the cookerdevel.php3 page. You can find an archive listing on the Lists page but that only goes to Mandrake's archive that isn't searchable. We criticize posters for reposting the same bugs time and time again yet we don't provide them easy access to find out if their bug has been reported and even possibly fixed. Thierry even states repeated bug reports are a source of irritation for him and I'm sure he isn't alone. c) Bug reports are consistently "bad." This usually means one thing. That the report didn't contain enough information to do anything about. Bug reports come in about X11 not working right without listing the video card. Users report problems with packages without reporting the versions they are using. Users report an error message is displayed without telling what the error message is. Random crashes get reported without any details regarding what was going on at the time. I would imagine that a considerable amount of developer time is spent writing emails requesting further details that probably should have been included in the original email. d) The mailing list has too many posts to it. Some people think the list has too much unnecessary chatter and discussion of things that don't matter. I'm not sure I agree. But I'm listing it here for completeness. However, I can certainly see that reporting a bug by a newbie seems rather overwhelming when they are told to report it here. Most of these people do not have time to read the list for replies to their issues. Nor do they have sophisticated email filtering that allows them to pull out what is important. Nor should they necessarily have to. e) There is not enough followup. Many users, myself included, feel there is not enough follow up on bug reports. While I understand a lot of them are poor, even good reports with fixes get neglected at times. This is bound to happen when developers receive, as Thierry put it, "thousands of messages a day," especially when many of them are irrelevant to what they work on. Some are bound to be lost in the hustle. As it stands, the only way to be sure your bug got fixed is to test again or possibly see it on the changelog list. However, most novice reporters are not going to subscribe to the changelog list. CVS is also an option, but how many novice users know how to work CVS? f) Problems with list stability. I don't think I really need to go into this too much because I think we all know what I'm talking about. However besides the obvious downtime some messages don't get through or at least some people don't receive all the emails. This certainly happens to me from time to time and developers too. This also leads to bugs getting missed. But most of all it leads to tons and tons of reposts. There is no instant feedback to know your bug has been received. Even the online archives sometimes don't get emails that were sent. g) There is no way to tell the stability and status of the distribution. Without a single place to report bugs, QA, developers, Mandrake executives, and even users have no way of gauging the quality of the distribution. Bug reports are scattered everywhere. Reports in the current bugzilla database or mandrake expert aren't a good way to gauge because hardly anyone posts bugs there. Mailing list traffic is also an imperfect way to gauge. Unless people constantly keep reposting problems it's hard to tell if things have been resolved or if people are just being quiet in the anticipation of a release. How many people haven't continued bugging the list about a bug because they thought it was too late for the release... Strengths: a) The list is wonderful at back and forth interactions. b) We've developed methods of dealing with large amounts of email and work around that. c) Bad reports are easy to get rid of. You just hit the delete key. d) It's a push system rather than pull. Meaning the reports come to us rather than requiring us to go get them. Solutions: 1) Mailing list changes or a Newsgroup. I'm gonna lump these together because they are the same thing, just a different technology - but ultimately they are basically just emails. There is no reasonable way to make the mailing list solve all of the above problems, or even make dent in it. (a) could be solved by still using the mailing list but not telling people other places to post bugs. (b) could be solved by working on the searching of the list. As far as (c) goes, ensuring quality reports on a mailing list is impossible unless you restrict posting. And then you're into something far less useful in getting bug reports because good bug reports can come from anyone. Plus, restricting posting would likely exacerbate the issue with multiple places to report. I think the only solution to (d) is to restrict posting... but honestly I think a lot of useful (even if seemingly extraneous to some people) discussion goes on here. And restrictions on posting is rather contrary to the "free speech" sort of ethic Mandrake and Open Source tries to uphold. (e) is difficult to do with a mailing lists. Nobody knows who is dealing with a given issue for sure. Pixel may assume that GC will reply to an issue and delete it (this is just an example). Then the issue gets forgotten. Auto replies are not useful responses. (f) is a hard issue to fix. Mailing lists always will have dropped mail and the lack of good instant feed back. And again auto replies don't ensure to you that everyone received your bug report that should. (g) Is just not possible to do from an automated standpoint. It's a crap shoot guess to figure out where we are at fixing things from the mailing list. Again same basic issues exist with a newsgroup. It's just a different technology to access to the messages. But it's still got all the same issues. 2) Bugzilla, anthill or some other bug database system. I think this is the best solution. Not necessarily ideal, but the best. But I think it does need to be open to posting by everyone. Priority of resolution might be up for voting. I think Buchan's suggestion of volunteers to help go through the bugs will work. Plenty of people have offered to do this so far... I disagree with fcrozat that it'd take too many people. Many of us would be willing to do more than one package. And some packages that have very few bugs reported on them probably wouldn't need a volunteer. Alastair seemed to dislike the idea of a web interface. But I don't think that's really valid. There's no reason that most of the interaction with bugzilla couldn't be done via email much as KDE's old bug system did (though I'm not sure what they are doing now). I don't think we should post all bug reports put in bugzilla on this list however. Perhaps a separate list... similar to how changelog is run. Replies for discussion of bugs on the list could be CC'ed to [EMAIL PROTECTED] and automatically added as a comment in the bugzilla database. Attachments can also be added automatically. fcrozat brought up a good point about eventually being able to migrate bugs to Mozilla, KDE or Gnome. I think it's telling that all 3 of these large projects are using this system. Now that I've covered the general arguments regarding bugzilla, let's cover how it solves the problems... (a) Just make it the only place to post. (b) bugzilla provides us with good searching. Duplicates can be closed by volunteers and linked to the open bug report, freeing developers to pay attention to the real bug reports. (c) KDE reduced bad bug reports by asking lots of questions and trying to walk a person through a search of the bug database before posting. Simply requiring users to enter relevant package versions, hardware information etc... would decrease developer's having to go back and ask for more information constantly. Poor bug reports could be moderated, closed, etc by volunteers, leaving developers with more time to deal with the issues that filter up to them. Plus, if bugzilla is setup to route bugs only to the maintainer of the package, we decrease the number of bad bugs developers have to go through. Bugzilla, however, does still allow for interactive question and response via email... Developer posts a response. The reporter and people who are interested in the issue get an email. Someone replies to the email and their response goes back in bugzilla. Everyone else gets a copy of it in their email box. Basically the same as the mailing list, except it's stored in a database and only truly interested parties get it (though there could be a list with *ALL* the traffic for masochists). (d) This would be solved too. The list would become a discussion area for just developers. Users wouldn't feel the need to come here much. The list would be easier to read and more relevant. Specifics to certain packages would stay on bugzilla. The few bug reports that showed up on the list could be told to report it on bugzilla. (e) Followup is built into the process. Reporters would get emailed as to the status of their bug(s) and all comments or changes made to it. (f) Not an issue. The list is less important. Users get instant feedback via email showing their bug report id(s) and URL(s) to look at their report. They can see their report in the searches. (g) It's easy to get a report of overall bug count. Even with bad bug reports we'll eventually get an idea of how many bugs we should get down to before we think things are looking stable. Basically we can maintain the benefits of the mailing list while resolving to some degree the issues we have now. I firmly believe that even spending a few seconds deleting lots of duplicate reports adds up to a lot of time. Plus, asking for more info takes time. Add the time up among all the developers and you've got a lot of time that could be spent improving the distribution not just reading emails. Now some of that will be offset by bugzilla maintenance. But I don't believe it would be all of it. Remember, duplicate bug reports go to everyone now, not just the package maintainers. Even if we don't really read the email, it takes us a few seconds to determine it's not relevant and we need to delete it. That wraps up the bug reporting issues. On a simple development issue, can we get "good" changelog entries in both CVS and in rpms? Things like "fix bug" or "cvcp commit from..." are not useful entries. Many people are making really poor changelog entries with their commits and packages. That makes it all that much harder to track things down 6 months from now when we're looking at what was done for security issues. Not to mention it means sometimes I have to go into CVS and look at the diff to get any idea of what was done. Further, some changes didn't get marked in the changelog at all. The Konqueror bug on Copy Link Location never showed up in the changelog. I know it got pulled in because, well, it works on my machine. But I have no idea when it got pulled in. Possible Laurent updated from CVS but there doesn't seem to be a matching changelog entry. That's it for my critique. -- Ben Reser <[EMAIL PROTECTED]> http://ben.reser.org Never take no as an answer from someone who isn't authorized to say yes.
