Here's a summary of thoughts I've had over the last month or so. Please try to take the following as positive / constructive. I'm happy to be involved in trying to alleviate what some people may agree are problems. Or people may not agree that they're problems, which is fine too - more on this below.
A few months ago I started looking at various (mainly Python) frameworks with an eye to using one for a project I'm working on. When I found Twisted (and friends) I was really happy. I've been writing code for what seems like a long time (30 years) and I think I'm pretty critical and demanding when it comes to good and clean design. I certainly had not expected to find something like Twisted and its various additional components. Without having written a line of Twisted code, I made the decision to throw away a full year of my own low-level, efficient, debugged, C code (networking, custom RPC, argument marshaling, tons of sweat). That was a pretty serious step, not taken lightly, and, I still think, a good one. That's why I'm still here, spending hours writing this email, wanting to press ahead, and willing to put in time and effort to help. In one corner then, we have a great foundation - elegant, dynamic, well designed, full of potential, nice programming language, written by talented and responsive developers, etc. But... in the other corner, documentation that very often seems to be a complete shambles. There, I've said it. I don't claim that this is actually a problem. That depends. It may well be that I've happened along at a moment which is not optimal for my needs and abilities; that there is a well-organized design, development, and documentation trajectory; and that I just happened to arrive at a bad point on that curve. I fully sympathize with things being less well documented than they could otherwise be. I love writing code. Following the rise of highlighting editors I've even become quite fond of comments. But I'm not even slightly fond of writing external docs. I think an important part of getting things right in design is to implement something that feels right and then rework it until it really feels right and clean, and is also useful. There's little point in documenting things before they have settled. Writing code, debugging, re-designing, building apps on top of libraries - all those things, to me, should take precedence over documentation. I don't believe in writing fully fleshed-out specs, docs, and tests ahead of implementation. I even have little time or sympathy for people (like myself in this case) who don't have what it takes to use emerging tools before they're fully documented for use. I've spent plenty of my own free time in the last 20 years writing, giving away, and supporting, code I felt like producing. So I think I understand the equation pretty well. But, Twisted and its related components are released. Valuable (to the project) people are trying to use them. This can be frustrating. I think I have a reasonably high frustration threshold (but maybe I'm wrong - maybe I'm just old & cranky). It's quite general to talk about "Twisted and its related components". There are many moving parts (some conveniently accompanied by moving names). So let me just pick on Nevow - it's perhaps in the middle in terms of maturity, documentation, frustration level, etc. I actually don't mean to single out Nevow in particular, but it has qualities that make it a good example target for discussion. Apologies in advance to anyone this seems to pick on - that's not my intention, I have no idea who wrote (or didn't write) some of this stuff. I know Matt is involved as he's been kind enough to several times send me long explanations and code in email. Again, I'm not claiming there's a problem here. I find Nevow to be extremely promising and yet quite cryptic. Sadly, I've built dozens of web sites, many of them commercial, done under contracts with real deadlines, budgets, staff, etc. I hate editing ugly templates in awkward ugly non-code file formats. There's just so much disgusting stuff out there. So to see Nevow and Stan is a real pleasure. But to try to understand it... argh. Hop onto google and try "nevow documentation" and read through some other people's experiences (e.g. http://www.third-bit.com/pywebblog/archives/cat_twisted.html) The recent "nevow documentation" thread (kicked off by Manlio, at http://twistedmatrix.com/pipermail/twisted-web/2006-May/002652.html) is mainly illustrative for its lack of conclusion. I don't find the suggested page at http://divmod.org/trac/wiki/TitleIndex to have much (if anything) to do with documentation - it seems to be links to code. It's not clear if the wiki is more a help or a hindrance (this is probably too condemning, but after spending more than enough time looking at things that one later discovers are out of date, replaced, deprecated, etc., you start to wonder about everything). Nevow comes with many examples, and that's great. But they're still cryptic. E.g., a grep url\\. */*.py in Nevow's examples directory gets me over 30 hits, but I don't see that class documented anywhere and when I try to use it in my code it doesn't work. E.g., formless is used in many examples, but is apparently a dead end. If you disagree with that, you'll be reinforcing my sentiment that things are a bit of a mess :-). I can give more examples. There's quite a bit that seems very cryptic, and I mean beyond the things described in the already very terse and piecemeal Nevow documentation. Of course it takes time to spell things out, but if people want them I will. I already decided to take the evening off, have a couple of beers, and write this email. Before trying to finish on a politically correct, upbeat, & constructive note, here's a question: How was it that Twisted (or just Nevow?) managed to lose the attentions of Kieran Holland? He (she?) was clearly into Nevow, put a lot of effort into understanding things, and yet headed off to Django, which, reading between the lines, he/she didn't really think was even nearly up to the Twisted/Nevow standard. That's a bad loss. People who can read and write code and who are willing to spend serious amounts of time helping ease others into projects are so valuable. The Addendum of Kieran's Stan page (http://www.kieranholland.com/code/documentation/nevow-stan/) seems to spell things out pretty clearly. It feels like the Nevow (and broader?) community has done itself a serious disservice in losing Kieran's attention (but, see above comments on whether this is perceived as a problem, trajectories, etc). To me, totally on the outside, I do not get the impression that there is a clear design, code, and release trajectory in place. It feels like there is a bunch of very talented designers and coders who are banging out an increasing number of cool projects at high speed, leaving a frustratingly incomplete, misleading, and atrophying breadcrumb trail of documentation behind them. On to another question: do people think there a problem here, or not? The answer may be no. I think that's fine, I'm happy to be told to read the code and to shut up. At least that would be definitive, and not at all unwelcome. But, if you think there is a problem, what can we do? Here are some possible concrete steps that wouldn't take huge amounts of time. 1) Go through all Twisted & friends web sites and wiki and _ruthlessly delete everything_ that mentions obsolete, deprecated, or renamed projects / modules, broken links, example code using such, etc. There are many of these things, and they are highly misleading and frustrating and give a bad overall impression. 2) State, where appropriate, that the source code is currently the best documentation. There's no shame in this, and it's much better than having people run across and waste time on obsolete docs, or spend time looking for things that do not exist. E.g., "At this point the nevow.url class is totally undocumented. See examples/{x,y,z} for example usage and the code and comments in nevow/url.py for the best current documentation." 3) Have someone, preferably an outsider (could be me), write up and maintain a no-punches-pulled overview document about using the various Twisted components. Stick a link to this in a prominent place. http://twistedmatrix.com/trac/wiki/WebDevelopmentWithTwisted is a good start. 4) Have someone go through example code to stick in more comments, get rid of old stuff (no need to rewrite if that takes too long, just throw it away). E.g., the widespread use of formless in Nevow examples. Is it in, is it out? If the latter, get rid of it before more people die. 5) Have someone do some web searches for stuff known to be fixed or out of date, and send mail to the authors asking if they'd please consider updating or otherwise clarifying what they've written, could they add a link or a postscript, to help out future people looking to use Twisted & friends? No need to try to hide the fact that things were a bit of a mess - just aim not to have people get an out-of-date impression or follow links to incorrect / obsolete pages (e.g., links to nevow.com). I could probably come up with more suggestions. Given the inclinations of (I assume) many people still reading this message, no-one is too much inclined towards producing actual documentation (and it may not be the right moment, as above). So it makes sense to focus on easy hits: purging old crud, minimizing wasting the time of people looking to get involved, making it clear what the barriers are (e.g., point people at the source, tell them that at this point the only way to understand is to ask questions (but time is precious) and to read the code). OK, enough words for now. Thanks again for all the help, all the code, etc. Terry _______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
