On Tue, Oct 14, 2003 at 12:25:02AM +0200, Menno Jonkers wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Toad wrote: > > | I would like to point out ONCE AGAIN that we are nowhere near 1.0. > > Development may not be, but your users seem to be. Apparently there's > quite a bunch of people that judged the quality of the software of 6-12 > months ago to be sufficient to start actually using it. Congratulations > on a job well done I'd say. > > But now the cat's out of the bag and they won't accept it that their > Freenet is crippled because of ongoing development efforts. Now we can > remind those people that they're simply beta testers and that they have > no right to complain if things don't work well for a few months, or we > can try to find a way to accommodate them. > > Apart from that, being "nowhere near 1.0" after 2004 minus 1999 years of > development might call for a redefinition of what's considered 1.0. Can > one expect real users to wait 5+ years? Can one determine priorities for > development if there's not a large user base showing what the actual > strengths and weaknesses of the software are?
Absolutely. Look at Wine! > > I don't think so, unless all of this is an entirely academic exercise. > > > |>Requirements for a production network: > |>1) Nodes run only software versions proved stable > | > | How do you enforce this? You can't. > > Enforce it? No, probably not. But you can take technical measures that > at minimum stimulate it and, more importantly, have the version endorsed > by the overall project. Sure, you can always have people modifying a > node with bad intent, but that's a different discussion, relevant > whichever path you choose. And the bad guys may be easier to spot if the > rest of the network is more well behaved. I will not endorse a "production network" if it means the development network is so small and so specialized that it can't realistically simulate routing. > > |>2) The number of different versions in the network is minimal > | > | Ditto. > > Same story. Endorsed versions only plus a few bad guys probably. > > > |>3) Updates (minor changes) are limited to security & anonymity fixes > |>4) Upgrades (major changes) are rare (say bi-annual) and done to only to > |>software versions proved stable > |> > |>Note that 3) implies a production fork. Pretty usual. > | > | And all its associated costs. > > These costs can be modest if the project team has the guts to update for > real security and anonymity fixes only. And my guess is that it won't be > hard to find a volunteer to back port one of those fixes to the > production branch. Upgrades would come from development, putting the two > back in sync. So you want to halt all development effort apart from a few bugfixes, on the grounds that your 10 node test network works fine? LOL > > > |>1) Development and extensive testing are done on a separate test > |>network; smoke tests are part of this > | > | Local test networks are useful for debugging a certain class of bugs. We > | do use them from time to time. > > Good. But what I had in mind was to in addition test on a large > non-local network with more of the real-world scale and complexity. Like > current Freenet, but then without bugging people that simply want to use > it. See more on this below. > > |>3) A limited number of RC1 nodes is introduced (or upgraded from > |>production versions) in the production network for a limited amount of > |>time. Their behavior is observed and analyzed, resulting in bug reports. > | > | How is this different to 99% of the network running stable and 1% > | running unstable? > > To begin with, if I check my 5028s routing table I see 30 5xxx nodes and > ~ 20 6xxx nodes and 14 different versions in total. Tracking back any odd > behavior in this network to a particular issue of a particular version > can be pretty hard. Adding a new version node and then accurately > judging the generic quality of that version can be quite tricky. Such a > environment stimulates software changes based on (educated) guesses, > trial and error, and a lot of hard work. We do need to have the same routing algorithms on stable and unstable. However it is not clear that unstable is yet of sufficient quality to merge. > > Secondly note that the definition of stable and unstable in the proposed > situation are different. The stable network consists of only a few > different and relatively old versions (e.g. today 692) and it has been > observed running stable for quite a while (weeks, months). This is the > baseline for the assessment of any new nodes. > > Unstable nodes are inserted only temporary and only of one version at a > time (RC1, RC2, ...). They blend in, measurements are done, odd behavior > is analyzed, bugs are reported and then they're removed. Now one has a > much better chance of making valid observations and one is forced to > take some time to analyze and prioritize bugs before running off to fix > them. Less trial and error, less alchemy. This is exactly what we have NOW. People are quite free to run either stable or unstable, and the vast majority of users especially new users will run stable - especially if stable actually works, which is currently debatable. The source of the whole problem was that stable wasn't working very well. That caused a large number of users to test unstable, which worked better. But if stable works, most users will run it as it is the default option. > > > |>4) The RC1 nodes are removed from the production network (or downgraded > |>to production versions). > |>5) It is explicitly decided which bugs are going to be fixed to go from > |>RC1 to RC2. No other bug fixes or features are introduced: code freeze. > | > | This is ridiculous. > > Yet a widely used strategy to get a complex piece of software to > converge into something releasable, containing hopefully no major bugs > and probably a lot of documented minor bugs. > > What alternative do you see to get into control? > > |>6) RC2 is tested, i.e. goto 3) > |>7) Once an RC is reached that seems acceptable for production use, one > |>could do a large scale test of it on the production network. And if it > |>runs fine on a large number of nodes for say a month, it could become > |>the official new release. > | > | See above. > > Ditto > > |>This is a fairly basic software development process (and as always there > |>will be some discussion on the line between feature and bug fix). > | > | Freenet is not Microsoft Word. > > I'll take your word for that. > > In some aspects Freenet has higher QA demands. Whereas an instance of MS > Word can largely be tested on its own, Freenet also requires testing of > the interaction between multiple nodes. Hence the requirement for a > somewhat controlled multi-node test environment and test/release > process. This can be done in a Bazaar, Cathedral or a web cafe, but > something needs to be done. Freenet cannot be tested on its own. Therefore, we need a reasonable size test network to test it on. We cannot do this with a development only network with a very small number of nodes, abnormal usage patterns and so on. > > > |>1) Developers move their activities to a separate test network. Current > |>Freenet becomes (stays) production network. > |>- - pro: no impact on the current user base, no need to migrate > |>- - con: no rapid performance improvement for current user base. Unclear > |>how to evolve this into clean, few version production network. Requires > |>developer effort. Test network will be small at first. > | > | Test network will ALWAYS be a lot smaller than the mainline network. > | This will in itself discourage use. Client usage patterns will be > different, > | as will node usage patterns - all the transients and semitranisents > | (perm nodes that don't run long enough to get many requests etc) will > | run on the prodnet. > > Most important: the message is not that testing cannot be done on > production but that it can only be done with a version already > thoroughly tested and in a highly controlled way. So in the end there is > a real life test phase and by then one will be able to much more focus > on tuning rather than going after more basic bugs. > > Yes, the test network will be a lot smaller than the mainline network. > But it can be of sufficient size to do extensive testing to reach an > RC1. Just see how many people are subscribed to the devl list, how many > are rapidly upgrading their nodes. A large number of these people are > likely willing to run a test node, possibly next to a production node. > Have some of them be transients, import some content from production, > base some regression test on production stats. We will NOT be able to test routing. And if there is a serious routing problem, or a serious problem in several other categories, it will probably only show up when most of the stable network has upgraded - hence breaking the "stable" network. > > When both the test environment and test process have been cleaned up it > will be much easier to check correct behavior and pin point problems. > And then one needs much less rely one large numbers to get a "feeling" > for what's wrong and right. > > Ian and others have been hammering on weeding the alchemy out of the > algorithms and have set some very nice principle of NGR that allow one > to largely optimize it locally. Things like these too help to make > fairly realistic testing possible at a modest scale network. > > Take it to extremes: if Freenet becomes highly popular (and I think it > has great potential for that), will the whole world be the primary > development and test bed? Obviously not, so at some point in time > changes need to be made. Based on user community feedback that point > won't wait for 1.0 and may well have been reached now. > > Don't get me wrong: without the developer efforts there would have been > no Freenet and there won't be any Freenet. So all users owe a lot to the > developers. But we need to take some measures for the two groups not to > get into each other's way and to allow the developers' work to really > flourish - through actual use. I will NOT support a no-developer-nodes network unless ordered to by Ian. Even if he did, unless there was a good reason I might resign over it. > > Regards, > > Menno -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so.
signature.asc
Description: Digital signature
_______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
