-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Toad wrote:
| I would like to point out ONCE AGAIN that we are nowhere near 1.0.
Development may not be, but your users seem to be. Apparently there's quite a bunch of people that judged the quality of the software of 6-12 months ago to be sufficient to start actually using it. Congratulations on a job well done I'd say.
But now the cat's out of the bag and they won't accept it that their Freenet is crippled because of ongoing development efforts. Now we can remind those people that they're simply beta testers and that they have no right to complain if things don't work well for a few months, or we can try to find a way to accommodate them.
Apart from that, being "nowhere near 1.0" after 2004 minus 1999 years of development might call for a redefinition of what's considered 1.0. Can one expect real users to wait 5+ years? Can one determine priorities for development if there's not a large user base showing what the actual strengths and weaknesses of the software are?
I don't think so, unless all of this is an entirely academic exercise.
|>Requirements for a production network: |>1) Nodes run only software versions proved stable | | How do you enforce this? You can't.
Enforce it? No, probably not. But you can take technical measures that at minimum stimulate it and, more importantly, have the version endorsed by the overall project. Sure, you can always have people modifying a node with bad intent, but that's a different discussion, relevant whichever path you choose. And the bad guys may be easier to spot if the rest of the network is more well behaved.
|>2) The number of different versions in the network is minimal | | Ditto.
Same story. Endorsed versions only plus a few bad guys probably.
|>3) Updates (minor changes) are limited to security & anonymity fixes |>4) Upgrades (major changes) are rare (say bi-annual) and done to only to |>software versions proved stable |> |>Note that 3) implies a production fork. Pretty usual. | | And all its associated costs.
These costs can be modest if the project team has the guts to update for real security and anonymity fixes only. And my guess is that it won't be hard to find a volunteer to back port one of those fixes to the production branch. Upgrades would come from development, putting the two back in sync.
|>1) Development and extensive testing are done on a separate test |>network; smoke tests are part of this | | Local test networks are useful for debugging a certain class of bugs. We | do use them from time to time.
Good. But what I had in mind was to in addition test on a large non-local network with more of the real-world scale and complexity. Like current Freenet, but then without bugging people that simply want to use it. See more on this below.
|>3) A limited number of RC1 nodes is introduced (or upgraded from |>production versions) in the production network for a limited amount of |>time. Their behavior is observed and analyzed, resulting in bug reports. | | How is this different to 99% of the network running stable and 1% | running unstable?
To begin with, if I check my 5028s routing table I see 30 5xxx nodes and ~ 20 6xxx nodes and 14 different versions in total. Tracking back any odd behavior in this network to a particular issue of a particular version can be pretty hard. Adding a new version node and then accurately judging the generic quality of that version can be quite tricky. Such a environment stimulates software changes based on (educated) guesses, trial and error, and a lot of hard work.
Secondly note that the definition of stable and unstable in the proposed situation are different. The stable network consists of only a few different and relatively old versions (e.g. today 692) and it has been observed running stable for quite a while (weeks, months). This is the baseline for the assessment of any new nodes.
Unstable nodes are inserted only temporary and only of one version at a time (RC1, RC2, ...). They blend in, measurements are done, odd behavior is analyzed, bugs are reported and then they're removed. Now one has a much better chance of making valid observations and one is forced to take some time to analyze and prioritize bugs before running off to fix them. Less trial and error, less alchemy.
|>4) The RC1 nodes are removed from the production network (or downgraded |>to production versions). |>5) It is explicitly decided which bugs are going to be fixed to go from |>RC1 to RC2. No other bug fixes or features are introduced: code freeze. | | This is ridiculous.
Yet a widely used strategy to get a complex piece of software to converge into something releasable, containing hopefully no major bugs and probably a lot of documented minor bugs.
What alternative do you see to get into control?
|>6) RC2 is tested, i.e. goto 3) |>7) Once an RC is reached that seems acceptable for production use, one |>could do a large scale test of it on the production network. And if it |>runs fine on a large number of nodes for say a month, it could become |>the official new release. | | See above.
Ditto
|>This is a fairly basic software development process (and as always there |>will be some discussion on the line between feature and bug fix). | | Freenet is not Microsoft Word.
I'll take your word for that.
In some aspects Freenet has higher QA demands. Whereas an instance of MS Word can largely be tested on its own, Freenet also requires testing of the interaction between multiple nodes. Hence the requirement for a somewhat controlled multi-node test environment and test/release process. This can be done in a Bazaar, Cathedral or a web cafe, but something needs to be done.
|>1) Developers move their activities to a separate test network. Current |>Freenet becomes (stays) production network. |>- - pro: no impact on the current user base, no need to migrate |>- - con: no rapid performance improvement for current user base. Unclear |>how to evolve this into clean, few version production network. Requires |>developer effort. Test network will be small at first. | | Test network will ALWAYS be a lot smaller than the mainline network. | This will in itself discourage use. Client usage patterns will be different, | as will node usage patterns - all the transients and semitranisents | (perm nodes that don't run long enough to get many requests etc) will | run on the prodnet.
Most important: the message is not that testing cannot be done on production but that it can only be done with a version already thoroughly tested and in a highly controlled way. So in the end there is a real life test phase and by then one will be able to much more focus on tuning rather than going after more basic bugs.
Yes, the test network will be a lot smaller than the mainline network. But it can be of sufficient size to do extensive testing to reach an RC1. Just see how many people are subscribed to the devl list, how many are rapidly upgrading their nodes. A large number of these people are likely willing to run a test node, possibly next to a production node. Have some of them be transients, import some content from production, base some regression test on production stats.
When both the test environment and test process have been cleaned up it will be much easier to check correct behavior and pin point problems. And then one needs much less rely one large numbers to get a "feeling" for what's wrong and right.
Ian and others have been hammering on weeding the alchemy out of the algorithms and have set some very nice principle of NGR that allow one to largely optimize it locally. Things like these too help to make fairly realistic testing possible at a modest scale network.
Take it to extremes: if Freenet becomes highly popular (and I think it has great potential for that), will the whole world be the primary development and test bed? Obviously not, so at some point in time changes need to be made. Based on user community feedback that point won't wait for 1.0 and may well have been reached now.
Don't get me wrong: without the developer efforts there would have been no Freenet and there won't be any Freenet. So all users owe a lot to the developers. But we need to take some measures for the two groups not to get into each other's way and to allow the developers' work to really flourish - through actual use.
Regards,
Menno -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/iyY49m/IbJq4o8sRAtCpAJ9kvIkGhNaTmyc7r80G3fBjz+fOzgCgiE6P 7EX6Rc1I8hgQ6TlKLID9FQg= =Q4qZ -----END PGP SIGNATURE-----
_______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
