On Mon, Oct 20, 2003 at 08:34:11PM +0200, Marc Lehmann wrote: > Hi, > > In this mail I want to let you know my general freenet experience so far, > including my experiences when writing a mass download program. It might > or might not be very useful, but sometimes it's useful to know what some > users think. > > I first used freenet in the 0.4 timeframe, thinking of it like a great > idea, but not yet useful. > > Then I left and came back to it in the 619x area (I think my first > unstable was 6192). > > Since I primarily want to use freenet as a bulk data sharing medium, which > is probably the most demanding use. > > I quickly decided that freenet would be ready for that "soon" (that's a > personal opinion) and moved my transient node to a well-connected, but > slightly older (an [EMAIL PROTECTED]/Linux) machine. > > Being written in java and massively using threads was killing that > machine quickly, however. > > I then tried to experiment with kaffe (which was roughly twicw as fast > on startup as the sun j2sdk), but I never could get it running stably, > neither with kaffe snapshots nor with stable freenet releases (only 5028 > tested, which never ever worked for me). I then decided that freenet was > the most unportable project that officially claims to be portable :) But > alas, fred is just a reference implementation, and in progress, too, so I > don't mind either, as the future will bring much good to the situation.
Heh. Freenet cannot be run with free software (except maybe the non-nio branch), because Kaffe, GCJ and so on don't yet implement NIO. > > So I basically took over another machine (a dual P3-600), which proved > almost adequate for running 250 threads (the high number of threads makes > freenet very cache-unfriendly and thus slow). > > Then I fell into the great NGRouting hole that freenet experienced for > some time. At the same time, I wrote my own freenet mass downloader, since > none was available for my platform. > > And frankly, my biggest problem was my node crashing or (worse) simply not > responding after a while. However, _when_ the node was working, I couldn't Interesting. Was there a specific bug that got fixed? Generally crashes would be JVM problems, hangs would be Fred problems... > complain about download speed, having had maybe an average of 100k/s at > any time the node was running. Cool. > > So, while node stability was a problem for me (and people were busy > working on this issue for the whole time), freenet as a whole network > (with regards to speed) never was an issue for me. > > I mean, running an unstable node (most of which are actually more reliable > than 5028) means some outages are expected. Especially since I download > the newest snapshot and restart my node daily. I have tried to make this point occasionally. I repeat it: if you run unstable, most of the time it will work fine, perhaps better than stable. But it may get serious outages too. Major changes are committed to the unstable branch *for testing*. Consequentially upgrading daily is vital. > > So I think NGRouting is a success, and even _was_ a success during the > recent weeks, as it did provide me with data I wanted. > > I hope you get me right, so I repeat it: freenet worked for me in the > recent weeks (or even months), at least to the level I was expecting it to > work, which includes being buggy at times. Interesting. > > And a technical sidenote: my mass downloader assigns a priority to each > request. Initial fetches get the highest priority, splitfile fetches > get a lower one, and blocks are getting downloaded (usually around 200 > at the same time, since yesterday 400 at the same time without the node > crashing) according got their priority (the priorities are not static > either, randomized like "static_priority*rand [0..1["). Woah. > > I experimented quite a bit with HTL's and priority setting. For example, I > initially used a higher priority for non-check-blocks, or blocks near the > beginning of a segment, but decided that this was a particularly bad idea. It is. We download the blocks *and the segments* in random order. > > My current algorithm (which seems (yes, seems!)) to work better than > what I came up with before, is: > > - use higher priorities for files that are near completion. The idea > behind this is that I have around 300-500 splitfiles in my queue, and > finishing a file will reduce the load considerably, both on my program > as well as on freenet, since the non-fetchable remaining blocks are no > longer requested. > > - use higher priorities for higher HTL's. This is just there so that > low-htl-requests don't block the high-htl requests that might be > necessary to finish the last few blocks of a file. > > I also experimented with lower priorities for higher HTLs, but that > didn't work out that well. > > - use the following HTL sequence repeatedly: 0, 15, 0, 20, 0, 25, 0. > Now, that looks rather strange, but the idea (which seems to work) > is that when I request block n at HTL 15, then re-requesting it some > time later at HTL 0 _often_ succeeds. Which is far better than > randomly requesting other blocks. Hmm. That is strange. Perhaps the corruption bug... Also, why do you increase the HTL rather than starting at 25? > > I am not sure (and I slightly doubt it) that the protocol is indeed > responsible for this interetsing freenet behaviour. Maybe it's just that > my receiving conenctions time out (I don't use any timeouts), but today, > after running continiously for 20 hours, I still get new blocks at the > same rate as in the beginning, so this seems not to be the case. > > I could also be mislead because I often had to restart freenet, because > it was not responding, but maybe it _was_ responding on the network > side... But again, I see the same behaviour right now, after 20 hours, > which is my current record. > > Right now, the average successful HTL is 14.023202 (decreasing), and the > median is 15, which makes me think about decreasing the initial HTL to 10 > or lower. It also makes me wonder wether that means that my node either > gets data at htl 15, or not at all even at higher htls. Ooh. Cool. Quite possible. > > All this also means that, when a file finished, I often have more blocks > per segment then necessary to decode, as the blocks trickle in rather > rapidly and in bursts at htl 0 often. That is probably a good thing. It is equivalent to doing a little healing... > > With regards to data corruption: I check every single CHK content I > receive against the CHK hash, and about one in 600 blocks is corrupted. > > When re-fetching the same block, it usually succeeds. I checked my client > iver and over and even looked at the received data, and it does not seem > to be premature connection closing or short data related, as my client > detects this reliable (modulo bugs of course). Also, the corrupt data > often "looks" like real data, somehow scrambled (in an mpg for example). Is it related to restarts after some data has been transferred? > > This is not a serious problem for me either, I haven't lost a file > because of that yet. and the only serious showstopper (fcp on non-local > connections) has been fixed rather quickly. I thought it took ages to fix that one :) > > All in all, it works very well. > > I hope this was useful or at leats interesting for you to read :) VERY. Thank you. > > Some data about my node that might or might not be interesting: > > Histogram of node versions in fred's Routing table > > Oct 19, 2003 5:48:22 PM > nodes: 50 > scale factor: 1.0 (This is used to keep lines < 64 characters) > > Fred,0.5,1.46,5015 |=== > Fred,0.5,1.46,5017 |= > Fred,0.5,1.46,5023 |= > Fred,0.5,1.46,5028 |========================= > Fred,0.5,1.46,598 |== > Fred,0.6,1.46,6214 |= > Fred,0.6,1.46,6235 |= > Fred,0.6,1.46,6245 |= > Fred,0.6,1.46,6248 |=== > Fred,0.6,1.46,6251 |== > Fred,0.6,1.46,6254 |= > Fred,0.6,1.46,6255 |== > Fred,0.6,1.46,6257 |======= > > Histogram of keys in in fred's data store [so my node isn't specialized > yet, due to a 75% full datastore only] Perhaps... > These are the keys to the data in your node's local cache (DataStore) > Oct 19, 2003 5:47:08 PM > keys: 48931 > scale factor: 0.020388659089803696 (This is used to keep lines < 64 characters) > > 0 |=============================================================== > 1 |============================================================== > 2 |================================================================ > 3 |============================================================== > 4 |============================================================== > 5 |============================================================== > 6 |============================================================ > 7 |=============================================================== > 8 |============================================================= > 9 |============================================================== > a |============================================================= > b |============================================================ > c |=============================================================== > d |=============================================================== > e |============================================================= > f |============================================================== > > Probability of success of an incoming request > Oct 19, 2003 5:47:58 PM > 0 | 0.050851833 > 1 | 0.044327572 > 2 | 0.05563835 > 3 | 0.04980033 > 4 | 0.048084054 > 5 | 0.04565902 > 6 | 0.049584758 > 7 | 0.04659147 > 8 | 0.046026126 > 9 | 0.04020678 > a | 0.03921804 > b | 0.039 > c | 0.03987525 > d | 0.046337817 > e | 0.044974815 > f | 0.043094557 Woah. What build number? > > Max: 0.05563835 > Most successful: 2 > > Key size histogram: > > keys: 48927 > scale factor: 0.0034797738771885633 (This is used to keep lines < 64 characters) > > 1k |== > 2k |====================================== > 4k |==== > 8k |== > 16k |=== > 32k |= > 64k | > 128k |== > 256k |================================================================ > 512k |=============== > 1M |==================================== > 2M | > 4M | > 8M | > 16M | > 32M | > 64M | > 128M | > 256M | > 512M | > 1G+| > > > > -- > -----==- | > ----==-- _ | > ---==---(_)__ __ ____ __ Marc Lehmann +-- > --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e| > -=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+ > The choice of a GNU generation | -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so.
signature.asc
Description: Digital signature
_______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
