Hi,
In this mail I want to let you know my general freenet experience so far,
including my experiences when writing a mass download program. It might
or might not be very useful, but sometimes it's useful to know what some
users think.
I first used freenet in the 0.4 timeframe, thinking of it like a great
idea, but not yet useful.
Then I left and came back to it in the 619x area (I think my first
unstable was 6192).
Since I primarily want to use freenet as a bulk data sharing medium, which
is probably the most demanding use.
I quickly decided that freenet would be ready for that "soon" (that's a
personal opinion) and moved my transient node to a well-connected, but
slightly older (an [EMAIL PROTECTED]/Linux) machine.
Being written in java and massively using threads was killing that
machine quickly, however.
I then tried to experiment with kaffe (which was roughly twicw as fast
on startup as the sun j2sdk), but I never could get it running stably,
neither with kaffe snapshots nor with stable freenet releases (only 5028
tested, which never ever worked for me). I then decided that freenet was
the most unportable project that officially claims to be portable :) But
alas, fred is just a reference implementation, and in progress, too, so I
don't mind either, as the future will bring much good to the situation.
So I basically took over another machine (a dual P3-600), which proved
almost adequate for running 250 threads (the high number of threads makes
freenet very cache-unfriendly and thus slow).
Then I fell into the great NGRouting hole that freenet experienced for
some time. At the same time, I wrote my own freenet mass downloader, since
none was available for my platform.
And frankly, my biggest problem was my node crashing or (worse) simply not
responding after a while. However, _when_ the node was working, I couldn't
complain about download speed, having had maybe an average of 100k/s at
any time the node was running.
So, while node stability was a problem for me (and people were busy
working on this issue for the whole time), freenet as a whole network
(with regards to speed) never was an issue for me.
I mean, running an unstable node (most of which are actually more reliable
than 5028) means some outages are expected. Especially since I download
the newest snapshot and restart my node daily.
So I think NGRouting is a success, and even _was_ a success during the
recent weeks, as it did provide me with data I wanted.
I hope you get me right, so I repeat it: freenet worked for me in the
recent weeks (or even months), at least to the level I was expecting it to
work, which includes being buggy at times.
And a technical sidenote: my mass downloader assigns a priority to each
request. Initial fetches get the highest priority, splitfile fetches
get a lower one, and blocks are getting downloaded (usually around 200
at the same time, since yesterday 400 at the same time without the node
crashing) according got their priority (the priorities are not static
either, randomized like "static_priority*rand [0..1[").
I experimented quite a bit with HTL's and priority setting. For example, I
initially used a higher priority for non-check-blocks, or blocks near the
beginning of a segment, but decided that this was a particularly bad idea.
My current algorithm (which seems (yes, seems!)) to work better than
what I came up with before, is:
- use higher priorities for files that are near completion. The idea
behind this is that I have around 300-500 splitfiles in my queue, and
finishing a file will reduce the load considerably, both on my program
as well as on freenet, since the non-fetchable remaining blocks are no
longer requested.
- use higher priorities for higher HTL's. This is just there so that
low-htl-requests don't block the high-htl requests that might be
necessary to finish the last few blocks of a file.
I also experimented with lower priorities for higher HTLs, but that
didn't work out that well.
- use the following HTL sequence repeatedly: 0, 15, 0, 20, 0, 25, 0.
Now, that looks rather strange, but the idea (which seems to work)
is that when I request block n at HTL 15, then re-requesting it some
time later at HTL 0 _often_ succeeds. Which is far better than
randomly requesting other blocks.
I am not sure (and I slightly doubt it) that the protocol is indeed
responsible for this interetsing freenet behaviour. Maybe it's just that
my receiving conenctions time out (I don't use any timeouts), but today,
after running continiously for 20 hours, I still get new blocks at the
same rate as in the beginning, so this seems not to be the case.
I could also be mislead because I often had to restart freenet, because
it was not responding, but maybe it _was_ responding on the network
side... But again, I see the same behaviour right now, after 20 hours,
which is my current record.
Right now, the average successful HTL is 14.023202 (decreasing), and the
median is 15, which makes me think about decreasing the initial HTL to 10
or lower. It also makes me wonder wether that means that my node either
gets data at htl 15, or not at all even at higher htls.
All this also means that, when a file finished, I often have more blocks
per segment then necessary to decode, as the blocks trickle in rather
rapidly and in bursts at htl 0 often.
With regards to data corruption: I check every single CHK content I
receive against the CHK hash, and about one in 600 blocks is corrupted.
When re-fetching the same block, it usually succeeds. I checked my client
iver and over and even looked at the received data, and it does not seem
to be premature connection closing or short data related, as my client
detects this reliable (modulo bugs of course). Also, the corrupt data
often "looks" like real data, somehow scrambled (in an mpg for example).
This is not a serious problem for me either, I haven't lost a file
because of that yet. and the only serious showstopper (fcp on non-local
connections) has been fixed rather quickly.
All in all, it works very well.
I hope this was useful or at leats interesting for you to read :)
Some data about my node that might or might not be interesting:
Histogram of node versions in fred's Routing table
Oct 19, 2003 5:48:22 PM
nodes: 50
scale factor: 1.0 (This is used to keep lines < 64 characters)
Fred,0.5,1.46,5015 |===
Fred,0.5,1.46,5017 |=
Fred,0.5,1.46,5023 |=
Fred,0.5,1.46,5028 |=========================
Fred,0.5,1.46,598 |==
Fred,0.6,1.46,6214 |=
Fred,0.6,1.46,6235 |=
Fred,0.6,1.46,6245 |=
Fred,0.6,1.46,6248 |===
Fred,0.6,1.46,6251 |==
Fred,0.6,1.46,6254 |=
Fred,0.6,1.46,6255 |==
Fred,0.6,1.46,6257 |=======
Histogram of keys in in fred's data store [so my node isn't specialized
yet, due to a 75% full datastore only]
These are the keys to the data in your node's local cache (DataStore)
Oct 19, 2003 5:47:08 PM
keys: 48931
scale factor: 0.020388659089803696 (This is used to keep lines < 64 characters)
0 |===============================================================
1 |==============================================================
2 |================================================================
3 |==============================================================
4 |==============================================================
5 |==============================================================
6 |============================================================
7 |===============================================================
8 |=============================================================
9 |==============================================================
a |=============================================================
b |============================================================
c |===============================================================
d |===============================================================
e |=============================================================
f |==============================================================
Probability of success of an incoming request
Oct 19, 2003 5:47:58 PM
0 | 0.050851833
1 | 0.044327572
2 | 0.05563835
3 | 0.04980033
4 | 0.048084054
5 | 0.04565902
6 | 0.049584758
7 | 0.04659147
8 | 0.046026126
9 | 0.04020678
a | 0.03921804
b | 0.039
c | 0.03987525
d | 0.046337817
e | 0.044974815
f | 0.043094557
Max: 0.05563835
Most successful: 2
Key size histogram:
keys: 48927
scale factor: 0.0034797738771885633 (This is used to keep lines < 64 characters)
1k |==
2k |======================================
4k |====
8k |==
16k |===
32k |=
64k |
128k |==
256k |================================================================
512k |===============
1M |====================================
2M |
4M |
8M |
16M |
32M |
64M |
128M |
256M |
512M |
1G+|
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
_______________________________________________
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
