I have noticed something that seems rather unusual. The default setting for threads and connections seem to be 128 and 512 respectively (or thereabout). At those settings, my node is constantly running a load of 110%+ (on number of threads), and serving about 5-10 times the average number of requests (e.g. 40,000-80,000 against the network average of about 8,000). At that rate, the throttled bandwidth doesn't appear to be used up (set for 40,000 bytes/sec). The consumption is barely around 2/3 of that (peak), and it frequently drops through the floor, presumably because my node is under such a heavy load and is rejecting connections.
OK, I thought, if it is running out of threads, I'll increase the number of threads. So, I doubled it to 256. The performance degraded horribly (only the threads setting was changed). It was still overloaded (110%+), but it was managing to serve fewer requests and the bandwidth use reduced to less than half of what it was with 128 threads. Is there an explanation for such behaviour? It is almost as if the throttling code starts to be the bottleneck when having to balance so many transfers simultaneously. I am running 6163 (latest unstable with NG routing), and the machine is most definitely not underpowered (Freenet is not using 100% of CPU time across all CPUs at any one time, running at nice -n 15). Before I try it (it will require re-starting the node and that takes forever), would it help if I just set the throttle to 0 (i.e. disable it) and use other external means of shaping traffic? Another issue I wanted to bring up is the file system startup time. If my understanding is correct from browsing through the documentation and some of the code, the entire file system "map" is kept in memory, and has to be pre-loaded before the node starts. When the data store reaches several tens of GB in size and hundreds of thousands of keys, it takes quite a long time to start up the file systems. Would it not be better to make the file system startup asynchronous? By this, I mean get the networking up first and start accepting connections. Then either forget about keeping a list of keys in memory all together and just take the performance hit from a file system look-up for the key (possibly a better option for very large data stores), or add the key to the map, if it exists, when the key is requested. Meanwhile, the map can be constructed in the background. Also, I've seen a post recently about possibly working on ways to use pthreads in a reasonably stable way. Is this any closer to fruition? The performance gain is considerable before the node just stops responding all together after a few hours. Gordan _______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
