On Mon, Sep 01, 2003 at 03:38:14PM +0100, Gordan wrote: > I have noticed something that seems rather unusual. > > The default setting for threads and connections seem to be 128 and 512 > respectively (or thereabout). At those settings, my node is constantly > running a load of 110%+ (on number of threads), and serving about 5-10 times > the average number of requests (e.g. 40,000-80,000 against the network > average of about 8,000). At that rate, the throttled bandwidth doesn't appear > to be used up (set for 40,000 bytes/sec). The consumption is barely around > 2/3 of that (peak), and it frequently drops through the floor, presumably > because my node is under such a heavy load and is rejecting connections. > > OK, I thought, if it is running out of threads, I'll increase the number of > threads. So, I doubled it to 256. The performance degraded horribly (only the > threads setting was changed). It was still overloaded (110%+), but it was > managing to serve fewer requests and the bandwidth use reduced to less than > half of what it was with 128 threads. > > Is there an explanation for such behaviour? It is almost as if the throttling > code starts to be the bottleneck when having to balance so many transfers > simultaneously. > > I am running 6163 (latest unstable with NG routing), and the machine is most > definitely not underpowered (Freenet is not using 100% of CPU time across all > CPUs at any one time, running at nice -n 15). > > Before I try it (it will require re-starting the node and that takes forever), > would it help if I just set the throttle to 0 (i.e. disable it) and use other > external means of shaping traffic? > > Another issue I wanted to bring up is the file system startup time. If my > understanding is correct from browsing through the documentation and some of > the code, the entire file system "map" is kept in memory, and has to be > pre-loaded before the node starts. When the data store reaches several tens > of GB in size and hundreds of thousands of keys, it takes quite a long time > to start up the file systems. Would it not be better to make the file system > startup asynchronous? > > By this, I mean get the networking up first and start accepting connections. > Then either forget about keeping a list of keys in memory all together and > just take the performance hit from a file system look-up for the key > (possibly a better option for very large data stores), or add the key to the > map, if it exists, when the key is requested. Meanwhile, the map can be > constructed in the background.
Very bad idea. We have various schemes for large stores, which will be implemented when they are more common. Delayed construction of the map is not very useful because most lookups are negative, so while it was constructing the map, it would have to go to disk every time. > > Also, I've seen a post recently about possibly working on ways to use pthreads > in a reasonably stable way. Is this any closer to fruition? The performance > gain is considerable before the node just stops responding all together after > a few hours. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so.
pgp00000.pgp
Description: PGP signature
_______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
