On Monday 09 November 2009 14:39:57 Evan Daniel wrote: > On Mon, Nov 9, 2009 at 8:35 AM, Matthew Toseland > <[email protected]> wrote: > > Some Chinese feedback: Main point is there are a lot of old PCs with very > > limited RAM, although new ones have plenty of RAM. Broadband is however > > reasonable in general, making it a better candidate than Iran. Suggested > > making startup on startup configurable, substantially reducing memory > > footprint etc. My worry with the former is that opennet won't last forever, > > and low-uptime darknet is very problematic. > > > > IMHO we could reasonably cut our default memory limit to 128M. The main > > issues are: > > - Sorting out Library memory issues (bug #3685) > > - Sorting out Freetalk memory issues (I haven't seen OOMs from Freetalk > > recently, maybe this really is fixed?) > > - Sorting out - or ignoring - the startup memory spike on big inserts (4G > > needs 192MB). > > > > On fast systems, a higher memory limit means less CPU spent in garbage > > collection, so maybe there is an argument for reinstating the panel in the > > wizard that asks the user how much memory to allocate... > > > > Our friend has also localised the wininstaller (this is subject to > > technical issues Zero3 hopefully will be able to resolve), and jSite (I > > will deal with this soon). > > It is my understanding (haven't personally tested) that we could > reduce the memory footprint by reducing the per-thread stack size > (-Xss option).
The problem with stack size is we don't know what an appropriate limit is, it probably depends on 32 vs 64 if not on other factors, and if we get it wrong fred crashes. > Similarly, reducing the max thread count might help > (we don't usually use all the threads, but worst-case behavior is what > causes OOMs). Maybe this could depend on the memory limit? In general it looks to me like we do currently need the 500 limit, however for resource constrained systems reducing it might help. In future other things might help e.g. rewriting the data transfer code to be simpler and threadless (we have a reliable transport layer, and we have central per-peer scheduling of packets for throttling, so it should be possible to get rid of most of BlockTransmitter etc). > > Should we base the memory for RAM buckets on the configured maximum, > instead of having a constant default? These are used for caching > decoded data, right? So more buckets means that Library gets far more > responsive on repeat searches. (Just tested this -- increasing RAM > buckets to 80 MiB does seem to mean that Library skips directly to the > "Parsing subindex" step on repeat or partially repeat searches. Of > course, it's still glacially slow after that.) Yes. https://bugs.freenetproject.org/view.php?id=3696 And in my testing, recent changes to Library have cut the parsing time from over 5 minutes o under 40 seconds, almost entirely encrypting and decrypting. With a high buckets limit, it ought to be possible to avoid most of this. > > For inserts, we could also estimate how much RAM will be required, and > warn the user instead of trying to start an insert that is likely to > OOM. Perhaps. The thing that will cause it to OOM is the peak memory usage, which occurs when starting the splitfile inserter, it is roughly linear with the number of blocks. It would be possible to eliminate it, but it is a low priority. More broadly, more memory helps a lot: - More temp files in RAM rather than encrypted on disk -> less disk, less CPU. - Garbage collection less often. - More bloom filters for sharing later on. - More parallel compression/decompression threads, if they are not disk-limited (which e.g. decompressing from a memory temp file to an XML parser would not be!). I had initially decided to reinstate the RAM usage dropdown (bug #3695) for these reasons - some computers will have very little RAM, some will have loads. However, it turns out we can autodetect it from the system memory fairly easily, at least in Windows. The systeminfo command initially seemed a good candidate, but its output is localised; the best option is for the wininstaller to tell us what it is, write it to some file. On Linux, /proc/meminfo is the obvious solution, although there may be a more cross-platform option; on OS/X, I have no idea, but maybe mrsteveman1 can tell us. https://bugs.freenetproject.org/view.php?id=3697 Zero3, any chance of writing the total system memory to a file in the wininstaller in the near future? If we have working autodetection, we should only show the memory drop-down if the user really needs to see it, e.g. if they have very little memory and the best options may limit their ability to use Freenet stably: 64MB is enough for most pure nodes, but if you use the client layer much you'll hit a brick wall fast. Also, if the user has asked for advanced mode - if we implement such a thing in the first-time wizard - we could do this.
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Devl mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
