Hello Zooko, > It is cool that the Tahoe-LAFS storage server works at all on that > tiny NAS device! Does it work okay? Is your desire to reduce the > memory footprint to eliminate some failures, or to improve > performance, or to allow concurrent use of other processes on the same > device, or to move it to an even smaller device, or something else?
I think it's cool as well. What I like especially is the idea that people can put such an inexpensive box as personal server to share information and make themselves more independent from data-gathering companies. It would also be quite easy to provide a ready-to use image or software package. For me, the server worked very well in the last eight days. Today however, we've seen on VG2 huge latencies of about 6 seconds, which occured when I did a backup upload at the same time. I ran also an NFS daemon to upload data and maybe that was sometimes becoming too much. Neither NFS nor tahoe-backup uses much RAM but I suspect that this leads occasionally to a kind of memory gridlock in kernel space; NFS can sometimes stall for many seconds. So, I think the reliability and speed could be increased when reducing RAM further. > > The storage-server side of Tahoe doesn't really do that much, so > > I'd be pleased as punch if it could fit into 10 or 20MB. To do > > that, I think we'd need to avoid loading all of the code when we're > > not running a client > > Wouldn't the code (.py's, .pyc's, or .so's) tend to be swapped out if > it unused? Or maybe unused code gets interspersed with code that is > needed, so it ends up bloating the memory usage? Memory is managed by the OS in 4k size pages. Entirely unused pages are simply swapped out. A reduction in the resident set size, in the amount of file buffering required or a grouping of often used code / data could probably achieve some speed-up. The simplest measure could be to import modules in a grouped order, with server and client functions in different groups. How difficult would it be to rip out a server-only program? This could run with increased priority. Is there a fundamental reason that both processes run in the same interpreter instance? I've experimented some time with real-time audio processing in Python, and a low-hanging fruit for multiprocessor systems is to separate different processes which are not serialized by Python's GIL (Global Interpreter Lock). Of course, my NAS has only one core but most future desktop and server systems will have much more than one. Regards, Johannes Am Sun, 4 Mar 2012 21:33:08 -0700 schrieb "Zooko Wilcox-O'Hearn" <[email protected]>: > Dear Johannes Nix: > > It is cool that the Tahoe-LAFS storage server works at all on that > tiny NAS device! Does it work okay? Is your desire to reduce the > memory footprint to eliminate some failures, or to improve > performance, or to allow concurrent use of other processes on the same > device, or to move it to an even smaller device, or something else? > > The numbers you give sound about right. I measured resident RAM usage > on 32-bit x86 servers a couple of times, and found that it started at > around 32 MB and the high water mark of resident RAM usage was about > 56 MB. > > I don't know of any technique that could further reduce it, aside from > profiling the memory usage and reporting back to this list. If your > profiling turned up some "low hanging fruit" of wasted memory, then > perhaps some Tahoe-LAFS developer would be motivated to try reducing > that. > > Regards, > > Zooko > _______________________________________________ > tahoe-dev mailing list > [email protected] > http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev _______________________________________________ tahoe-dev mailing list [email protected] http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
