On Mon, Mar 8, 2010 at 2:01 PM, Brian Warner <[email protected]> wrote: > I've identified a O(n**2) CPU/string-allocation misbehavior in Foolscap > when receiving large strings. This could explain the memory and CPU > problems in Tahoe storage servers when you're uploading large mutable > files to them (where "large" means segments that are more than about > 10MB, which for the default 3-of-10 means filesizes above 30MB). In > particular, uploading a 100MB mutable file at 3-of-10, leading to 33MB > blocks, appears to take about three minutes to unpack, using 100MB in > the process (on each server).
So this could explain Jody Harris's CPU-usage problem (reported in this thread), since he uses mutable files for his backups. Jody, was the "largish" file that you were uploading mutable? What was its size (order of magnitude)? This would also explain Stott's report of CPU-usage problem when uploading a large mutable file in the initial problem report of #962 > I still don't have an explanation for reports of slowdowns and large > memory consumption for large *immutable* files. What reports? > http://foolscap.lothar.com/trac/ticket/149 has some more details. The > fix for this will be inside Foolscap's receive-side token parser, and > Tahoe won't even notice. Do you think we can try to get this foolscap fix into Ubuntu Lucid? Hm, it looks like this might be related to #383. (See also #327.) Regards, Zooko http://allmydata.org/trac/tahoe-lafs/ticket/327# performance measurement of directories http://allmydata.org/trac/tahoe-lafs/ticket/383# large directories take a long time to modify http://allmydata.org/trac/tahoe-lafs/ticket/962# Performance problems testing on a 45 disk Dual core 3.3Ghz 4G memory Box _______________________________________________ tahoe-dev mailing list [email protected] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
