Okay folks we just now had a breakthrough and figured out what is going on with this performance issue.
It has to do with downloading large amounts of data. The new 1.8.0c2 downloader wins against the 1.7.1 downloader for short downloads (I've tried a few, mostly less than 10 MB out of this 1.5 GB file), but loses when I download 100 MB of the file. We ran 100MB downloads under a profiler and found evidence of a computation hotspot in the new spans.py code. See #1170 for details. So here's what you should do: 1. Expect a new release candidate of Tahoe-LAFS v1.8.0 in the next few days which fixes this issue. 2. Do your testing and benchmarking on small reads for now. We currently believe that Tahoe-LAFS v1.8.0 performs better than v1.7.1 on immutable file download as long as you are downloading less than about 10 MB. If you see evidence to the contrary, please let us know! By the way, is anyone interested in how Tahoe-LAFS handles lots of small reads? I am. The v1.8 downloader brings Tahoe-LAFS a small step closer to being able to compete with "NoSQL Databases" like Cassandra. That's because the v1.8 downloader makes it possible to have a potentially large immutable file of fixed-length records and read them in "random access" mode. For what it is worth, Kevan's MDMF work (coming in Tahoe-LAFS v1.9) should move it another small step closer by making it possible to efficiently make small modifications to mutable files, too. Regards, Zooko http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170# new-downloader performs badly when downloading a lot of data from a file _______________________________________________ tahoe-dev mailing list [email protected] http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
