Some of my earlier responses to Pierre's message seem to have been lost in the ether. Here they are again:
On Tue, May 10, 2011 at 01:51:22PM -0400, W. Trevor King wrote: > On Tue, May 10, 2011 at 12:45:14AM +0200, Pierre THIERRY wrote: > > I have seen that in 2010, someone here ran a few performance tests on > > BE, that showed this performance drop at most at a few thousands bugs. > > My own tests was to create empty bugs with just a message and to list > > them (discarding the output to /dev/null) and it showed the following > > results: > > > > number of commits time > > 0 baseline > > 128 1.2x slower > > 512 4x slower > > 1024 17x slower > > > > On my system, the baseline was actually 400ms, which is barely > > noticeable, but at 512 bugs, it already becomes 1.6s, which easily seems > > like a small pause to the user. > > By "number of commits" I expect you mean "number of empty bugs". I > expect BE should scale well vs number of commits. I just ran some > tests (benchmark-be-list.sh attached) on my own system (ext4 > filesystem), and got something like > <`be list` time> = 470 ms + 8.3 ms * N + 2.6 us * N**2 > where N is the number of bugs. > > Which is a bit slower than your times, but qualitatively similar. > I'll run the tests on my btrfs system when I get home tonight. > > Note that this is only a problem for the current file-based storage > backends, and it is only a major problem (as far as I know) for `be > list`, because it has to load all the bugs to decide which ones should > be displayed. > > > I didn't see this on the planned features, has there already been any > > thoughts on this issue? > > As far as I know the "Planned features" section of > http://bugseverywhere.org/ is pretty outdated. I think about this > periodically, but haven't come up with an elegant solution. > > Possible ideas: > > 1) Use a better file system. There's no fundamental difference > between a file system and a database like MySQL. With a properly > tuned filesystem geared towards many small files, BE's current > file-based backends should perform acceptably. > 2) Cache "useful" index information to on-disk files. In the case of > `be list`, that would mean status, severity, assigned, and > extra_strings for each bug. The problem with a cache are: > a) Keeping it up to date despite possible concurrent `be` commands > (via file locking, etc.) > b) Unless you version the cache file, it doesn't accelerate access > to previous commits. > 3) Use `be serve` to cache in memory. As long as you avoid other > direct repository manipulation, this should work well until you run > out of memory ;). > > Ditz dealt with this issue by "archiving" bugs in directories that > were not searched by the standard list command, but I'm not > comfortable with that. > > Anyhow, I'll report back with details on caching with `be serve` and > running on btrfs once I have time to run those tests. On Tue, May 10, 2011 at 02:29:27PM -0400, W. Trevor King wrote: > On Tue, May 10, 2011 at 01:51:22PM -0400, W. Trevor King wrote: > > (benchmark-be-list.sh attached) > > Oops. Forgot to attach the script. Here it is. > > > <`be list` time> = 470 ms + 8.3 ms * N + 2.6 us * N**2 > > After the 4096 bugs, the best fit was > > <`be list` time> = 470 ms + 7.6 ms * N + 3.0 us * N**2 > > Data attached. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
benchmark-be-list.sh
Description: Bourne shell script
1 0.47 2 0.49 4 0.49 8 0.52 16 0.58 32 0.70 64 0.97 128 1.56 256 2.79 512 5.48 1024 11.72 2048 28.47 4096 82.66
pgpcYiVTuzg1b.pgp
Description: PGP signature
_______________________________________________ Be-devel mailing list [email protected] http://void.printf.net/cgi-bin/mailman/listinfo/be-devel
