I've been working recently on an automated performance testing framework for GNOME Shell. You can see the GNOME Shell part of it at:
https://bugzilla.gnome.org/show_bug.cgi?id=618189 The basic idea of an automated performance test: * You provide a Javascript script that drives the shell * While the script is running, timings and statistics are recorded to an event log. * You provide additional Javsascript code that computes "metrics" based upon replaying the event log. Metrics includes things like "How much memory does the shell use after startup". "What frame rate is observed when animating zooming out to the activities overview". At the end of the performance test, a report is written out to a file. The next step after that is to upload it to a server where we can compare across different systems and on individual systems over time. I've begun some work on writing a server to collect and display these reports with the idea to deploy that on gnome.org. It's not in a functional state yet, but wanted to throw some questions/ideas out there for discussion early: * What I've written so far is in Django. I picked that: - Because we're using that a bit on GNOME already (Mango rewrite) - Because it seemed simple and self-contained - Because compared to some over-engineered clever framework I might come up with myself, it seems likely someone else could pick it up and continue the maintenance. My hope that it would be easy to get something going seems to be working out. I'm not loving either the templating language or the ORM of Django ... I think I'd find them pretty limiting and annoying on a big project. But they should be OK for this. * I'm uncertain where it would live in the GNOME web hierarchy. Is it: http://perf.gnome.org/shell (Hope to add similar things for other parts of GNOME) http://shell.gnome.org/perf (Hope to add things like a Shell extension repository site) http://shell-perf.gnome.org (Just make it a standalone vhost) http://people.gnome.org/otaylor/shell-perf/ One advantage of not using a separate vhost for individual small apps like like this is being able to HTTPS-secure them without having to get separate certificates. (I'm not even sure if it's possible to have multiple HTTPS sites on the same IP - there used to be problems with HTTPS-based vhosts, but maybe that has been resolved?) * Which system does it live on? It obviously is conceptually similar to the build master function of fixed.gnome.org, but I don't think there is any synergy, and in general I'd rather not set up anything new on physical machines. Would this go on wiki.gnome.org as our "dynamic python hosting" VM? * Disk usage for this is medium. Currently the performance reports are 4k each, but they'll get bigger as we record more data. Rough estimate for an upper bound: 100k/report * 10 systems * 1 report/day/system * 365 days/year * 2 years of data retained Or about 1GB. (The reason that the reports are big is that they include the event log - not completely clear if that will be useful, but being able to chart memory usage over the course of a run would potentially be pretty interesting in figuring out why we suddenly started using more memory.) Anyways, just wanted to give an initial heads up, and get any immediate feedback people have. Once I get a little further, I'll put it up for test purposes on my server so people can take a look. - Owen _______________________________________________ gnome-infrastructure mailing list [email protected] http://mail.gnome.org/mailman/listinfo/gnome-infrastructure
