Hi, this is third first weekly report on my Summer of Code project 'Provide some metrics in Debile'[1][2].
clang build ----------- I have finally succeeded in making debile build packages with clang. The bug in my code that I was stuck on last week was a stupid typo, but then I had another problem. In order to make sbuild use clang instead of gcc/g++, we use a (hideous) hack[3], courtesy of Sylvestre: with a chroot hook, the gcc and g++ binaries are replaced by clang's before the build starts. However, sbuild has not been able to run chroot hooks as root for a few years: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840. Fortunately, someone has posted a patch[3] on that thread allowing just that, which I applied[4]. That taught me how to use quilt for adding patches to a debian package. clanganalyzer/scan-build ------------------------ This is still work in progress. I ported the existing debile plugin (developed by Léo Cavaillé during his GSoC last year) to the current debile-slave API. It took me some time to realize the plugin was using an option not available in sbuild vanilla, and I had to patch it again, this time using Léo's[5]. At the moment, while clanganalyzer jobs seem to run correctly, they don't actually use scan-build, for some reason. I have to look into that. debile-incoming (huge) memory consumption and bad performances -------------------------------------------------------------- Yesterday evening, tried to login on ursae (the serveur hosting debile-master and http://debile.debian.net/), and the ssh connection would timeout, or I would get a "broken pipe" error. I suspected the swap was full, which was confirmed when I finally got a prompt. A few hours later, I managed to login again and type pkill debile, and the system was responsive again. The culprit was debile-incoming, the script that scans debile-master's upload queue for importing new packages or job results into the database. A cron was executing debile-incoming every minute, which did not terminate in that time: dozens of processes were running at the same time. Worse: for some reason, it uses about 2GiB of ram (or even 5GiB in some cases), and now takes at least a hour to finish its work. I run a profiler on debile-incoming[6], which confirmed my suspicion that it was because of firewoes[7]' analysis of firehose XML files. Nicolas Dandrimont gave me some hints to reduce python's memory use[8], but I'll also need to improve the algorithm, since firewoes seems unable to process in a reasonable time the huge files that some checks are generating. loofah/ricky ------------ In order to test debile in a more realistic scale, I used loofah[9] to send dozens of packages to debile. This made me aware of debile-incoming's ram/performance problems above. That's it for this week. Thanks for reading. --------- Clément [1] [https://wiki.debian.org/SummerOfCode2014/StudentApplications/ClementSchreiner] [2] [http://lists.alioth.debian.org/pipermail/soc-coordination/2014-May/002037.html] [3] [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840#27] [4] [http://www.mux.me/debile/] [5] [http://leo.cavaille.net/public/gsoc/sbuild-patches/0001-Added-support-for-schroot-sessions.patch] [6] [http://paste.debian.net/103741/] [7] [https://github.com/Debian/firewoes/] [8] [http://guillaume.segu.in/blog/code/487/optimizing-memory-usage-in-python-a-case-study/] [9] [https://github.com/sylvestre/loofah] _______________________________________________ Soc-coordination mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/soc-coordination
