Hi, this is the eleventh weekly report on my Summer of Code project 'Provide some metrics in Debile'[1].
(Previous report: http://lists.alioth.debian.org/pipermail/soc-coordination/2014-July/002237.html) coccinelle check ---------------- I met Matthieu Caneill on IRC, and we managed to fix the coccinelle check. We found the source of the problem in the changelogs for coccinelle version 1.0rc20. That version introduced backwards-incompatible changes in the syntax of scripts. Matthieu updated the firehose patches[2], and the coccinelle check now runs fine.[2.5] uniquify performance -------------------- Also with Matthieu (he originally wrote the uniquify function), we looked for a better way to improve its performance. Rather than using the same algorithm and making it faster, I will write a new one with better SQL requests. Some explanation first: a firehose result is actually a tree of results. In addition to the list of errors found in the analysis, there are elements that give the position of the error in a source code file (line/column, function, etc.) and various other information. For some results, this tree can become very large, with numerous elements that are duplicated (for example functions, files, positions in files, etc.). The goal of uniquify is to ensure we only add each element once, thus reducing the size of the database. In order to do that, it calculates a hash of each element (well, the actual calculation is done in another function, idify[3]), then looks up for that hash in the database, and insert the element only if it isn't in the database already. This is currently done in awkward way because SQLAlchemy doesn't allow to create an object only if it doesn't exist already. I'll have to bypass the ORM and do something like this: http://www.the-art-of-web.com/sql/upsert/#section_4 I'm still not sure how to go about that. As far as I currently understand, I will need to write a sql statement for each type of firehose object, though I guess I'll find a nice solution by digging into sqlalchemy's documentation a little further. (And also by asking for help, since I don't do that often enough.) create jobs for new checks / new versions of checks --------------------------------------------------- Once uniquify (and thus debile-incoming) is able to process all firehose objects generated by debile's checks, there is still something to do before we can consider debile production-ready. While I added a command of adding new checks in the beginning of the summer (before that we could only enable checks when we initialized debile-master), this is not enough. Indeed, we want debile to - create jobs for existing packages when a new check is added - create jobs for existing packages when a check has been modified (new release of the analyzer, or improvement in the debile plugin, etc.) This will need some refactoring in debile.master.orm.create_jobs()[4]. I want to be certain I won't break anything, and to that end I want to write unit tests for that function (debile desperately needs more unit tests anyway, imho). I've had some difficulty with setting up the basis of those tests, but I managed to get somewhere and all I need now is to actually test something.[5] (my most recent (not pushed) code uses factory_boy[6] for easier objects creation. It seems really useful and nice to use, but I'm currently having problems integrating it with the sqlalchemy's session correctly.) Thanks for reading, Clément [1] [https://wiki.debian.org/SummerOfCode2014/StudentApplications/ClementSchreiner] [2] [https://github.com/coccinelle/coccinellery/pull/4] [2.5] [http://anonscm.debian.org/cgit/pkg-debile/debile.git/commit/?id=1247a6b38d9b44f1817f5d8c530e35ba207c76a1] [3] [https://github.com/Debian/firewoes/blob/master/firewoes/lib/hash.py#L47] [4] http://anonscm.debian.org/cgit/pkg-debile/debile.git/tree/debile/master/orm.py#n865 [5] [http://anonscm.debian.org/cgit/pkg-debile/debile.git/commit/?h=tests_orm] [6] [https://factoryboy.readthedocs.org/en/latest/] _______________________________________________ Soc-coordination mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/soc-coordination
