Hello Bastien, We can use travis-ci for automated tests, I just run tests for org-mode¹ on travis with emacs-snapshot. Magit has been setup recently to run tests for multiple emacs versions² (emacs23, emacs24 and snapshot). travis has a facility to send mails if a test fails.
I see you have org-mode repo on your github account, all we need to do put a post recieve hook to mirror org-mode repo to github. we can adopt magit script³ to org-mode as it won't be synced to emacs trunk. what do you think? Thanks., ¹ https://travis-ci.org/yyr/org-mode/builds/5692183 ² https://travis-ci.org/magit/magit ³ https://github.com/magit/magit/blob/master/.travis.yml -- ఎందరో మహానుభావులు అందరికి వందనములు. YYR