Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On Fri, Mar 08, 2013 at 02:52:48PM +0100, Thomas Koch wrote: Hi Daniel et al, I'm also thinking a lot about how to improve Debian by improving our Git tooling. Therefor I'm packaging Gerrit (#589436). But gerrit and its dependencies is a big project... Now that Git slowly becomes the de facto standard VCS for Debian[1] (resistance is futile) it might be time to review our setup and think whether we could improve our Git infrastructure. Should we start a wiki page to collect thoughts? [1] http://www.lucas-nussbaum.net/blog/?p=751 My thoughts are: - I'd like to have support for reviews (e.g Gerrit) - pull requests (e.g Gerrit) - I'd like continuous integration (triggered e.g. by Gerrit[2]) Gerrit's Jenkins integration is awesome. Verifying if a package still builds and runs its autopkgtests after a commit would be a huge step forward. Is that what you're after? Do you run a test system somewhere for that already? Should we start setting something like this up? Cheers, -- Guido - Easy for anybody to submit patches (e.g Gerrit) - A frontpage that doesn't take ages to load - Easier project creation without the need to SSH into alioth - regular fetching of the upstream branch from upstreams master [2] http://openstack-ci.github.com/publications/ I was also thinking whether Debian should cooperate with other projects so that the workload of maintaining such a setup could be shared. I started to collect candidates for collaboration here: http://wiki.debian.org/Alioth/OtherForges Best regards, Thomas Koch, http://www.koch.ro -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201303081452.50647.tho...@koch.ro -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130322204421.ga12...@bogon.sigxcpu.org
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 2013-03-22 21:44:21 +0100 (+0100), Guido Günther wrote:
Gerrit's Jenkins integration is awesome.
[...]
OpenStack CI has some additional tools which help avoid the need to
interact directly with Jenkins too much. There's Zuul (the
gatekeeper) which watches the Gerrit event stream and triggers jobs
in Jenkins as a result of matching again patterns defined a YAML
configuration file--the gerrit-trigger plugin for Jenkins lacked
enough AI for our needs. Also Jenkins Job Builder which allows you
to keep your Jenkins jobs in a templated YAML format rather than
resorting to its WebUI or editing XML configs. And we've also got a
Gearman plug-in in the works for Jenkins, so that Gearman queues can
be used to gain finer-grained control over Jenkins jobs and slaves.
https://github.com/openstack-infra/zuul
https://github.com/openstack-infra/jenkins-job-builder
https://github.com/openstack-infra/gearman-plugin
We're always happy to see others putting this stuff to use if it
suits their needs, and welcome outside contributions as well.
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130322210818.ge29...@yuggoth.org
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 2013-03-22 21:08:18 + (+), Jeremy Stanley wrote:
[...]
watches the Gerrit event stream and triggers jobs in Jenkins as a
result of matching again patterns defined a YAML configuration
file
[...]
Yeesh. I clearly shouldn't write E-mail when I'm rushing off to eat.
What I meant to say is that Zuul triggers jobs in Jenkins as a
result of matching Gerrit events against patterns defined in a YAML
file, and also uses a predictive pipeline heuristic to merge and
oversee parallel tests on sequences of patches. We enqueue approved
changes from Gerrit (some dependent on one another, others
independent of each other), and ensure that they only make it into
the target branch if they pass a battery of unit and integration
tests when merged on the patches ahead of them in the pipeline.
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130322234722.gf29...@yuggoth.org
Re: RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
]] Thomas Goirand Did anyone try buildbot? It might be better for what I need. Buildbot is pretty crap at managing slaves that disappear and come back and such. I quite disliked the fact that most of Jenkins is done through a web GUI, which was in fact, more a nuisance than anything else. Maybe buildbot would fit my needs better, so I would really appreciate if anyone can share his experience with it. Just use jenkins-job-builder? -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87d2uyunzx@qurzaw.varnish-software.com
Re: RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
Hi Tollef, Tollef Fog Heen tfh...@err.no writes: Buildbot is pretty crap at managing slaves that disappear and come back and such. This works fine for me, I have never had any trouble with that (and yes, my build slaves have disconnected/reconnected quite a few times). Using buildbot since more than a year to do after-push builds of Debian/Ubuntu packages for i3wm.org, see http://i3wm.org/docs/buildbot.html I am pretty happy with buildbot. -- Best regards, Michael -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/x6k3p5bz3j@midna.zekjur.net
Re: RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 03/09/2013 12:36 AM, PICCA Frédéric-Emmanuel wrote: I start to really love the CI thing. I first invested a bit of time in setting-up everything, do you have a step by step cookbook for your setup. Maybe on the debian wiki ? Unfortunately, no. But it's really easier than what I thought. I might try writing such a cookbook if I find the time, and reinstall everything from scratch on a new server. Also, with Jenkins, you just start a script who builds for you. What I wrote is quite a hack, I'm not sure if I want to publish that... :) Or probably with lots of !!!warning!!! added... I also would like to add some goodies to it (like piuparts tests, lintian runs, etc.). I also need to understand how to secure Jenkins. Because by default, it's impressive how much Jenkins is a security hole where you can execute any command. I was tempted to file a bug report against the package because of it. Then I saw #697617 and #700761, then gave up... :) So yeah, Jenkins is nice, but I wouldn't leave it on a public facing internet without any sort of protection (like an htpass over HTTPS). Did anyone try buildbot? It might be better for what I need. I quite disliked the fact that most of Jenkins is done through a web GUI, which was in fact, more a nuisance than anything else. Maybe buildbot would fit my needs better, so I would really appreciate if anyone can share his experience with it. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/513b565b.1040...@debian.org
Re: RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 2013-03-09 23:33:47 +0800 (+0800), Thomas Goirand wrote:
[...]
I also need to understand how to secure Jenkins. Because
by default, it's impressive how much Jenkins is a security
hole where you can execute any command. I was tempted
to file a bug report against the package because of it. Then
I saw #697617 and #700761, then gave up... :)
[...]
Yes, it's a chore to keep up with the security vulnerabilities for
Jenkins, particularly if you're following mainline instead of stable
since updates become a grab bag of (sometimes unintended) API
changes as well as new bugs and regressions. We try to be as
proactive as we can, scrape the security index on their wiki and
just plain shutdown Jenkins services on our servers until we can
validate the security fixes and get them applied in production. It's
not for the faint of heart.
At this point we're close enough to having Jenkins interactions
externally integrated with our other systems that its WebUI isn't
much use except for administrative functions. I expect it's not too
far in the future that we'll be able to lock it down such that only
administrators will have access to that interface.
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130309155027.gg29...@yuggoth.org
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 2013-03-08 14:52:48 +0100 (+0100), Thomas Koch wrote:
[...]
http://openstack-ci.github.com/publications/
[...]
I'm one of the core developers for the team which manages all that
tooling and integration for the OpenStack Project, so I'm happy to
discuss some of the nitty-gritty details, any gotchas/unpleasantness
we experience and how we work around it.
A better starting URL is http://ci.openstack.org/ and we're also
very active on freenode in #openstack-infra for those who desire
more synchronous conversation.
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130308143448.gv29...@yuggoth.org
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 03/08/2013 10:34 PM, Jeremy Stanley wrote: On 2013-03-08 14:52:48 +0100 (+0100), Thomas Koch wrote: [...] http://openstack-ci.github.com/publications/ [...] I'm one of the core developers for the team which manages all that tooling and integration for the OpenStack Project, so I'm happy to discuss some of the nitty-gritty details, any gotchas/unpleasantness we experience and how we work around it. A better starting URL is http://ci.openstack.org/ and we're also very active on freenode in #openstack-infra for those who desire more synchronous conversation. I've started copying others, and I now have a a KGB bot, and a Jenkins VM. Now, the only thing I have to do is git push, and here's the result on the #debian-openstack channel: PKG-Openstack python-json-patch thomas debian/experimental * ffa137a debian/ changelog rules PKG-Openstack python-json-patch Now running the unit tests, thanks to Michael Terry mte...@ubuntu.com for the patch (Closes: #702443). [Openstack-Cowbuild] Starting build #2 for job python-json-patch (previous build: SUCCESS) [Openstack-Cowbuild] Project python-json-patch build #2:SUCCESS in 46 sec: https://117.121.243.213/job/python-json-patch/2/ I start to really love the CI thing. I first invested a bit of time in setting-up everything, then it's crazy how much work that saves me, especially with a lot of packages (Openstack and its Python module (build-)dependencies represents nearly 50 source packages now). Once the package is finished building (in a cowbuilder, using git-buildpackage), my script puts it automatically in my private repository, and runs dpkg-scanpackages / dpkg-scansources to keep up-to-date my package repository. I think I'll add piuparts tests as well, and will also run lintian, so it appears in the build log. Jenkins helps being lazy (in the good way). Do a commit, then wait for the result. That's quite cool! Though it took me few days to have this setup. It would be nice to spare all this to other DDs, and have the infrastructure already setup for everyone. Apart from the fact that this kind of tools helps saving a lot of maintainer's time, the Gerrit thing would help giving some more restrictive access. Because for the moment, it's either we give all access, or nothing. Many times, I've granted access to others who, at the end, didn't commit anything. For these, if I had something like Gerrit, I would first ask them to send patches, which wouldn't require a full unix right into /git/openstack, which makes me nervous. Cheers, Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/513a1079.8030...@debian.org
RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
I start to really love the CI thing. I first invested a bit of time in setting-up everything, do you have a step by step cookbook for your setup. Maybe on the debian wiki ? Cheers Frederic -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/a2a20ec3b8560d408356cac2fc148e5358e63...@sun-dag1.synchrotron-soleil.fr
Re: RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 08/03/2013 17:36, PICCA Frédéric-Emmanuel wrote: I start to really love the CI thing. I first invested a bit of time in setting-up everything, do you have a step by step cookbook for your setup. Maybe on the debian wiki ? I love what Michael Prokop did and documented here: http://jenkins-debian-glue.org/ Jenkins + Debian packaging using cowbuilder The code is very clean and easy to hack. Sylvestre -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/513a186b.5000...@debian.org
RE : RE : Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
I love what Michael Prokop did and documented here: http://jenkins-debian-glue.org/ Jenkins + Debian packaging using cowbuilder The code is very clean and easy to hack. Thanks, yes it looks great. Cheers Fred -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/a2a20ec3b8560d408356cac2fc148e5358e63...@sun-dag1.synchrotron-soleil.fr
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
Thomas Koch tho...@koch.ro writes: I'm also thinking a lot about how to improve Debian by improving our Git tooling. Therefor I'm packaging Gerrit (#589436). But gerrit and its dependencies is a big project... Thank you very much for working on this! We use Gerrit extensively but so far just haven't packaged it because it was too intimidating. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87obetliyz@windlord.stanford.edu
Re: Gerrit, Git requirements, cooperation with others. was: git dangerous operations on alioth
On 2013-03-08 12:44:36 -0800 (-0800), Russ Allbery wrote:
Thank you very much for working on this! We use Gerrit extensively but so
far just haven't packaged it because it was too intimidating.
Agreed, if Gerrit gets packaged in Debian/Ubuntu I'll likely push
OpenStack to start using DEBs of it on our CI infrastructure (though
chances are we'll still rebuild from the source package because we
carry patches for features in which Google has thus far been wholly
disinterested).
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130308215201.gb29...@yuggoth.org
