Re: buildbot - an experiment
On Jan 4, 2008, at 7:09 AM, nadim khemir wrote: I received an answer from Eric : I wish I did have some kind of comparison. Here's what one user wrote about choosing cabie: http://www.golden-gryphon.com/blog/manoj/blog/2007/11/06/Continuous_Automated_Build_and_Integration_Environment.html I looked here: http://damagecontrol.codehaus.org/Continuous+Integration+Server+Feature+Matrix Maybe Eric could sign up as a Confluence user (http://docs.codehaus.org/signup.action ) and add cabie to that matrix? Seems like he has thought through the answers to all the entries in the table. I say: The more choices the better! --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On Jan 4, 2008, at 5:56 PM, James E Keenan wrote: David Cantrell wrote: On Tue, Jan 01, 2008 at 08:23:52PM -0500, James E Keenan wrote: David Cantrell wrote: If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. svn co https://svn.perl.org/parrot/trunk/ parrot_test cd parrot_test perl Configure.pl make ... /usr/local/bin/perl /home/david/parrot_test/tools/build/pmc2c.pl --c subproxy.pmc Cannot restore overloading on HASH(0x823a074) (package Parrot::Pmc2c::Emitter) at blib/lib/Storable.pm (autosplit into blib/lib/auto/Storable/_retrieve.al) line 328, at /home/david/parrot_test/tools/build/dynpmc.pl line 199 make[1]: *** [all] Error 255 make[1]: Leaving directory `/home/david/parrot_test/src/dynpmc' make: *** [dynpmc.dummy] Error 2 That's on Linux. There' probably not much point me testing it on more obscure platforms right now -) FWIW, the checkout and build and test also got failures for me on: Linux 2.6.22.14-72.fc6 #1 SMP x86_64 x86_64 x86_64 GNU/Linux using perl, v5.8.8 built for x86_64-linux-thread-multi gcc (GCC) 4.1.2 20070626 (Red Hat 4.1.2-13) t/stm/basic_mt.2/4 # Failed test (t/stm/basic_mt.t at line 93) # Exited with error code: [SIGNAL 11] # Received: # # Expected: # okay # t/stm/basic_mt.4/4 # Looks like you failed 1 test of 4. t/stm/basic_mt. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests (less 1 skipped subtest: 2 okay) Test Summary Report --- t/configure/115-auto_warnings-01.t (Wstat: 0 Tests: 4 Failed: 0) TODO passed: 4 t/src/intlist.t(Wstat: 0 Tests: 4 Failed: 0) TODO passed: 1-4 t/src/io.t (Wstat: 0 Tests: 20 Failed: 0) TODO passed: 16-17, 19 t/stm/basic_mt.t (Wstat: 256 Tests: 4 Failed: 1) Failed test number(s): 2 Non-zero exit status: 1 Files=545, Tests=10580, 1602 wallclock secs ( 4.26 usr 1.88 sys + 1234.14 cusr 53.52 csys = 1293.80 CPU) Result: FAIL Failed 1/545 test programs. 1/10580 subtests failed. --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
Turns out the parrot build/test failed on SVN revision 24566, but passed in revision 24567 So, I am gonna see if i can make a buildbot config to build and test parrot, using an SVN polling configuration where I'll try and check the SVN repo every N minutes and then do a checkout/build/test if there are no further commits for another X minutes. --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
# from Matisse Enzer # on Saturday 05 January 2008 13:25: Turns out the parrot build/test failed on SVN revision 24566, but passed in revision 24567 So, I am gonna see if i can make a buildbot config to build and test parrot, using an SVN polling configuration where I'll try and check the SVN repo every N minutes and then do a checkout/build/test if there are no further commits for another X minutes. Is there any sort of build_ok/test_ok matrix for $svn_rev x $platform for parrot? Distributed, cross-platform projects tend to suffer from oh yeah, trunk is broken on $platform right now (i.e. as of 10 minutes ago), which is hard to know if you're not in the know. --Eric -- [...proprietary software is better than gpl because...] There is value in having somebody you can write checks to, and they fix bugs. --Mike McNamara (president of a commercial software company) --- http://scratchcomputing.com ---
Re: buildbot - an experiment
On Jan 5, 2008, at 4:20 PM, Eric Wilhelm wrote: Is there any sort of build_ok/test_ok matrix for $svn_rev x $platform for parrot? Distributed, cross-platform projects tend to suffer from oh yeah, trunk is broken on $platform right now (i.e. as of 10 minutes ago), which is hard to know if you're not in the know. Well this is exactly the sort of problem buildbot is supposed to handle: - someone sets up a central buildmaster which watches the SVN repo and sends out change notices to the appropriate build slaves (maybe some slaves only get changes once a day, others get every commit, etc.) - many people set up build slaves, on different platforms - the buildmaster shows the results from all slaves on a web page, and perhaps also on an IRC channel, email list, etc. --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
I've set up a parrot buildmaster/slave, currently located at: http://buildbot.eigenstate.net:8040/ --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On Thursday 03 January 2008 19.42.26 nadim khemir wrote: I did send a mail to cabie author asking if he had made a comparison between cabie and other systems. I also invited him to join us on this mailing list. I'll forward his answer if he does answer me. I received an answer from Eric : I wish I did have some kind of comparison. Here's what one user wrote about choosing cabie: http://www.golden-gryphon.com/blog/manoj/blog/2007/11/06/Continuous_Automated_Build_and_Integration_Environment.html I looked here: http://damagecontrol.codehaus.org/Continuous+Integration+Server+Feature+Matrix And if I go line by line here's how I stack up: Project origin: me, hosted on tigris.org Implementation language: Perl, Javascript, and the new UI dashboard in PHP (nearing completion) Free: yes Issue tracker: User definable, simple bug tracking and linking, no updating Online demo: no, but being worked on Number of active developers: 1 (lots of lurkers though, I need help) SCM support: Subversion, CVS, Perforce (utilizes a cmbroker, with integration of other tools being pretty simple) SCM filtering: limited, and definable Multi-SCM: not withing the same job Parallel builds: yes Distributed builds: Not within cabie Agents code auto updated: no Manually force builds: yes SCM triggered builds: if the trigger is defined and uses calls to the cabie build client SCM poll based builds: yes Temporal build scheduling: VIA the os, as a cron job or an at job Builds promotion: yes Interproject Dependencies: limited, can 'lock' a job that may be dependent on the output of another job Builds deletion: yes Reproduce history builds: No, but that's high on my priority list Proactive (can prevent build breakages): Not sure what they mean, but don't think so Detect new failing tests while build: User definable through 'errors' customization Notify when first test in build fails: Only if the buildname.errors file has been defined with searchable strings that would include test conditions User authentication: in new UI, weak, MD5 based User authorization schemes: HTTP basic auth LDAP integration: not at this time Kerebos: no Single sign-on: no Custom JAAS: no RSA SecurID: no Confluence: no Email: yes Run executable: yes FTP: Custom IRC: no Jabber: no Lotus sametime: no ProjectStart: no RSS: yes SCP: Custom Windows system tray: no Formatted logging: mysql, xml (via mysql) Yahoo Messenger: no MSN messenger: no X10: no View changesets: yes, past, present and pending including full historical data = New Web Interface == Add new projects: yes Clone projects: yes Delete projects: yes, 2 modes, from view and permanent Modify projects: yes Kill builds: yes Pause builds: no Access to build artifacts: yes Browse CI's working copy: no Search in builds: for defects Historic graphs: no Self-updating webpage: yes Multi project support: yes Multi project view: yes Add/remove agent machines: no =Directly supported build tools== Shell/command script: yes, only mode supported since ant, nant, make, and msdev studio projects can all be run from the command line =Tools Integration = Bugtracking systems and source browsers and user definable by job properties defined in mysql =Remote management api== XML-RPC =Installation and Configuration= Windows installer: no Self contained distribution: no (mostly yes, but there are dependencies for perl) Additional Dependencies: mysql, perl, subversion, cvs, perforce, required perl modules identified by cabieconfig.pl Execution platform: perl (PHP for new UI) Project platform: anything that can be build from the command line Preferred build tool: any Requires modification to build scripts: some Supports multiple projects: yes Automatic configuration from build script: no Text file configuration: projects - no (mysql), errors, disclaimers, exclusion lists yes... The shots are from a non-authenticated session, an admin session and a super-admin session. (NKH: I put the images here http://khemir.net/files_for_download/cabbie/ in one big html page. Don't ask for better presentation this is my second html page and probably one of the 10 pages I'll write before I retire. I'll leave them there for a few days) I left out a few things, those that display passwords. The screenshots are for the 'interactive' sections of the cabie ui. The server is entirely in perl using rpc-xml, and the ui is in php (I figured I'd checkout php, although I'm not really great at either). It continues to be a work in progress. I have had great suggestions from a variety of companies including Facebook, just wish I could work on it full time Right now most of my pending changes are changes required to the cm broker to handle html for CVS and Subversion, if the project uses Perforce then
Re: buildbot - an experiment
David Cantrell wrote: On Tue, Jan 01, 2008 at 08:23:52PM -0500, James E Keenan wrote: David Cantrell wrote: If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. svn co https://svn.perl.org/parrot/trunk/ parrot_test cd parrot_test perl Configure.pl make ... /usr/local/bin/perl /home/david/parrot_test/tools/build/pmc2c.pl --c subproxy.pmc Cannot restore overloading on HASH(0x823a074) (package Parrot::Pmc2c::Emitter) at blib/lib/Storable.pm (autosplit into blib/lib/auto/Storable/_retrieve.al) line 328, at /home/david/parrot_test/tools/build/dynpmc.pl line 199 make[1]: *** [all] Error 255 make[1]: Leaving directory `/home/david/parrot_test/src/dynpmc' make: *** [dynpmc.dummy] Error 2 That's on Linux. There' probably not much point me testing it on more obscure platforms right now -) Right now I unfortunately don't have the tuits to follow perl.qa closely and experiment with buildbots. As for the failure above, is this occurring while being run by a buildbot ... or just run by a human. If the latter, then it's a bug of some sort in Parrot and you should file it at [EMAIL PROTECTED], attaching your 'make' output up to the point of failure. Thank you very much for trying this out. Jim Keenan
Re: buildbot - an experiment
On 2 Jan 2008, at 20:02, nadim khemir wrote: Cabie seems to be as good, if not better (psst, it's written in Perl). Does that make it better? :) -- Andy Armstrong, Hexten
Re: buildbot - an experiment
On 3 Jan 2008, at 09:15, Adrian Howard wrote: On 2 Jan 2008, at 20:02, nadim khemir wrote: On Saturday 29 December 2007 10.11.41 Matisse Enzer wrote: I've spent some of this holiday season learning how to set up BuildBot ... Cabie seems to be as good, if not better (psst, it's written in Perl). http://cabie.tigris.org/ There's also Michael's rather swizzy Smolder project. http://sourceforge.net/projects/smolder Indeed. Smolder++. You still need something to manage a build farm though. I got about 80% done on Test::SmokeStack (leaving only another few 80%s to do). That was going to talk to Smolder. Turns out that buildbot is almost exactly what SmokeStack was going to be - so I'll probably divert my attention to buildbot and get that talking to Smolder instead. My Python Fu is weak but I spent a productive couple of hours hacking about with buildbot last night. My aim is to make it easy to test all the modules in my SVN against multiple Perl's on multiple machines and have the config changes to add a new module be as minimal as possible. I haven't quite got it right yet: all my modules are currently in a single buildbot project which obviously isn't the intended way of doing things judging by how bb's web UI has scaled to handle it (I have a page about 20,000 pixels wide :) Once I get it working a bit more satisfactorily I'll make the code available somewhere. -- Andy Armstrong, Hexten
Re: buildbot - an experiment
Andy Armstrong wrote: On 3 Jan 2008, at 09:15, Adrian Howard wrote: There's also Michael's rather swizzy Smolder project. http://sourceforge.net/projects/smolder Indeed. Smolder++. You still need something to manage a build farm though. I got about 80% done on Test::SmokeStack (leaving only another few 80%s to do). That was going to talk to Smolder. Thanks for endorsement :) Turns out that buildbot is almost exactly what SmokeStack was going to be - so I'll probably divert my attention to buildbot and get that talking to Smolder instead. Actually I've been thinking the same thing! Buildbot just needs to send a TAP Archive to a smolder install. That way buildbot can concentrate on making it easy to do builds and runs on multiple platforms/machines. And Smolder can concentrate on making useful reports. My Python Fu is weak but I spent a productive couple of hours hacking about with buildbot last night. My aim is to make it easy to test all the modules in my SVN against multiple Perl's on multiple machines and have the config changes to add a new module be as minimal as possible. Shouldn't be too much python required. Just tell prove to --archive and then find some HTTP library to send it to smolder. Or you could just fork off and use the smolder_smoke_signal script that comes with smolder to do the actual work. In fact that's probably the best way since that's the supported interface right now for sending reports to smolder in an automated way. -- Michael Peters Developer Plus Three, LP
Re: buildbot - an experiment
On 3 Jan 2008, at 12:53, Michael Peters wrote: My Python Fu is weak but I spent a productive couple of hours hacking about with buildbot last night. My aim is to make it easy to test all the modules in my SVN against multiple Perl's on multiple machines and have the config changes to add a new module be as minimal as possible. Shouldn't be too much python required. Just tell prove to --archive and then find some HTTP library to send it to smolder. Or you could just fork off and use the smolder_smoke_signal script that comes with smolder to do the actual work. In fact that's probably the best way since that's the supported interface right now for sending reports to smolder in an automated way. Sure - the Python is to simplify the process of managing buildbot's config to support multiple Perl versions and stuff like that. -- Andy Armstrong, Hexten
Re: buildbot - an experiment
On Jan 2, 2008, at 12:02 PM, nadim khemir wrote: On Saturday 29 December 2007 10.11.41 Matisse Enzer wrote: I've spent some of this holiday season learning how to set up BuildBot ... Cabie seems to be as good, if not better (psst, it's written in Perl). http://cabie.tigris.org/ Can you explain your comparison in more detail? I could not find any documentation for CABIE and so it's hard for me to compare. One of the things that is attractive to me about BuildBot is that it makes it fairly easy to create many builders on different platforms that all report back to a central buildmaster, but maybe CABIE does this also? --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On Thursday 03 January 2008 11.50.03 Andy Armstrong wrote: Cabie seems to be as good, if not better (psst, it's written in Perl). Does that make it better? :) RTFM! In the feature list the authors writes * It's really cool!. Since cool is better than good, I assume it is ;) The written in Perl did not refer to the better. I would prefer something that I can modify without having to swear too much about forgotten ':'s and '\t's. On Thursday 03 January 2008 18.52.11 Matisse Enzer wrote: Can you explain your comparison in more detail? I could not find any documentation for CABIE and so it's hard for me to compare. I did a quick comparison on the feature list no more. I've been following cabie for a while now without using it. last time, I was out voted by my own development team and we used cruise control instead. We have a project starting in a few weeks maybe this time I can use cabie (or buildbot) One of the things that is attractive to me about BuildBot is that it makes it fairly easy to create many builders on different platforms that all report back to a central buildmaster, but maybe CABIE does this also? I believe so. http://www.yolinux.com/TUTORIALS/CabieBuildSystem.html I did send a mail to cabie author asking if he had made a comparison between cabie and other systems. I also invited him to join us on this mailing list. I'll forward his answer if he does answer me. Cheers, Nadim
Re: buildbot - an experiment
On Tue, Jan 01, 2008 at 08:23:52PM -0500, James E Keenan wrote: David Cantrell wrote: If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. svn co https://svn.perl.org/parrot/trunk/ parrot_test cd parrot_test perl Configure.pl make ... /usr/local/bin/perl /home/david/parrot_test/tools/build/pmc2c.pl --c subproxy.pmc Cannot restore overloading on HASH(0x823a074) (package Parrot::Pmc2c::Emitter) at blib/lib/Storable.pm (autosplit into blib/lib/auto/Storable/_retrieve.al) line 328, at /home/david/parrot_test/tools/build/dynpmc.pl line 199 make[1]: *** [all] Error 255 make[1]: Leaving directory `/home/david/parrot_test/src/dynpmc' make: *** [dynpmc.dummy] Error 2 That's on Linux. There' probably not much point me testing it on more obscure platforms right now -) So anyway, I'll wrap this in a small shell script. How often should I run tests, and to where should I send the results? And should I send anything at all if everything appears to work? -- David Cantrell | Cake Smuggler Extraordinaire Your call is important to me. To see if it's important to you I'm going to make you wait on hold for five minutes before putting you through to Dave's mobile. This call will be recorded for quality and amusement purposes.
Re: buildbot - an experiment
David Cantrell wrote:
On Sat, Dec 29, 2007 at 05:51:50PM -0500, James E Keenan wrote:
If anyone can give me an idiots' guide to how to grab the most recent
source tree, build it, and test it, then I can test it on the same boxes
as I do CPAN testing, plus maybe a couple of others.
svn co https://svn.perl.org/parrot/trunk/ parrot_test
cd parrot_test
perl Configure.pl
make
make test
(If you're not looking to commit, you may be able to get by with http
rather than https: UNTESTED.)
('make smoke' can be substituted for 'make test'.)
Re: buildbot - an experiment
On Jan 1, 2008, at 5:23 PM, James E Keenan wrote:
svn co https://svn.perl.org/parrot/trunk/ parrot_test
cd parrot_test
perl Configure.pl
make
make test
(If you're not looking to commit, you may be able to get by with
http rather than https: UNTESTED.)
('make smoke' can be substituted for 'make test'.)
Make sure to do a checkout manually using the same account that the
build slave is using - and accept the SSL certificate permanently
that first time.
The BuildBot 'ChangeSource' class to use is probably:
http://buildbot.net/repos/release/docs/buildbot.html#SVNPoller
something like this in master.cfg:
from buildbot.changes.svnpoller import SVNPoller
c['change_source'] = SVNPoller(https://svn.perl.org/parrot/
trunk)
Maybe this weekend I'll see if i can set up a configuration in my own
buildbot buildmaster that works.
---
Matisse Enzer [EMAIL PROTECTED]
http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On Saturday 29 December 2007 10.11.41 Matisse Enzer wrote: I've spent some of this holiday season learning how to set up BuildBot ... Cabie seems to be as good, if not better (psst, it's written in Perl). http://cabie.tigris.org/ Cheers, Nadim.
Re: buildbot - an experiment
On Saturday 29 December 2007 10.11.41 Matisse Enzer wrote: I've spent some of this holiday season learning how to set up BuildBot ... I've also found this one to be written in Perl: http://www.johnkeiser.com/mozilla/tbox3.html other links http://en.wikipedia.org/wiki/Continuous_Integration
Re: buildbot - an experiment
# from nadim khemir # on Wednesday 02 January 2008 12:02: Cabie seems to be as good, if not better (psst, it's written in Perl). http://cabie.tigris.org/ Well, it says it is written in PERL, which I have found to be a very scary and unworkable language derived from FORTRAN or maybe C ;-) That said, CABIE appears to be written in PErl. It is a bit verbose and seems to have no tests, but is not as entirely horrid as the PERL description would lead you to suspect. --Eric -- We who cut mere stones must always be envisioning cathedrals. --Quarry worker's creed --- http://scratchcomputing.com ---
Re: buildbot - an experiment
On Dec 31, 2007, at 4:24 PM, David Cantrell wrote: On Sat, Dec 29, 2007 at 05:51:50PM -0500, James E Keenan wrote: How might this be used to perform smoke-testing for a project like Parrot, where we want to test on many combinations of operating system, platform and C compiler? If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. Also, any updates to the wiki would be helpful: http://perl-qa.hexten.net/wiki/index.php/Buildbot --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On Sat, Dec 29, 2007 at 05:51:50PM -0500, James E Keenan wrote: How might this be used to perform smoke-testing for a project like Parrot, where we want to test on many combinations of operating system, platform and C compiler? If anyone can give me an idiots' guide to how to grab the most recent source tree, build it, and test it, then I can test it on the same boxes as I do CPAN testing, plus maybe a couple of others. -- David Cantrell | Cake Smuggler Extraordinaire PLEASE NOTE: This message was meant to offend everyone equally, regardless of race, creed, sexual orientation, politics, choice of beer, operating system, mode of transport, or their editor.
Re: buildbot - an experiment
On 29 Dec 2007, at 09:11, Matisse Enzer wrote: For now I have the build status pages for the three projects at:` http://buildbot.eigenstate.net:8010/ http://buildbot.eigenstate.net:8020/ http://buildbot.eigenstate.net:8030/ That's very interesting Matisse. We currently have a fairly ad-hoc smoking setup for Test::Harness that runs on a few machines here. It looks as if I could replace that and make it more flexible using buildbot. Do you have any useful config scripts you could share? And if so would you like to make a page on the wiki? I've created a stub here: http://perl-qa.hexten.net/wiki/index.php/Continuous_Integration -- Andy Armstrong, Hexten
Re: buildbot - an experiment
On Dec 29, 2007, at 6:57 AM, Andy Armstrong wrote: That's very interesting Matisse. We currently have a fairly ad-hoc smoking setup for Test::Harness that runs on a few machines here. It looks as if I could replace that and make it more flexible using buildbot. Do you have any useful config scripts you could share? And if so would you like to make a page on the wiki? I've created a stub here: http://perl-qa.hexten.net/wiki/index.php/Continuous_Integration Cool. I've started to dump very rough notes into http://perl-qa.hexten.net/wiki/index.php/Buildbot and will do more over the weekend. --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: buildbot - an experiment
On 29 Dec 2007, at 18:46, Matisse Enzer wrote: On Dec 29, 2007, at 6:57 AM, Andy Armstrong wrote: That's very interesting Matisse. We currently have a fairly ad-hoc smoking setup for Test::Harness that runs on a few machines here. It looks as if I could replace that and make it more flexible using buildbot. Do you have any useful config scripts you could share? And if so would you like to make a page on the wiki? I've created a stub here: http://perl-qa.hexten.net/wiki/index.php/Continuous_Integration Cool. I've started to dump very rough notes into http://perl-qa.hexten.net/wiki/index.php/Buildbot and will do more over the weekend. Thanks Matisse :) -- Andy Armstrong, Hexten
Re: buildbot - an experiment
Matisse Enzer wrote: I've spent some of this holiday season learning how to set up BuildBot (http://buildbot.net/) which is a Continuous Integration system that is especially aimed at open-source style projects: You set up a central build master, and one or more build slaves - and it is very easy for someone to set up a new build slave, so if you have some new platform you want to test a project on you can add a build slave. The buld-master admin has to add your slave on the master side, and the slave needs to be able to access the source code repository. How might this be used to perform smoke-testing for a project like Parrot, where we want to test on many combinations of operating system, platform and C compiler?
