Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Fri, Nov 4, 2011 at 2:00 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 12:21 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 12:20 AM, Stefan Marr p...@stefan-marr.de wrote: Hi: On 04 Nov 2011, at 00:12, Ferenc Kovacs wrote: I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. Does that meant that the committer could get an unconditional email with the result of his/her commit? Sounds very reasonable to me. yep. -- Ferenc Kovács @Tyr43l - http://tyrael.hu I set up the email notifications for the 5.3 build project: it should send a notification to: - everybody, for every build, regardless of the build result, who either triggered a build manually through the web interface, or commited to svn for that job since the last successful build. - and to internals but only for the fixed or failure results. if this works out, I will also set up the two other build and tests jobs also. Of course we still have to agree and solve the failing tests jobs(either picking a solution from my ABC list, or coming up with a better alternative), else the commiters will start getting the still failing mail for every tests build execution. -- Ferenc Kovács @Tyr43l - http://tyrael.hu I just noticed that the mail will be sent on each configuration build (debian 32bit, debian 64bit, freebsd 32bit, freebsd 64bit), which is of course a little bit too much, I'm looking into resolving this, so to only send email on the parent build result. Until that is fixed, I won't enable the email notifications for the other jobs (5.4, trunk) -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Sat, Nov 5, 2011 at 10:42 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 2:00 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 12:21 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 12:20 AM, Stefan Marr p...@stefan-marr.de wrote: Hi: On 04 Nov 2011, at 00:12, Ferenc Kovacs wrote: I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. Does that meant that the committer could get an unconditional email with the result of his/her commit? Sounds very reasonable to me. yep. -- Ferenc Kovács @Tyr43l - http://tyrael.hu I set up the email notifications for the 5.3 build project: it should send a notification to: - everybody, for every build, regardless of the build result, who either triggered a build manually through the web interface, or commited to svn for that job since the last successful build. - and to internals but only for the fixed or failure results. if this works out, I will also set up the two other build and tests jobs also. Of course we still have to agree and solve the failing tests jobs(either picking a solution from my ABC list, or coming up with a better alternative), else the commiters will start getting the still failing mail for every tests build execution. -- Ferenc Kovács @Tyr43l - http://tyrael.hu I just noticed that the mail will be sent on each configuration build (debian 32bit, debian 64bit, freebsd 32bit, freebsd 64bit), which is of course a little bit too much, I'm looking into resolving this, so to only send email on the parent build result. Until that is fixed, I won't enable the email notifications for the other jobs (5.4, trunk) -- Ferenc Kovács @Tyr43l - http://tyrael.hu The symfony2 testsuite segfaults currently for 5.4 and trunk (I did see that before, but I also see that segfault disappear :/) I created a ticket explained what I was able to dig up so far: https://bugs.php.net/bug.php?id=60223 -- Ferenc Kovács @Tyr43l - http://tyrael.hu
[PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Sat, Nov 5, 2011 at 10:42, Ferenc Kovacs tyr...@gmail.com wrote: I just noticed that the mail will be sent on each configuration build (debian 32bit, debian 64bit, freebsd 32bit, freebsd 64bit), which is of course a little bit too much, I'm looking into resolving this, so to only send email on the parent build result. Until that is fixed, I won't enable the email notifications for the other jobs (5.4, trunk) If a build fails only on one specific platform, that will need to be mailed out too. -Hannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Sat, Nov 5, 2011 at 4:23 PM, Hannes Magnusson bj...@php.net wrote: On Sat, Nov 5, 2011 at 10:42, Ferenc Kovacs tyr...@gmail.com wrote: I just noticed that the mail will be sent on each configuration build (debian 32bit, debian 64bit, freebsd 32bit, freebsd 64bit), which is of course a little bit too much, I'm looking into resolving this, so to only send email on the parent build result. Until that is fixed, I won't enable the email notifications for the other jobs (5.4, trunk) If a build fails only on one specific platform, that will need to be mailed out too. If any configuration job fails(for example http://ci.qa.php.net/job/php-src-5.3-matrix-build/architecture=x86,os=linux-debian-6.0/), it will make the whole job ( http://ci.qa.php.net/job/php-src-5.3-matrix-build/), so thats not a problem. The parent build status reflects the configuration jobs, it will be only successful if all of the configuration builds were successful too. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
[PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
2011/11/3 Ferenc Kovacs tyr...@gmail.com: On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira cont...@klaussilveira.comwrote: That's kind of a general setup, Stefan. Sending an email to the commiter that broke the build with details of the build process, as well sending an email to a mailing list. I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an access control based on IRC operators and voices? yeah, using voices would work, but then that would mean that it is only a read only channel for most people, which imo would be a bad move. I mean it would be really nice, if people could drop by to our qa related irc room and start talking about that pesky build failure. another solution would be to set up a custom irc bot, independently from jenkins, which could fetch the build results through the mailing list or rss and send it to the channel. of course extending the jenkins irc bot is also a possibility, the easiest way would be having a configuration option for disabling the whole control stuff, as we don't really need the interactivity there, you can use the web interface for that. -- Ferenc Kovács @Tyr43l - http://tyrael.hu Nice work with this Jenkins stuff! The system seems pretty useful. I like the idea to have a mailing list for build result notification, as well as send mail to the build breaker. -- Regards, Felipe Pena -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
Yes, it's a wonderful setup. Great work, Ferenc! :D This does give a nice motivation to write more tests and increase the code coverage, doesn't it? On Thu, Nov 3, 2011 at 7:22 PM, Felipe Pena felipe...@gmail.com wrote: 2011/11/3 Ferenc Kovacs tyr...@gmail.com: On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira cont...@klaussilveira.comwrote: That's kind of a general setup, Stefan. Sending an email to the commiter that broke the build with details of the build process, as well sending an email to a mailing list. I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an access control based on IRC operators and voices? yeah, using voices would work, but then that would mean that it is only a read only channel for most people, which imo would be a bad move. I mean it would be really nice, if people could drop by to our qa related irc room and start talking about that pesky build failure. another solution would be to set up a custom irc bot, independently from jenkins, which could fetch the build results through the mailing list or rss and send it to the channel. of course extending the jenkins irc bot is also a possibility, the easiest way would be having a configuration option for disabling the whole control stuff, as we don't really need the interactivity there, you can use the web interface for that. -- Ferenc Kovács @Tyr43l - http://tyrael.hu Nice work with this Jenkins stuff! The system seems pretty useful. I like the idea to have a mailing list for build result notification, as well as send mail to the build breaker. -- Regards, Felipe Pena -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php -- Klaus Silveira (011) 8564-2492 www.klaussilveira.com
[PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Thu, Nov 3, 2011 at 10:22 PM, Felipe Pena felipe...@gmail.com wrote: 2011/11/3 Ferenc Kovacs tyr...@gmail.com: On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira cont...@klaussilveira.comwrote: That's kind of a general setup, Stefan. Sending an email to the commiter that broke the build with details of the build process, as well sending an email to a mailing list. I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an access control based on IRC operators and voices? yeah, using voices would work, but then that would mean that it is only a read only channel for most people, which imo would be a bad move. I mean it would be really nice, if people could drop by to our qa related irc room and start talking about that pesky build failure. another solution would be to set up a custom irc bot, independently from jenkins, which could fetch the build results through the mailing list or rss and send it to the channel. of course extending the jenkins irc bot is also a possibility, the easiest way would be having a configuration option for disabling the whole control stuff, as we don't really need the interactivity there, you can use the web interface for that. -- Ferenc Kovács @Tyr43l - http://tyrael.hu Nice work with this Jenkins stuff! The system seems pretty useful. I like the idea to have a mailing list for build result notification, as well as send mail to the build breaker. Thanks for the feedback. We could set up the email notification any time, we would just have to agree where to send (internals, php-qa, creating a dedicated mailing list?) and when to send (as I added to the RFC, having failing tests can be a problem from the notification POV). We have to find a good middle way, else we will either miss the notifications, or we will be overwhelmed, so people will just ignore it. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Thu, Nov 3, 2011 at 10:25 PM, Klaus Silveira cont...@klaussilveira.comwrote: Yes, it's a wonderful setup. Great work, Ferenc! :D This does give a nice motivation to write more tests and increase the code coverage, doesn't it? Yes, and first of all, to fix the currently failing tests. We also had a discussion with Hannes, that the build machines could also be used for hunting down bugs, for which the developer doesn't have an environment to test and reproduce the problem. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On 11/03/2011 02:26 PM, Ferenc Kovacs wrote: We could set up the email notification any time, we would just have to agree where to send (internals, php-qa, creating a dedicated mailing list?) and when to send (as I added to the RFC, having failing tests can be a problem from the notification POV). We have to find a good middle way, else we will either miss the notifications, or we will be overwhelmed, so people will just ignore it. Ideally they should go to internals if you can get them to be non-noisy. As long as it only reports a test status change from the previous run, hopefully that wouldn't create much noise. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Thu, Nov 3, 2011 at 11:33 PM, Rasmus Lerdorf ras...@lerdorf.com wrote: On 11/03/2011 02:26 PM, Ferenc Kovacs wrote: We could set up the email notification any time, we would just have to agree where to send (internals, php-qa, creating a dedicated mailing list?) and when to send (as I added to the RFC, having failing tests can be a problem from the notification POV). We have to find a good middle way, else we will either miss the notifications, or we will be overwhelmed, so people will just ignore it. Ideally they should go to internals if you can get them to be non-noisy. true As long as it only reports a test status change from the previous run, hopefully that wouldn't create much noise. yeah, and that is a little bit of a problem, because as I mentioned by default the jenkins way is that your tests passes: http://jenkins.361315.n4.nabble.com/Test-failure-baseline-td386476.html so we have three option as far as I can see: A, it will report every test failure, but we fix all of our tests so we are cool. (my preference) B, same as A, but until we fix all of the tests, we mark the failing test as XFAIL, which won't be counted as failed test (I'm against it, as time taught us that we are lazy, and XFAILs will be kept that way. :/) C, Somebody(me) do some scripting, and our currently failing tests won't fail the build, only make it unstable, but any other test failure will fail the build. I think I didn't explained in the RFC, because currently we don't use it, but in jenkins there is a third result type for a build apart success and failure: unstable. Most plugins support this, as does the email notification plugin also, so this should also work. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On 11/03/2011 03:47 PM, Ferenc Kovacs wrote: A, it will report every test failure, but we fix all of our tests so we are cool. (my preference) But even if we do that, when a test does fail, it may take a couple of days to fix it. It would be nice if we didn't get an internals email for every commit that triggers a build during that time. So even in this ideal case I think the notification mechanism needs to be smarter. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Thu, Nov 3, 2011 at 11:58 PM, Rasmus Lerdorf ras...@lerdorf.com wrote: On 11/03/2011 03:47 PM, Ferenc Kovacs wrote: A, it will report every test failure, but we fix all of our tests so we are cool. (my preference) But even if we do that, when a test does fail, it may take a couple of days to fix it. It would be nice if we didn't get an internals email for every commit that triggers a build during that time. So even in this ideal case I think the notification mechanism needs to be smarter. -Rasmus using the email-ext plugin: https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin one can bind email notification on the following events: - Failure - Unstable - Still-Failing - Success - Fixed - Still-Unstable - (Before-Build) So we could configure to only send emails on Failure and Fixed(I ignored the Unstable on purpose for the sake of simplicity, of course we also use this for option C). That way we would only see the notification on the first error, and the next mail would be when we finally fixed it. This is still suboptimal, as we wouldn't get email notifications about the other test failures introduced between the first failure and the fix, but I think that having a failed build would mean that somebody/everybody is working on fixing the build, so those test failures would also be spotted(on the web interface) and fixed (without that, the build won't succeed). I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. How does it sounds to you? -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
Hi: On 04 Nov 2011, at 00:12, Ferenc Kovacs wrote: I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. Does that meant that the committer could get an unconditional email with the result of his/her commit? Sounds very reasonable to me. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Fri, Nov 4, 2011 at 12:20 AM, Stefan Marr p...@stefan-marr.de wrote: Hi: On 04 Nov 2011, at 00:12, Ferenc Kovacs wrote: I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. Does that meant that the committer could get an unconditional email with the result of his/her commit? Sounds very reasonable to me. yep. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] Re: [PHP-QA] Re: [PHP-DEV] CI for 5.4
On Fri, Nov 4, 2011 at 12:21 AM, Ferenc Kovacs tyr...@gmail.com wrote: On Fri, Nov 4, 2011 at 12:20 AM, Stefan Marr p...@stefan-marr.de wrote: Hi: On 04 Nov 2011, at 00:12, Ferenc Kovacs wrote: I almost forget to mention, but the email notification also supports defining different recipients for each event, so for example the commiters could still get the notification for each of their commits/builds until the build is back to normal, that way the list wouldn't be spammed, but some active contributors would be still continuously bugged. Does that meant that the committer could get an unconditional email with the result of his/her commit? Sounds very reasonable to me. yep. -- Ferenc Kovács @Tyr43l - http://tyrael.hu I set up the email notifications for the 5.3 build project: it should send a notification to: - everybody, for every build, regardless of the build result, who either triggered a build manually through the web interface, or commited to svn for that job since the last successful build. - and to internals but only for the fixed or failure results. if this works out, I will also set up the two other build and tests jobs also. Of course we still have to agree and solve the failing tests jobs(either picking a solution from my ABC list, or coming up with a better alternative), else the commiters will start getting the still failing mail for every tests build execution. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
