I could be wrong, I speak to my understanding of things, not gospel from on-high :-)
Response inline: On Tue, May 20, 2008 at 1:27 PM, Marcin Owsiany <[EMAIL PROTECTED]> wrote: > After I managed to invoke the rspec test runner (see #1237), I realized > with sadness that they are far from usable. I got: > 3053 examples, 219 failures, 30 pending - when running as root > and > 3053 examples, 145 failures, 31 pending - when running as non-root > This is on master branch. Odd. I get *very* different results. As myself: "3099 examples, 76 failures, 31 pending" As root: "3099 examples, 191 failures, 30 pending" - but we could have different states. My master is at commit fe157f239a301abb52f81c62719355c8e50c970c > For me personally (and I think for anyone new to puppet, who would like > to produce high-quality code) this is a huge barrier to puppet > development. Development should not be taking place on the master branch. Most contributions should be coded against the current stable branch (0.24.x as of this writing). Test results against 0.24.x commit 84a787a2a764a5035f7cbb8d30f94fc601bed154 look much better: Non-root: 2697 examples, 0 failures, 27 pending root: 2697 examples, 35 failures, 27 pending I'm not sure whether we should address the failures that occur when running as root - that's a call for someone more test-savvy than I. > The reason is that until you gain intimate knowledge of how puppet works > inside, there is no way you can be even remotely sure that a change that > you are about to introduce won't break something. (Even with tests you > cannot be certain, but at least you can be reasonably confident). Well, as I mentioned above, you can develop against the stable branch which doesn't have failures (except running as root). This should help you to see if you break something. Also, keep in mind we don't have 100% test coverage yet. I am working on migrating tests to RSpec from Test::Unit and I'm afraid the work is pretty slow going. > So I have several questions: > > - do we at all agree that there should be NO failing tests at any point > in time, and that any failing test is a bug? Even in master branch? > And as a result, no commit should introduce a test failure? Ideally, there would be no failing tests in the "official" repository. I definitely think Luke is working towards this goal. I know for a fact he will not accept new commits which lack tests or cause an existing test to fail (I've submitted commits like that). We should definitely work to fix the tests. Whether that takes priority over testing code that currently lacks tests is another question for Luke. > - is there anyone at all for whom all tests work at the moment? Or am I > just very unlucky and it does work for everyone except for me? :) See above. > - how do we tackle this problem? I cannot go through 200 failing tests > alone and fix them in reasonable time. We need to split the work > somehow. If you are eager to contribute, this is clearly an area that needs work. So, instead of thinking of it as fixing 200 failing tests, think of it as fixing 1. When you are done, fix another. Hopefully others will also help with this :-) Certainly I will if Luke says that this should be higher-priority than the work I'm doing to migrate tests to RSpec. > - how do we make sure that this problem does not reappear? Should we > set up some continuous integration environment and assume "project > culture" would convince developers to fix the problems? Or should we > go a step further and only ever release software tested and built by > the continuous integration service? Right now I believe we are still in the "benevolent dictator" stage (possibly dictators, I think James Turnbull can commit to the official repo, too). We just need a confirmation from the people with commit access to the official repo that commits which cause test failures (or lack tests) will not be integrated into the official repository going forward. Once we have that, we will be able to make progress fixing the broken tests because there won't be more broken tests cropping up. > I know someone (Luke?) mentioned that it would be nice to have a > build farm which would run the tests on all supported platforms. But > I think that running them on a regular basis even on a single > platform would be much better than the current situation. To that end, I'll send a mail to the list offering up some cycles on my FreeBSD box for this. Thanks for bringing this up, I think it is a good idea to address this. --Paul --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en -~----------~----~----~----~------~----~------~--~---
