On May 21, 2008, at 9:22 AM, Marcin Owsiany wrote:
>
> On Tue, May 20, 2008 at 09:54:23PM -0500, Luke Kanies wrote:
>>
>> On May 20, 2008, at 3:27 PM, Marcin Owsiany wrote:
>>
>>>
>>> Hi all,
>>>
>>> After I managed to invoke the rspec test runner (see #1237), I
>>> realized
>>> with sadness that they are far from usable. I got:
>>> 3053 examples, 219 failures, 30 pending - when running as root
>>> and
>>> 3053 examples, 145 failures, 31 pending - when running as non-root
>>> This is on master branch.
>>
>> I get zero failures on master or 0.24.x, although I haven't run
>> either
>> of them as root.
>
> Hm, even on Debian? I'm wondering what's the reason of #1244
90% of the time you get "no providers for...", it's a problem related
to the type in question getting reloaded when it shouldn't be.
E.g., this process happens:
1 - type gets loaded
2 - type loads all of its providers
3 - type gets reloaded, replacing the provider list
4 - new type tries loading providers but nothing happens because
they've all been loaded already (but lost)
The problem is that Puppet's loader doesn't play well with Ruby's
'require'. Clearly a bug, but not usually something that's hit except
during testing, and easily fixed, so I haven't refactored to fix it.
However, the ldap providers I recently merged seem to have added a new
kind of broken there, and I'm not sure that James has merged in my
recent fixes. I *expect* this problem is ldap related, but I've been
away from home all week and thus have only tested on my mac..
>
>> no tests should fail for anyone ever.
>
> Good to hear that, it's encouraging. In that case I will work on them
> for a while, maybe most of them do have a common cause that will be
> easy
> to fix.
Great.
>
>> If we have a continuous integration service, then I would definitely
>> never release a product that had non-green tests on any supported
>> platform. Is anyone in a position to set such a thing up and
>> maintain
>> it?
>
> This would be a fun project, however I know little of that EC2 thing.
> Let me see if I can get the tests to pass first. Then I'll try to
> create
> a basic set of puppet manifests which would turn a bare base Debian
> install into a basic continuous integration service, in a VM. If that
> works, I'll have a look at EC2.
That's a great place to start.
>
>> they're only running during the actual test process (e.g., once a day
>> for an hour or so, rather than 24hrs a day).
>
> The tests seem to take only a couple of minutes on my oldish laptop,
> so
> I guess 5 minute runs every, say, 6 hours would be better.
I was just thinking of cost. Either we have registered VMs that are
consistent and we can load whenever we want, but we can't run all the
time because it's expensive, or we host the VMs ourselves somewhere.
The former is easier because of ec2, I think, but the latter is
cheaper. My concern with the latter is the difficulty in federating
the results of disparate test runs on different systems maintained by
essentially random community members. Guaranteeing consistent results
will always be difficult.
--
SELF-EVIDENT, adj. Evident to one's self and to nobody else.
-- Ambrose Bierce
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
