Issue #2879 has been updated by Markus Roberts. Target version changed from Puppet - 0.25.2 to Puppet - 0.25.3
These issues aren't visible to people who aren't looking for them, and this ticket is probably too much of a grab-bag in any case. I'm bumping to 0.25.3 and we'll probably want to split it up into more actionable items. ---------------------------------------- Bug #2879: Test isolation: test idempotency http://projects.reductivelabs.com/issues/2879 Author: Markus Roberts Status: Code Insufficient Priority: Low Assigned to: Jesse Wolfe Category: testing Target version: 0.25.3 Affected version: 0.25.1 Keywords: Branch: This is a grab bag of issues centered around the observation that a valid test should produce the same results if run twice in succession. It's a rather low bar, but some of our tests still fail to meet it. It's also something that you wouldn't see unless you specifically went looking for it (so no one is going to be mislead by these test failures in the normal course of things). Still, they're worth cleaning up since each one is, at least potentially, a source of future spurious test failures. A test that changes it's behavior when run twice in succession must 1) depend in some way on external state, 2) change some aspect of the external external state, and thus could be either the cause or the victim in a more normal test isolation problem. To reproduce any of these just run the same test twice: ruby spec/unit/myspec.rb spec/unit/myspec.rb -t 60 --format s While I am lumping these all into one ticket there are actually several sub-categories. h3. Setup/tear down problems. First off we have specs that fail one or more test on the second pass spec/unit/application/puppetmasterd.rb fails one test on the second pass: should dispatch to parseonly if parseonly is set (FAILED - #) spec/unit/util/autoload.rb fails four tests on the second pass: should not die an if a RuntimeError exception is thrown (FAILED - #) should not die an if a LoadError exception is thrown (FAILED - #) should not die an if a SyntaxError exception is thrown (FAILED - #) should require the full path to the file (FAILED - #) spec/unit/provider/ssh_authorized_key/parsed.rb fails three tests on the second pass: should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys1 (FAILED - #) should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys2 (FAILED - #) should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys (FAILED - #) h3. Registration Some parts of the system perform registrations; tests that use these parts of the system may unwittingly change persistent state: spec/integration/indirector/rest.rb ./lib/puppet/indirector.rb:21:in `indirects': Already handling indirection for test_indirected_foo; cannot also handle test_indirected_foo (ArgumentError) from ./spec/integration/indirector/rest.rb:11 spec/unit/util/queue.rb ./lib/puppet/util/queue.rb:55:in `register_queue_type': Queue type default is already registered (Puppet::Error) from ./spec/unit/util/queue.rb:22 h3. Head scratchers Several specs contain tests that fail if run once, but succeed (on both passes) if the spec is run twice. I have no ready explanation for this. The specs and the tests are: spec/unit/network/client.rb should start the http client up on creation (FAILED - #) Test isolation: spec/unit/provider/mount/parsed.rb should write the mount to disk when :flush is called (FAILED - #) Test isolation: spec/unit/indirector/indirection.rb should not create a terminus instance until one is actually needed (FAILED - #) h3. Constant redefinition Finally, we have a number of tests which generate warnings because they define a constant in the global name space and do not subsequently remove it (thus leading to an "already initialized constant" warning the second time): spec/unit/network/handler/fileserver.rb PLUGINS spec/unit/util/settings/file_setting.rb FileSetting spec/unit/transaction/event.rb Event spec/unit/transaction/change.rb Change -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
