Jira (PUP-4659) environment_scenario test failures when master has agent role

[Jeremy Barlow (JIRA)]

Title: Message Title

Jeremy Barlow created an issue   Puppet / PUP-4659 environment_scenario test failures when master has agent role Issue Type: Bug Affects Versions: PUP 4.1.0 Assignee: Kylo Ginsberg Components: Client Created: 2015/05/27 12:22 PM Priority: Normal Reporter: Jeremy Barlow When we upgraded the Puppet Server acceptance pipeline to using Puppet 4.1.0 / Puppet Agent 1.1.0, we started encountering some failures with the environment_scenario tests. I believe these tests are not failing when run on the platform pipeline only because some of the test steps are only run when the agent under test resides on the master. According to Josh Cooper, the master has not been run with the agent role in the platform AIO pipeline for some time. An example of one failing matrix is here: https://jenkins.puppetlabs.com/view/Puppet%20Server/view/All/job/platform_puppet-server_integration-system_no-conditional_full-master/64/#showFailuresLink Note that the same failures are occurring across all OSes under test. Here's a quick analysis of what it looks to me that may be going on with the failures: environment_scenario-bad.rb Same pattern like the one below for “puppet_config”, “puppet_apply”, “puppet_module_install”, and “puppet_module_uninstall”: ------------------------------------------------------------------------------- puppet_config: *UNEXPECTED FAILURE* ------------------------------------------------------------------------------- Expected the output: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'testing' anywhere in the path: /doesnotexist. Does the directory exist? (Puppet::Environments::EnvironmentNotFound) from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:29:in `push_application_context' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:337:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:124:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute' from /opt/puppetlabs/bin/puppet:5:in `<main>'   To match: (?-mix:Could not find a directory environment named 'testing' anywhere in the path.*\/etc\/puppetlabs\/code) The test sets the “environmentpath” in the “main” section of the puppet.conf file to “/doesnotexist” but the test appears to be failing because it expects the master’s “codedir” setting to appear in the message - “/etc/puppetlabs/code” - rather than the “environmentpath”. The “environmentpath" actually seems to me like the right value to include in the message here. I can reproduce this locally by just running “puppet config … —environment testing” (with no master involved). environment_scenario-default.rb Same pattern like the one below for “puppet_config”, “puppet_module_install”, and “puppet_module_uninstall”. ------------------------------------------------------------------------------- puppet_config: *UNEXPECTED FAILURE* ------------------------------------------------------------------------------- Expected the output: basemodulepath = /etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules modulepath = /etc/puppetlabs/code/environments/production/modules:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules manifest = /etc/puppetlabs/code/environments/production/manifests config_version =   To match: (?-mix:manifest.*\/etc\/puppetlabs\/code\/environments\/\/manifests$) And match: (?-mix:modulepath.*\/etc\/puppetlabs\/code\/environments\/\/modules:.+) In each case, it appears that the test is expecting the target directory to not include a directory environment name where the result of the test does - i.e., “production”. Where directory environments are mandated by default in Puppet v4, I wonder if this is just a case where the default expectation needs to include the “production” directory environment name. environment_scenario-master_environmentpath.rb For the “puppet_apply” test, the following output appeared: ------------------------------------------------------------------------------- puppet_apply: *UNEXPECTED FAILURE* ------------------------------------------------------------------------------- Expected the output: Notice: Compiled catalog for fd7ncmpi1suilmb.delivery.puppetlabs.net in environment testing in 0.33 seconds Notice: include directory testing environment testing_mod Notice: /Stage[main]/Testing_mod/Notify[include directory testing environment testing_mod]/message: defined 'message' as 'include directory testing environment testing_mod' Notice: Applied catalog in 0.08 seconds   To match: (?-mix:include default environment testing_mod) ------ It seems like the expectation that the output would use “default environment” is incorrect since the test is actually trying to use the “testing” directory environment. — For the “puppet_module_install” test, the following output appeared: ------------------------------------------------------------------------------- puppet_module_install: *UNEXPECTED FAILURE* ------------------------------------------------------------------------------- Expected the output: Notice: Preparing to install into /etc/puppetlabs/code/environments/testing/modules ... Notice: Downloading from https://forgeapi.puppetlabs.com ... Notice: Installing -- do not interrupt ... /etc/puppetlabs/code/environments/testing/modules └── pmtacceptance-nginx (v0.0.1)   To match: (?-mix:Preparing to install into \/etc\/puppetlabs\/code\/modules) ------ Notes: Runs in user mode and doesn't see the master environmenetpath setting. Again, the expectation that the message does not include “…environments/testing” in the path seems wrong since the test is specifying that the “testing” directory environment be used. — For the “puppet_module_uninstall” test, the following output appeared: ------------------------------------------------------------------------------- puppet_module_uninstall: *UNEXPECTED FAILURE* ------------------------------------------------------------------------------- Expected the output: Notice: Preparing to uninstall 'pmtacceptance-nginx' ... Removed 'pmtacceptance-nginx' (v0.0.1) from /etc/puppetlabs/code/environments/testing/modules   To match: (?-mix:Removed.*pmtacceptance-nginx.*from \/etc\/puppetlabs\/code\/modules) ------ Same comment as above for the “puppet_module_install” test. all tests It appears that across all of these tests that there’s a problem in the test cleanup where the occurrence of a failure is causing the test to abort with a stack trace: #<NoMethodError: undefined method `empty?' for nil:NilClass> /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/ruby/puppet/acceptance/lib/puppet/acceptance/environment_utils.rb:355 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/ruby/puppet/acceptance/lib/puppet/acceptance/environment_utils.rb:354 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/ruby/puppet/acceptance/lib/puppet/acceptance/environment_utils.rb:354 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/ruby/puppet/acceptance/tests/environment/environment_scenario-bad.rb:53 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_case.rb:124 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_case.rb:124 /usr/local/rvm/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/benchmark.rb:295 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_case.rb:121 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_suite.rb:286 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_suite.rb:283 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_suite.rb:283 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/test_suite.rb:325 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/cli.rb:151 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/lib/beaker/cli.rb:93 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/gems/beaker-2.12.0/bin/beaker:6 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/bin/beaker:23 /var/lib/jenkins/workspace/platform_puppet-server_integration-system_no-conditional_full-master/LAYOUT/redhat6-64ma-64a/LDAP_TYPE/PLATFORM/label/beaker/vendor/bundler/ruby/1.9.1/bin/beaker:23 Unstubbing address forge.puppetlabs.com to IP 10.32.23.69 on machine fd7ncmpi1suilmb.delivery.puppetlabs.net Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.