Jira (PUP-10942) Listing environments during code deploys can prevent environment cache invalidation

[Josh Cooper (Jira)]

Title: Message Title

Josh Cooper created an issue   Puppet / PUP-10942 Listing environments during code deploys can prevent environment cache invalidation Issue Type: Bug Assignee: Unassigned Created: 2021/02/25 1:51 PM Priority: Normal Reporter: Josh Cooper If environments are listed via the environment_classes REST API while a new environment directory is deployed, it's possible for the default modulepath to be cached in settings, and cause future compilations with that environment to fail. It's can also cause agents to randomly delete pluginsync'ed files (due to omissions in the file metadata response). On the puppetserver node, allow access to the REST API to listing and delete environments: { "allow": "ADD SERVER FQDN", "match-request": { "method": "delete", "path": "/puppet-admin-api/v1/environment-cache", "query-params": {}, "type": "path" }, "name": "puppetlabs environment cache", "sort-order": 500 }, { "allow": "ADD SERVER FQDN", "match-request": { "method": "get", "path": "/puppet/v3/environment_classes", "query-params": {}, "type": "path" }, "name": "puppetlabs environment classes", "sort-order": 500 }, Then run: #!/bin/sh   server="ADD SERVER NAME HERE" certname="${server}"   # use server cert for privileged REST APIs curl_opts="-kf --cert /etc/puppetlabs/puppet/ssl/certs/${server}.pem --key /etc/puppetlabs/puppet/ssl/private_keys/${server}.pem"   api_environments="https://${server}:8140/puppet/v3/environments" api_classes="https://${server}:8140/puppet/v3/environment_classes" api_environment_cache="https://${server}:8140/puppet-admin-api/v1/environment-cache" api_catalog="https://${server}:8140/puppet/v3/catalog" api_catalog4="https://${server}:8140/puppet/v4/catalog"   name="env_${RANDOM}" basedir="/etc/puppetlabs/code/environments" envdir="${basedir}/${name}" mkdir -p "${envdir}" chown puppet:puppet "${envdir}" echo "create random environment: ${name}"   echo "list environments, cache modulepath" curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name}   echo "finish deployment" envconf="${envdir}/environment.conf" cat <<EOF > "${envconf}" modulepath=site:dist:\$basemodulepath EOF   mkdir "${envdir}/manifests" cat <<EOF > "${envdir}/manifests/init.pp" include role EOF   mkdir "${envdir}/dist" mkdir -p "${envdir}/site/role/manifests" cat <<EOF > "${envdir}/site/role/manifests/init.pp" class role { notify { 'in role': } } EOF chown -R puppet:puppet "${envdir}"   echo mark environments as expired curl ${curl_opts} -X DELETE -H 'Accept: application/json' ${api_environment_cache}   echo we see the modulepath from before the deploy finished curl ${curl_opts} -X GET -H 'Accept: application/json' ${api_environments} | jq .environments.${name}   echo compile v3 catalog fails curl -kf \ --cert "/etc/puppetlabs/puppet/ssl/certs/${certname}.pem" \ --key "/etc/puppetlabs/puppet/ssl/private_keys/${certname}.pem" \ -X POST \ -d "%257B%2522facts_format%2522%253A%2522application%252Fjson%2522%252C%2522facts%2522%253A%257B%2522name%2522%253A%2522${certname}%2522%257D%257D" \ -H 'Content-Type: application/x-www-form-urlencoded' \ -H 'Accept: application/json' \ ${api_catalog}/${certname}\?environment\=${name} Due to the bug, we expire all environments, but still see the cached environment's modulepath and compilation fails: ... mark environments as expired we see the modulepath from before the deploy finished % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6900 100 6900 0 0 87341 0 --:--:-- --:--:-- --:--:-- 87341 { "settings": { "modulepath": [ "/etc/puppetlabs/code/environments/env_1077/modules", "/etc/puppetlabs/code/modules", "/opt/puppetlabs/puppet/modules" ], "manifest": "/etc/puppetlabs/code/environments/env_1077/manifests", "environment_timeout": 0, "config_version": "" } } compile v3 catalog fails curl: (22) The requested URL returned error: 500 Server Error And puppetserver.log contains: 2021-02-25T20:56:08.058Z ERROR [qtp1507614676-35] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::role for afferent-trot.delivery.puppetlabs.net (file: /etc/puppetlabs/code/environments/env_32225/manifests/init.pp, line: 1, column: 1) on node afferent-trot.delivery.puppetlabs.net /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:401:in `block in evaluate_classes' org/jruby/RubyArray.java:2572:in `collect' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:400:in `evaluate_classes' Add Comment   This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.388981.1614289885000.152502.1614289920096%40Atlassian.JIRA.