Thanks Justin/Josh. Josh, I did come across this during my recent searches on trying to resolve my original issue.
On Tuesday, September 15, 2020 at 1:48:47 PM UTC-4 Josh Cooper wrote: > > > On Tue, Sep 15, 2020 at 5:25 AM [email protected] <[email protected]> > wrote: > >> So I've done some research on the puppet generate types command. I'm >> seeing many different results from not having issues to causing issues with >> puppet apply and puppet agent executions. > > > We fixed an issue in 6.18.0 ( > https://tickets.puppetlabs.com/browse/PUP-9602) that caused "puppet > apply" to fail in some cases if "puppet generate types" had been run > previously. > >> If I was to run this command and things go wrong, how do you reverse >> it? Remove the .resource_types directory? >> >> On Friday, August 28, 2020 at 2:47:26 PM UTC-4 [email protected] wrote: >> >>> Justin, yes it's happening in all environments which leads me to believe >>> it's related to an old copy >>> in /opt/puppetlabs/puppet/cache/lib/puppet/type. Still trying to wrap my >>> head around why one domain installation is fine and the other domain >>> installation is not. >>> >>> Here is the contents of that directory which works: >>> [root@myserverlab type]# pwd >>> /opt/puppetlabs/puppet/cache/lib/puppet/type >>> [root@myserverlab type]# ls -al >>> total 24 >>> drwxr-xr-x 2 root root 77 Jul 16 16:04 . >>> drwxr-xr-x 6 root root 61 Feb 3 2020 .. >>> -rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb >>> -rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb >>> -rw-r--r-- 1 root root 1863 May 1 2017 httparch.rb >>> -rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb >>> [root@myserverlab type]# >>> >>> Here is the contents of the directory that doesn't work: >>> [root@myserverprod type]# pwd >>> /opt/puppetlabs/puppet/cache/lib/puppet/type >>> [root@myserverprod type]# ls -al >>> total 24 >>> drwxr-xr-x 2 root root 77 Sep 30 2018 . >>> drwxr-xr-x 6 root root 61 Apr 24 2017 .. >>> -rw-r--r-- 1 root root 1752 Sep 30 2018 anchor.rb >>> -rw-r--r-- 1 root root 7113 Sep 30 2018 file_line.rb >>> -rw-r--r-- 1 root root 1863 May 15 2017 httparch.rb >>> -rw-r--r-- 1 root root 6357 Apr 24 2017 httpfile.rb >>> [root@myserverprod type]# >>> >>> You can clearly see the date and size difference of httpfile.rb. I >>> quadruple checked the puppet module directory on the prod server and the >>> code does have the quick_check parm. For some reason it is just not >>> refreshing the server cache. Both domains have a value of 0 for >>> environment_timeout for each environment. >>> >>> On Friday, August 28, 2020 at 2:32:05 PM UTC-4 Justin Stoller wrote: >>> >>>> On Fri, Aug 28, 2020 at 10:14 AM [email protected] <[email protected]> >>>> wrote: >>>> >>>>> Great info but I think I might have found the issue. >>>>> >>>>> So we don't use r10k to deploy code we use a different tool. But what >>>>> I found is on the puppet server (master) the httpfile.rb in >>>>> /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. >>>>> >>>> >>>> I think puppet/cache is read by the agent not the server. I would >>>> expect that to cause problems on applying a catalog from the server, not >>>> result in a failed compilation. But barring a .resource_tyeps directory >>>> existing in an environment it must be an incorrect version of the >>>> httpfile.rb in the server's loadpath. >>>> >>>> >>>>> I didn't find any ./resource_types directory in our environment >>>>> directories so not sure if we are using environment isolation or not. >>>>> >>>> >>>> Just to clarify it will be ".resource_types" with a leading dot and >>>> will be hidden by default. [1] >>>> >>>> As part of Justin's suggestion to allow the DELETE option to be >>>>> valid, I had to restart each of our 4 puppet servers so according to some >>>>> of this conversation, that should have refreshed the cache right? >>>>> >>>> >>>> Restarting or reloading will evict the in memory cache so if you have a >>>> very long environment_timeout it will work as well as doing an eviction of >>>> all your environments. It will not however remove any old files in your >>>> .resources_types directory. You will need to run the `puppet generate >>>> types` command with `--force` for that. >>>> >>>> >>>>> >>>>> What else is odd is the domain where the quick_check parm is work >>>>> seems to be getting refreshed somehow in >>>>> /opt/puppetlabs/puppet/cache/lib/puppet subdirectories (just looking at >>>>> time stamps). The deploy process works the same in that domain along >>>>> with >>>>> the domain where quick_check is not working. >>>>> >>>> >>>> Can you validate that the failures happen not along a "domain" but >>>> along puppet environments. Like all the nodes that use httpfile in >>>> production have this failure but those in staging don't have this issue? >>>> If >>>> you have some succeeding and some failing I would expect this to be the >>>> environment to be the condition causing the different behavior. >>>> >>>> >>>>> Since the /opt/puppetlabs/bin/puppet generate types --environment >>>>> production --force operates by environment, could this possible break the >>>>> environment as well? These are production boxes I need to run this on >>>>> and >>>>> want to make sure I don't break anything. Also using the environment >>>>> parm >>>>> will this then setup environment isolation and do i have to manually >>>>> manage >>>>> that each time code is deployed to that environment? >>>>> >>>> >>>> The environment param to `puppet generate types` specifies which >>>> environment to act on, without it the command will only act on the >>>> environment specified in the puppet.conf for the "main" section (The >>>> "main" >>>> or "user" sections are almost always unmanaged and adopt the default >>>> values >>>> which would be "production" for "environment" setting). >>>> >>>> Running the command should be a relatively safe command, however I'm >>>> going to advocate for anyone "doing it live" on a production box. In PE we >>>> deploy this code and run this command in a staging area and then either >>>> lock the server while we copy the files over or atomically manage a >>>> symlink. If you are using environment caching as well it should be even >>>> safer because types will only be read from disk on the first compilation >>>> that uses them and then cached in memory after that. >>>> >>>> hth, >>>> justin >>>> >>>> 1. >>>> https://puppet.com/docs/puppet/6.17/environment_isolation.html#env_generate_types >>>> >>>> >>>> On Tuesday, August 25, 2020 at 5:09:28 PM UTC-4 Justin Stoller wrote: >>>>> >>>>>> > why wouldn't puppet just do this automatically when a module >>>>>> changes? >>>>>> >>>>>> Some background. Puppet's type and provider system modifies the >>>>>> running Puppet instance when they're _loaded_. This causes issues when >>>>>> you >>>>>> try to load multiple conflicting versions of a type in different >>>>>> environments. To work around this we have a kind of header file for your >>>>>> types that Puppet can read w/o actually loading the type. This way >>>>>> Puppet >>>>>> Server can load multiple versions of a type (as long as those different >>>>>> versions are in different environments) and check that each environment >>>>>> uses the type correctly for that version. >>>>>> >>>>>> The command Dirk gave you, loads those types safely in a separate >>>>>> process and then serializes their parameter information into a format >>>>>> for >>>>>> Puppet to later read that doesn't corrupt its global state. It places >>>>>> this >>>>>> information in the ".resource_types" directory at the root of your >>>>>> environment (like "/etc/puppetlabs/code/environments/production") >>>>>> >>>>>> Also, in order to speed up Puppet Server catalog compilation, we >>>>>> attempt to cache information like type parameters. >>>>>> >>>>>> In PE, if you use our built in code management facilities, we >>>>>> generate this type information on every commit (if needed), distribute >>>>>> it >>>>>> to your compilers, and then evict the environment cache so that any new >>>>>> information will be read. >>>>>> >>>>>> In FOSS, r10k has a config setting to generate this info when it >>>>>> deploys an environment [1]. >>>>>> >>>>>> >>>>>> Now, the error you're ultimately getting involves Puppet Server >>>>>> thinking that you're using the httpfile class wrong. It looks like the >>>>>> "quick_check" field was added in the latest version. So really the first >>>>>> question would be, are you using the latest version in this environment? >>>>>> >>>>>> Assuming you're doing that you probably either have the environment >>>>>> cache containing an older version of the module (which should be >>>>>> resolved >>>>>> by restarting the server or evicting the environment cache) or an old >>>>>> .resource_types in the root of your environment that should be removed >>>>>> and >>>>>> regenerated like Dirk said. Possibly you could have an older version in >>>>>> a >>>>>> different environment that's being loaded first, but I don't think >>>>>> that'd >>>>>> cause a problem for uncached, new parameters on a type. >>>>>> >>>>>> HTH, >>>>>> Justin >>>>>> >>>>>> 1. >>>>>> https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#generate_types >>>>>> >>>>>> On Tue, Aug 25, 2020 at 9:42 AM [email protected] < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> So a followup to the original question. >>>>>>> >>>>>>> As a test we created a simple module on the node which is failing >>>>>>> when puppet agent executes. Running puppet apply, the parameter >>>>>>> quick_check is found and the module completes successfully. So why >>>>>>> would >>>>>>> puppet apply work and not puppet agent? >>>>>>> >>>>>>> Code: >>>>>>> >>>>>>> class testmod() >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> { >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> httpfile >>>>>>> >>>>>>> { "ansible-2.8.0a1.tar.gz": >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ensure => present, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> path => >>>>>>> >>>>>>> "/u01/testmod/ansible-2.8.0a1.tar.gz", >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> source => " >>>>>>> https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz >>>>>>> >>>>>>> <https://nexus.aetna.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz> >>>>>>> ", >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> quick_check => true, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> # >>>>>>> >>>>>>> hash => 'hex form SHA2 hash OR an URL to the .sha file >>>>>>> >>>>>>> with that hash' >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is my run: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> [root@node testmod]# puppet apply --modulepath=/home/toor --test -e >>>>>>> "include >>>>>>> >>>>>>> testmod" --verbose >>>>>>> >>>>>>> On Tuesday, August 25, 2020 at 12:38:05 PM UTC-4 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> Dirk, why wouldn't puppet just do this automatically when a module >>>>>>>> changes? Is there a bug somewhere? >>>>>>>> >>>>>>>> On Tuesday, August 25, 2020 at 2:43:03 AM UTC-4 Dirk Heinrichs >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Am Montag, den 24.08.2020, 11:06 -0700 schrieb [email protected]: >>>>>>>>> >>>>>>>>> Justin, I implemented the suggestion you made however after >>>>>>>>> running the curl command against the 2 environments having the issue >>>>>>>>> and >>>>>>>>> receiving the 204 response, the puppet module is still getting the >>>>>>>>> 500 >>>>>>>>> error. Do you or anyone else have any other suggestions? Is it >>>>>>>>> possible >>>>>>>>> it's related to ruby and/or java? Frankly I'm stumped. >>>>>>>>> >>>>>>>>> >>>>>>>>> Didn't see this earlier, sorry. >>>>>>>>> >>>>>>>>> The "no parameter named 'xxx'" error can usually be resolved by >>>>>>>>> recreating the metadata for your Puppet environment(s). This can be >>>>>>>>> done on >>>>>>>>> the Puppet master using the following command (for the production >>>>>>>>> environment): >>>>>>>>> >>>>>>>>> /opt/puppetlabs/bin/puppet generate types --environment production >>>>>>>>> --force >>>>>>>>> >>>>>>>>> I've added this command to my environment update script after >>>>>>>>> running into this problem myself a few months ago after updating some >>>>>>>>> external modules from the forge. >>>>>>>>> >>>>>>>>> See https://puppet.com/docs/puppet/5.5/environment_isolation.html for >>>>>>>>> the details. >>>>>>>>> >>>>>>>>> HTH... >>>>>>>>> >>>>>>>>> Dirk >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *Dirk Heinrichs* >>>>>>>>> Senior Systems Engineer, Delivery Pipeline >>>>>>>>> OpenText ™ Discovery | Recommind >>>>>>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>>>>>> *Email*: [email protected] >>>>>>>>> *Website*: www.recommind.de >>>>>>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>>>>>> <https://www.google.com/maps/search/Von-Liebig-Stra%C3%9Fe+1,+53359+Rheinbach?entry=gmail&source=g> >>>>>>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>>>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>>>>>> Registernummer HRB 10646 >>>>>>>>> This e-mail may contain confidential and/or privileged >>>>>>>>> information. If you are not the intended recipient (or have received >>>>>>>>> this >>>>>>>>> e-mail in error) please notify the sender immediately and destroy >>>>>>>>> this >>>>>>>>> e-mail. Any unauthorized copying, disclosure or distribution of the >>>>>>>>> material in this e-mail is strictly forbidden >>>>>>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>>>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>>>>>> E-Mail >>>>>>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender >>>>>>>>> und >>>>>>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die >>>>>>>>> unbefugte >>>>>>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>> >>>>>>> >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Puppet Users" group. >>>>>>> >>>>>>> >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>> >>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Users" group. >>>>> >>>>> >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> >>>> >> >> >> >> >> >> >> >> -- >> >> >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> >> >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/10f72a8f-b9b3-4686-ab6d-5e39a252f339n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/puppet-users/10f72a8f-b9b3-4686-ab6d-5e39a252f339n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/9abd2472-4511-413b-b72f-da492d9d9560n%40googlegroups.com.
