Justin, yes it's happening in all environments which leads me to believe it's related to an old copy in /opt/puppetlabs/puppet/cache/lib/puppet/type. Still trying to wrap my head around why one domain installation is fine and the other domain installation is not.
Here is the contents of that directory which works: [root@myserverlab type]# pwd /opt/puppetlabs/puppet/cache/lib/puppet/type [root@myserverlab type]# ls -al total 24 drwxr-xr-x 2 root root 77 Jul 16 16:04 . drwxr-xr-x 6 root root 61 Feb 3 2020 .. -rw-r--r-- 1 root root 1706 Jul 16 16:04 anchor.rb -rw-r--r-- 1 root root 6921 Jul 16 16:04 file_line.rb -rw-r--r-- 1 root root 1863 May 1 2017 httparch.rb -rw-r--r-- 1 root root 6957 Jul 13 17:04 httpfile.rb [root@myserverlab type]# Here is the contents of the directory that doesn't work: [root@myserverprod type]# pwd /opt/puppetlabs/puppet/cache/lib/puppet/type [root@myserverprod type]# ls -al total 24 drwxr-xr-x 2 root root 77 Sep 30 2018 . drwxr-xr-x 6 root root 61 Apr 24 2017 .. -rw-r--r-- 1 root root 1752 Sep 30 2018 anchor.rb -rw-r--r-- 1 root root 7113 Sep 30 2018 file_line.rb -rw-r--r-- 1 root root 1863 May 15 2017 httparch.rb -rw-r--r-- 1 root root 6357 Apr 24 2017 httpfile.rb [root@myserverprod type]# You can clearly see the date and size difference of httpfile.rb. I quadruple checked the puppet module directory on the prod server and the code does have the quick_check parm. For some reason it is just not refreshing the server cache. Both domains have a value of 0 for environment_timeout for each environment. On Friday, August 28, 2020 at 2:32:05 PM UTC-4 Justin Stoller wrote: > On Fri, Aug 28, 2020 at 10:14 AM pkraw...@gmail.com <pkraw...@gmail.com> > wrote: > >> Great info but I think I might have found the issue. >> >> So we don't use r10k to deploy code we use a different tool. But what I >> found is on the puppet server (master) the httpfile.rb in >> /opt/puppetlabs/puppet/cache/lib/puppet/type is the older version. >> > > I think puppet/cache is read by the agent not the server. I would expect > that to cause problems on applying a catalog from the server, not result in > a failed compilation. But barring a .resource_tyeps directory existing in > an environment it must be an incorrect version of the httpfile.rb in the > server's loadpath. > > >> I didn't find any ./resource_types directory in our environment >> directories so not sure if we are using environment isolation or not. >> > > Just to clarify it will be ".resource_types" with a leading dot and will > be hidden by default. [1] > > As part of Justin's suggestion to allow the DELETE option to be valid, I >> had to restart each of our 4 puppet servers so according to some of this >> conversation, that should have refreshed the cache right? >> > > Restarting or reloading will evict the in memory cache so if you have a > very long environment_timeout it will work as well as doing an eviction of > all your environments. It will not however remove any old files in your > .resources_types directory. You will need to run the `puppet generate > types` command with `--force` for that. > > >> >> What else is odd is the domain where the quick_check parm is work seems >> to be getting refreshed somehow in /opt/puppetlabs/puppet/cache/lib/puppet >> subdirectories (just looking at time stamps). The deploy process works the >> same in that domain along with the domain where quick_check is not working. >> > > Can you validate that the failures happen not along a "domain" but along > puppet environments. Like all the nodes that use httpfile in production > have this failure but those in staging don't have this issue? If you have > some succeeding and some failing I would expect this to be the environment > to be the condition causing the different behavior. > > >> Since the /opt/puppetlabs/bin/puppet generate types --environment >> production --force operates by environment, could this possible break the >> environment as well? These are production boxes I need to run this on and >> want to make sure I don't break anything. Also using the environment parm >> will this then setup environment isolation and do i have to manually manage >> that each time code is deployed to that environment? >> > > The environment param to `puppet generate types` specifies which > environment to act on, without it the command will only act on the > environment specified in the puppet.conf for the "main" section (The "main" > or "user" sections are almost always unmanaged and adopt the default values > which would be "production" for "environment" setting). > > Running the command should be a relatively safe command, however I'm going > to advocate for anyone "doing it live" on a production box. In PE we deploy > this code and run this command in a staging area and then either lock the > server while we copy the files over or atomically manage a symlink. If you > are using environment caching as well it should be even safer because types > will only be read from disk on the first compilation that uses them and > then cached in memory after that. > > hth, > justin > > 1. > https://puppet.com/docs/puppet/6.17/environment_isolation.html#env_generate_types > > > On Tuesday, August 25, 2020 at 5:09:28 PM UTC-4 Justin Stoller wrote: >> >>> > why wouldn't puppet just do this automatically when a module changes? >>> >>> Some background. Puppet's type and provider system modifies the running >>> Puppet instance when they're _loaded_. This causes issues when you try to >>> load multiple conflicting versions of a type in different environments. To >>> work around this we have a kind of header file for your types that Puppet >>> can read w/o actually loading the type. This way Puppet Server can load >>> multiple versions of a type (as long as those different versions are in >>> different environments) and check that each environment uses the type >>> correctly for that version. >>> >>> The command Dirk gave you, loads those types safely in a separate >>> process and then serializes their parameter information into a format for >>> Puppet to later read that doesn't corrupt its global state. It places this >>> information in the ".resource_types" directory at the root of your >>> environment (like "/etc/puppetlabs/code/environments/production") >>> >>> Also, in order to speed up Puppet Server catalog compilation, we attempt >>> to cache information like type parameters. >>> >>> In PE, if you use our built in code management facilities, we generate >>> this type information on every commit (if needed), distribute it to your >>> compilers, and then evict the environment cache so that any new information >>> will be read. >>> >>> In FOSS, r10k has a config setting to generate this info when it deploys >>> an environment [1]. >>> >>> >>> Now, the error you're ultimately getting involves Puppet Server thinking >>> that you're using the httpfile class wrong. It looks like the "quick_check" >>> field was added in the latest version. So really the first question would >>> be, are you using the latest version in this environment? >>> >>> Assuming you're doing that you probably either have the environment >>> cache containing an older version of the module (which should be resolved >>> by restarting the server or evicting the environment cache) or an old >>> .resource_types in the root of your environment that should be removed and >>> regenerated like Dirk said. Possibly you could have an older version in a >>> different environment that's being loaded first, but I don't think that'd >>> cause a problem for uncached, new parameters on a type. >>> >>> HTH, >>> Justin >>> >>> 1. >>> https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#generate_types >>> >>> On Tue, Aug 25, 2020 at 9:42 AM pkraw...@gmail.com <pkraw...@gmail.com> >>> wrote: >>> >>>> So a followup to the original question. >>>> >>>> As a test we created a simple module on the node which is failing when >>>> puppet agent executes. Running puppet apply, the parameter quick_check is >>>> found and the module completes successfully. So why would puppet apply >>>> work and not puppet agent? >>>> >>>> Code: >>>> >>>> class testmod() >>>> >>>> { >>>> >>>> httpfile { "ansible-2.8.0a1.tar.gz": >>>> >>>> ensure => present, >>>> >>>> path => "/u01/testmod/ansible-2.8.0a1.tar.gz", >>>> >>>> source => " >>>> https://mynexus.domain.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz >>>> >>>> <https://nexus.aetna.com/nexus/repository/ae-raw-ansible-group/ansible/ansible-2.8.0a1.tar.gz> >>>> ", >>>> >>>> quick_check => true, >>>> >>>> # hash => 'hex form SHA2 hash OR an URL to the .sha file with that >>>> hash' >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> Here is my run: >>>> [root@node testmod]# puppet apply --modulepath=/home/toor --test -e >>>> "include testmod" --verbose >>>> >>>> On Tuesday, August 25, 2020 at 12:38:05 PM UTC-4 pkraw...@gmail.com >>>> wrote: >>>> >>>>> Dirk, why wouldn't puppet just do this automatically when a module >>>>> changes? Is there a bug somewhere? >>>>> >>>>> On Tuesday, August 25, 2020 at 2:43:03 AM UTC-4 Dirk Heinrichs wrote: >>>>> >>>>>> Am Montag, den 24.08.2020, 11:06 -0700 schrieb pkraw...@gmail.com: >>>>>> >>>>>> Justin, I implemented the suggestion you made however after running >>>>>> the curl command against the 2 environments having the issue and >>>>>> receiving >>>>>> the 204 response, the puppet module is still getting the 500 error. Do >>>>>> you >>>>>> or anyone else have any other suggestions? Is it possible it's related >>>>>> to >>>>>> ruby and/or java? Frankly I'm stumped. >>>>>> >>>>>> >>>>>> Didn't see this earlier, sorry. >>>>>> >>>>>> The "no parameter named 'xxx'" error can usually be resolved by >>>>>> recreating the metadata for your Puppet environment(s). This can be done >>>>>> on >>>>>> the Puppet master using the following command (for the production >>>>>> environment): >>>>>> >>>>>> /opt/puppetlabs/bin/puppet generate types --environment production >>>>>> --force >>>>>> >>>>>> I've added this command to my environment update script after running >>>>>> into this problem myself a few months ago after updating some external >>>>>> modules from the forge. >>>>>> >>>>>> See https://puppet.com/docs/puppet/5.5/environment_isolation.html for >>>>>> the details. >>>>>> >>>>>> HTH... >>>>>> >>>>>> Dirk >>>>>> >>>>>> -- >>>>>> >>>>>> *Dirk Heinrichs* >>>>>> Senior Systems Engineer, Delivery Pipeline >>>>>> OpenText ™ Discovery | Recommind >>>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>>> *Email*: dhei...@opentext.com >>>>>> *Website*: www.recommind.de >>>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>>> Registernummer HRB 10646 >>>>>> This e-mail may contain confidential and/or privileged information. >>>>>> If you are not the intended recipient (or have received this e-mail in >>>>>> error) please notify the sender immediately and destroy this e-mail. Any >>>>>> unauthorized copying, disclosure or distribution of the material in this >>>>>> e-mail is strictly forbidden >>>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>>> E-Mail >>>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und >>>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>>> >>>>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-users...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/puppet-users/80022945-ecdc-43ef-b857-ca2813c37919n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/puppet-users/9cbe27fa-f4a9-476b-8f88-6490bd83435an%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/93d1956d-afdc-4e75-b5cc-231c44f42771n%40googlegroups.com.
