On Thursday, October 20, 2016 at 10:31:59 AM UTC-5, jcbollinger wrote:
>
>
>
> On Wednesday, October 19, 2016 at 4:51:20 PM UTC-5, re-g...@wiu.edu wrote:
>>
>>
>>
>> On Tuesday, October 18, 2016 at 9:49:23 AM UTC-5, jcbollinger wrote:
>>>
>>>
>>>
>>> On Monday, October 17, 2016 at 3:15:51 PM UTC-5, re-g...@wiu.edu wrote:
>>>>
>>>>
>>>>
>>>> On Friday, October 14, 2016 at 5:08:31 PM UTC-5, re-g...@wiu.edu wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Thursday, October 13, 2016 at 9:15:58 AM UTC-5, Rob Nelson wrote:
>>>>>>
>>>>>> I've worked with saz in the past and believe he would be very
>>>>>> receptive to PRs that address this issue, as well, if RG or anyone else
>>>>>> wanted to contribute them. That would be one of the better solutions to
>>>>>> the
>>>>>> problem.
>>>>>>
>>>>>> Rob Nelson
>>>>>> rnel...@gmail.com
>>>>>>
>>>>>>
>>>>>> I think I've given you a pretty good handle on the nature of the
>>>>>>> problem, and I'm inclined to think that a solution that addresses the
>>>>>>> problem at its root will take care of the whole suite of issues.
>>>>>>>
>>>>>>
>>>>> Lots and lots of thanks to jcbollinger.
>>>>> Here is what I have done as a result of this thread:
>>>>>
>>>>> Submitted an issue with saz-rsyslog:
>>>>> Duplicate Declaration Class[Rsyslog] with The Foreman as ENC during
>>>>> catalog compilation #237
>>>>> https://github.com/saz/puppet-rsyslog/issues/237
>>>>>
>>>>> Posted a follow-up question to a related thread on The Foreman mail
>>>>> list to determine if The Foreman also may be exhibiting a bug, or if the
>>>>> issue may be my configuration:
>>>>> Duplicate declaration error.
>>>>> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion
>>>>> (my post is waiting for approval as of 22:00 GMT 10/14/2016)
>>>>>
>>>>> My current thought at this time is two points:
>>>>> 1. saz-rsyslog is written in such a way that declaring a subclass
>>>>> before the parent class causes duplicate declaration errors - but I am
>>>>> told
>>>>> the module inheritance should be addressed, and would resolve this issue.
>>>>> 2. The Foreman, under undetermined circumstances (maybe my
>>>>> misconfiguration), will randomly declare parent classes and subclass out
>>>>> of
>>>>> order - but I am told this is not an issue with puppet modules that use
>>>>> inheritance correctly - thus there might be an underlying bug with The
>>>>> Foreman because this error will not appear under modules' expected
>>>>> inheritance.
>>>>>
>>>>>
>>>> This is what was posted on The Foreman mail list group:
>>>> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion
>>>>
>>>>> This is an issue with the format of the ENC YAML used between Foreman
>>>>> and Puppet, and is best fixed in the module.
>>>>> The list of classes is given as a hash/dictionary and so has no
>>>>> particular order defined - it's down to the Puppet master/server to
>>>>> iterate over it, and it probably does so in no particular order. It
>>>>> isn't under Foreman's control.
>>>>
>>>>
>>>> I now believe this is truly the root of my problem.
>>>>
>>>> The evidence is looking at the ENC output and the Puppet-generated
>>>> Catalog file on each of the master puppet servers.
>>>> As you can see below, the puppet-generated catalog on the
>>>> secondary-master puppet server has put rsyslog::client before rsyslog -
>>>> this causes the failure.
>>>> However, The Foreman ENC output is the same on both primary and
>>>> secondary master puppet servers.
>>>>
>>>>
>>>
>>> You have taken a step backward there. I remind you that duplicate
>>> declaration errors occur *during catalog building*, and that catalog
>>> building fails if one occurs. It is therefore impossible to compare a
>>> catalog from the failure case to a catalog from the success case, because
>>> no catalog is ever created in the failure case.
>>>
>>>
>> Then I was mistaken in thinking that the node file
>> (/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml) was the data
>> (catalog pre-data) written to a file before the catalog compilation
>> procedure.
>>
>
>
> But you said you were looking at the *catalog* not "catalog pre-data".
> I'm sure you appreciate the tremendous difference between those.
>
>
>
>> I assumed this because both the foreman and node files are created new
>> every time server1 performs a puppet agent update, even when the result is
>> a 400 error for the catalog compilation.
>> If I am mistaken in this, then I need someone to tell me how to get the
>> data of the catalog that is failing to be compiled, so that I can compare
>> it with other sources.
>>
>> Here is the procedure I have been assuming (I am not an expert on puppet
>> internals):
>> 1. (server1)puppet-agent -> puppet-master
>> 2. puppet-master -> (exec)puppet[external_nodes]
>> server1.mydomain.example.com -> foreman ENC
>> 3. foreman ENC ->
>> /var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml ->
>> puppet-master
>> 4. puppet-master
>> -> /var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml ->
>> puppet-master(compile-catalog)
>> 5. puppet-master(compile-catalog) -> Error -> 400 Server Error ->
>> (server1)puppet-agent
>>
>> I was assuming the node file is the data of the catalog to be compiled. I
>> had noticed it is updated in every puppet agent run, even when the run is
>> in error. And it was the differing piece between primary and secondary
>> puppet masters.
>> Please correct me.
>>
>
>
> The YAML files in question are just byproducts of catalog building, not,
> as I understand it, intermediate files in the catalog-building process. In
> any event, an ENC certainly delivers its results to Puppet via its standard
> output, not by writing them to a regular file. Nor would I expect any ENC
> to create or update files under /var/lib/puppet in any case.
>
>
>
>> Please let me know how to get the data of the catalog the puppet master
>> is failing to compile for the puppet agent.
>>
>
>
> If you are not assigning any classes or resources directly via a site
> manifest, and you are not using Hiera or any other form of external data
> besides variables assigned via Foreman, then the facts reported by the
> node, the ENC output for that node, and the master's standard top-scope
> variables together constitute the data from which catalog building
> proceeds. It is possible, however, for evaluation of specific manifests to
> result in arbitrary data being gleaned from the catalog-building
> environment by use of Puppet functions, such as generate(), which can run
> an external command, and template() / inline_template(), which can run
> arbitrary Ruby code.
>
> You can get the latest facts reported by a given node via the 'puppet
> facts find <node_certname>' command on the master. If you're using
> puppetdb, then you can get them from there, too.
>
>
Between primary and secondary masters, this information is the same, in the
same order, and formatted the same (including spaces).
1. the facts reported by the node
2. ENC output for that node
3. master's standard top-scope variables
I only have the following set in the top scope, on all masters:
File: /etc/puppet/environments/production/manifests/site.pp
Package { allow_virtual => false }
puppet 'environments' directory is synced from primary to all secondary
puppet master servers.
>
>>
>>
>>>
>>>
>>>> Is this correct, that the yaml file in /var/lib/puppet/yaml/node/ is
>>>> exclusively generated by Puppet based on the ENC yaml?
>>>> If this is correct, then this is the root of the problem.
>>>>
>>>
>>>
>>> I'm not sure what significance you're attributing to those particular
>>> files, but it's probably the wrong significance. Yes, to the best of my
>>> knowledge and understanding, Puppet generates those files based on the
>>> node's certificate and facts, and the associated ENC output (if any). But
>>> no, there is no reason to think that the root of your problem lies here.
>>>
>>>
>> As I said earlier, I now assume I misunderstood this as being the file
>> holding the data to be compiled as the catalog.
>> I apologize for the misunderstanding, but that I intended it as the
>> "root" of the catalog-compilation "problem". You are right that it is not
>> the root of the entire underlying problem.
>> Again, I need help to find the catalog data trying to be compiled, but
>> which fails, if this /.../node/.. file is not that data.
>>
>
>
> The starting point for classes to be applied is the combination of ENC
> output with the site manifest. I cannot speak to what may be in your site
> manifest. What file or files constitute your site manifest depends on your
> Puppet configuration. Some of the more likely possibilities are '<puppet
> root>/manifests/site.pp', '<puppet
> root>/environments/production/manifests/site.pp', or jointly all files of
> the form '<puppet root>/environments/production/manifests/*.pp'. Within
> the site manifest, declarations at top scope and within the node block best
> matching the target machine (if any match at all) are relevant.
>
>
/etc/puppet/manifests/site.pp - empty
/etc/puppet/environments/production/manifests/site.pp - content provided
earlier, and is same on all secondary puppet masters
There are no other .../manifests/*.pp files
>
>>
>>>
>>>> Is there a configuration issue causing this issue? Where would I begin
>>>> looking for something that may be causing this puppet-catalog problem?
>>>>
>>>
>>>
>>> There is an unsupported-use-of-module issue ultimately causing all your
>>> problems, as I have been saying since the beginning. It would be best to
>>> solve *that* problem, either by choosing a different module or by
>>> restricting yourself to supported uses of the module you have chosen (e.g.
>>> by using automated data binding to assign values to class parameters).
>>>
>>>
>> "restricting yourself to supported uses of the module"
>> So using The Foreman as ENC has not been an official tested use of the
>> module and thus is not supported? This is true for all modules on forge I
>> have reviewed, except for the ones published by The Foreman group.
>> To be supported, should I instead restrict my use of either declaring my
>> classes in a site.pp or use hiera?
>>
>
>
> Puppet supports declaring classes via an ENC generally. Inasmuch as its
> implementation violates the DSL's documented constraints, however, the '
> rsyslog::client' class is not supported at all in the sense of the term I
> was using.
>
> Since avoiding that class is not a viable solution (unless by choosing an
> altogether different module), the next best thing would be to stick to uses
> that are well known to work despite not actually being supported -- in this
> case, that would be to avoid using (the ENC analog of) a resource-like
> declaration for the base class, rsyslog. You have exactly one option for
> customizing that class's parameters without modifying it or using a
> resource-like declaration: automated data binding.
>
>
>
>>
>> "using automated data binding to assign values to class parameters"
>> What is automated data binding? Is that what I get when using hiera?
>>
>
>
> It is documented here:
> https://docs.puppet.com/hiera/3.2/puppet.html#automatic-parameter-lookup.
> It is based on Hiera, but it would not be quite correct to say it's "what I
> get when using hiera", because Hiera supports other uses as well.
>
>
>
>> Would that mean I would forgo the use of The Forman to instead use hiera?
>>
>
>
> It would mean that you forgo using The Foreman for setting the parameters
> of class rsyslog, at least for those nodes to which you also assign
> another class from the same module, such as rsyslog::client. You should
> otherwise be able to continue using The Foreman as you currently do.
>
>
Okay, I understand your recommendation.
To continue using this rsyslog module, I should use another method (that is
supported, like Hiera) other than The Foreman to configure this class.
Unfortunately this is not ideal, since that makes two places to maintain
configuration.
It would be better to find a replacement for the saz-rsyslog module, which
you have previously recommended.
>
>
>> I have been assuming hiera was relatively the same as using The Foreman
>> as ENC? Reference:
>> https://ask.puppet.com/question/527/are-hiera-and-foreman-parameters-mutually-exclusive/
>>
>> <https://www.google.com/url?q=https%3A%2F%2Fask.puppet.com%2Fquestion%2F527%2Fare-hiera-and-foreman-parameters-mutually-exclusive%2F&sa=D&sntz=1&usg=AFQjCNFi7mdJ0ZSw80VSK1NGgNAFWZuVTA>
>>
>>
>
>
> No, they are not the same thing at all, though they can serve many of the
> same purposes (and the answer you linked says pretty much the same thing).
> Hiera is not an ENC, and Puppet does not use it in the same way or via the
> same interface that it uses an ENC. Binding values to class parameters via
> (hiera-supported) automatic data binding has different semantics than does
> binding parameter values via an ENC. From a Foreman perspective, you might
> view automated data binding as affecting the default values for class
> parameters.
>
>
Got it. That is clear.
>
>
>> I am grasping at straws. I have been assuming what I can find different
>> between the servers would reveal the issue.
>>
>
>
> Indeed, the problems are mysterious, and it is entirely reasonable to
> suppose that differing behavior arises from discoverable differences
> between machines. Myself, I had high hopes for an explanation in terms of
> differing versions of Ruby. I guess there's still a chance that a
> difference in the installed Ruby modules might factor in. And I wasn't
> kidding when I suggested replacing the machine on which the misbehavior is
> observed with a clone of one of those that behave as you want.
>
I combed through the servers. They are exactly the same down to the
installed RPM packages. They were auto installed with The Foreman, via a
kickstart. So their origins are the same.
However, I installed and configured The Foreman and secondary Puppet
masters manually.
All Puppet configuration is maintained by puppet itself.
> However, I also hold out some suspicion that the problem may not be in the
> machine at all, but rather in the inputs (ENC output varying over time;
> differences in other parts of the ENC output; node facts). Or the
> difference may be in the dynamic operating environment, including Puppet
> runtime state. I'm not sure how to test that directly, but it might be
> interesting to swap masters to see whether the problem sticks with the
> machine or with the workload.
>
>
I had pointed a client at the primary master, and it runs successfully.
Only on the secondary puppet masters I have issues.
>
>> I guess I need someone to tell me that if a puppet module is having the
>> issue we describe in this thread, that due to the intended and expected
>> nature of some function/process "X" in The Foreman or Puppet, I can expect
>> to see the module's inheritance issue cause catalog-compilation errors at
>> random times.
>>
>>
>
>> Example: it was suggested that the hash keys are not always ordered the
>> same. So due to python sometimes ordering the hash keys differently, a
>> module with inheritance issues like saz-rsyslog will sometimes cause a
>> catalog compilation error, and sometimes not be an issue. This means Puppet
>> expects modules to not have an inheritance issue like we are describing in
>> order to guarantee there will not be issues during deployment, like in my
>> case.
>>
>
>
> Well, the problem with hashes is that hash iteration order is not
> necessarily predictable in advance. However, given the same hash
> implementation, initialized the same way, and subjected to the same
> sequence of additions and removals, you should see the same iteration order
> on every trial. On the other hand, if the sequence of additions and
> removals changes, and especially if more, fewer, or different keys are
> added in different trials, then you cannot rely on iteration order to be
> consistent, even in part. That's one reason why I earlier pointed out that
> the problem does not have to be in the portion of the ENC output that
> pertains directly to the rsyslog classes.
>
> So no, I cannot tell you that with everything remaining the same, you
> should expect that Puppet's catalog building behavior might vary from run
> to run. Quite the opposite. But on the other hand, you can never have
> *everything* exactly the same.
>
> I sympathize with you in your difficulty discovering the specific
> difference(s) that explain the variation in outcome, and some of my
> suggestions have been aimed at helping you there. But you should not
> assume that figuring it out will lead you to a solution you like better
> than those I have proposed. It might do. It also might not. And I cannot
> evaluate for you whether it's worthwhile to keep looking.
>
>
> John
>
>
After taking a look at the rsyslog puppet module code, it would seem saz
expects either the rsyslog::client or rsyslog::server to be declared alone.
Then inherit the parent rsyslog class so that "top-level" parameters (I use
that generically) can be overwritten in your node scode.
Unfortunately, The Foreman see every class and subclasses independently,
and lets you configure the parameters of each.
So, when The Foreman ENC sends a declaration for Rsyslog to Puppet, it is
already wrong according to the module's expected use. But (like you were
saying) the compilation will not necessarily fail if it so happens that
Rsyslog is declared before Rsyslog::client in the puppet agent catalog -
which is not necessarily dependent upon the ENC output which may have them
declared in the correct order, but dependent upon puppet master's discovery
of class declaration in the internal ruby hash created from ENC output
(e.g. the order of hash keys) and the subsequent ordering of those classes
during catalog compilation.
As best as I am able to figure out, the primary and secondary puppet
servers receive the same ENC YAML data. So, somewhere after that YAML is
hashed inside puppet master, this issue described with the saz-rsyslog
module will cause the catalog compilation error should the Rsyslog::client
class be declared before Rsyslog.
Being it might seem saz-rsyslog will stay as it is, it would seem my best
course of action is to use an alternate rsyslog module if I wish to
continue using The Foreman.
Thank you for all of your help.
-RG
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/f57e26fd-953c-4be4-9605-714abafc796e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
