Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele commented on PUP-10491 Re: Allow adding descriptive or administrative information to resources Noting for the future: Given that this is fairly trivial to implement (PR is already up), the main consideration for the future is really just what to call it, and what use cases motivate it. If we're waiting for other Puppet products to drive the motivation to decide on a name and merge the PR, I expect that if it comes, it will come from HDP. HDP's analytics and reporting platform is the potential Puppet product benefactor of richer annotations on resources describing their business purpose. If I were confident on the naming I would advocate for merging as-is, but without naming confidence I won't take that position yet. When we're ready, this should be fairly trivial to accomplish. Add Comment This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.357687.1588699805000.24357.1645812540106%40Atlassian.JIRA.
Jira (PUP-11365) Prevent thundering herd via deterministic run scheduling
Title: Message Title Reid Vandewiele commented on PUP-11365 Re: Prevent thundering herd via deterministic run scheduling Ciprian Badescu if I'm understanding it correctly, the idea of step 3 collision handling above violates the first non-volatility characteristic given in the description: Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times Receiving a 503 is a variable-time event. It is permissible for receipt of a 503 to delay the start time of a single run event, but it cannot be permitted to shift the schedule. A single run can be delayed up to a point, but If the single run cannot be started within its allocated portion of the schedule, it should be cancelled. The next scheduled run should still occur at the next non-volatile pre-computed run-interval offset. Permitting the normal schedule to be shifted or changed by variable events is what permits Thundering Herds to form. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.423928.1637151634000.9597.1637167800032%40Atlassian.JIRA.
Jira (PDB-5343) PQL dotted fields don't support double-quoted path components
Title: Message Title
Reid Vandewiele updated an issue
PuppetDB / PDB-5343
PQL dotted fields don't support double-quoted path components
Change By:
Reid Vandewiele
PuppetDB documents its use of "dotted field[s], which represents a path into a JSON tree" [here|https://puppet.com/docs/puppetdb/7/api/query/v4/ast.html#dotted-field-syntax]. Notably, "Any path component can be double-quoted" in order to accommodate specification of path components which themselves contain dot characters.When writing PQL queries, however, such as the following, it is currently impossible to use this documented dotted field syntax. {code} inventory[certname, facts."my.fact.has.dots"] { LIMIT 1 }{code} The following error will be returned. {code} 2021/10/25 05:37:14 ERROR - [GET /pdb/query/v4][400] getQueryBadRequest PQL parse error at line 1, column 27:inventory[certname, facts."my.fact.has.dots"] { } ^Expected one of:#"match\(.*?\)"#"[^\s\.\",\[\]!<=>~]+(\[\d+\])?"#"\".*?\"(?=\.|\s)"{code} h3. Expected outcomeThe PQL given above *should* be equivalent to this AST query. {code} ["from","inventory",["extract", ["certname", "facts.\"my.fact.has.dots\""]]]{code}
Add Comment
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Jira (PDB-5343) PQL dotted fields don't support double-quoted path components
Title: Message Title
Reid Vandewiele created an issue
PuppetDB / PDB-5343
PQL dotted fields don't support double-quoted path components
Issue Type:
Bug
Assignee:
Unassigned
Created:
2021/10/24 10:46 PM
Priority:
Normal
Reporter:
Reid Vandewiele
PuppetDB documents its use of "dotted field[s], which represents a path into a JSON tree" here. Notably, "Any path component can be double-quoted" in order to accommodate specification of path components which themselves contain dot characters. When writing PQL queries, however, such as the following, it is currently impossible to use this documented dotted field syntax.
inventory[certname, facts."my.fact.has.dots"] { LIMIT 1 }
The following error will be returned.
2021/10/25 05:37:14 ERROR - [GET /pdb/query/v4][400] getQueryBadRequest PQL parse error at line 1, column 27:
inventory[certname,
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title Reid Vandewiele commented on PUP-11163 Re: Prevent thundering herd via deterministic run scheduling Possibly. There are definitely similarities. If there's any difference, I think it would be that this ticket is more "smooth workload distribution is something I want; thundering herds shouldn't be possible", while 4212 may have taking the slightly different focus of "fix the acute problem / symptom thundering herds by breaking them up when they occur". The tighter you get to workload tolerances at scale, the less tolerable it is to have herds in the first place. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.406175.1625677218000.141554.1632864180023%40Atlassian.JIRA.
Jira (PUP-10665) Add weak dependencies to puppet resources
Title: Message Title Reid Vandewiele commented on PUP-10665 Re: Add weak dependencies to puppet resources It's really too bad we can't use before and after for weak ordering. isolated_* isn't exactly intuitive. Maybe instead we could for those ones do order_before order_after It's true we can never remove the original before, but it might be a good idea to introduce a new alias for that functionality which we can recommend new code use instead. We have require, which is good; we could pair require with the new before alias: required_by Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.371714.1599956556000.139514.1632518640040%40Atlassian.JIRA.
Jira (PUP-10665) Add weak dependencies to puppet resources
Title: Message Title Reid Vandewiele commented on PUP-10665 Re: Add weak dependencies to puppet resources Language check: to match existing grammar (require, subscribe, before, notify), the suggested weak dependency keywords should be singular, not plural. That is, optional_require optional_notify want wanted_by Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.371714.1599956556000.138602.1632430320084%40Atlassian.JIRA.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting _+ a random seed.†_ * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a previously scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.† It is not important for a given agent to always select the same wall-clock start times; only for agents to have different, evenly spread start times, and for subsequent runs to be +consistently+ spaced. Including a random seed element permits
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting _+ a random seed.†_ * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a previously scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.† It is not important for a given agent to always select the same wall-clock start times; only for agents to have different, evenly spread start times, and for subsequent runs to be +consistently+ spaced. Including a random seed element permits
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting _+ a random seed.†_ * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a previously scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations. † It is not important for a given agent to always select the same wall-clock start times; only for agents to have different, evenly spread start times, and for subsequent runs to be +consistently+ spaced.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title
Reid Vandewiele commented on PUP-10491
Re: Allow adding descriptive or administrative information to resources
Josh Cooper I put a note out to puppet-dev on May 14, 2020, here: https://groups.google.com/g/puppet-dev/c/cDCyp_U8Sv0/m/FiN1EqxWAwAJ. No responses or commentary in that forum. What else needs to happen to move this ticket forward?
Henrik Lindberg that's a good consideration. Type validation is a powerful feature, provides some compelling benefits, and that idea inspires the example below. However, I think use of types needs to be optional, not something that could act as a barrier to entry. To be most useful, annotations need to be very, very easy to use. Users who want type validation could adopt this pattern:
type OurOrg::Annotation =
Struct[Optional[owner] => String[1],
Optional[department] => Enum[operations, support, engineering]]
notify { 'example':
annotation => OurOrg::Annotation({'owner' => 'Riley'})
}
In the proposed implementation, the metaparameter is validated by a type already (Data). If this is a compelling adjustment, we could choose someday to make that configurable. E.g.
annotation_assert_type = Data # default; could set to OurOrg::Annotation instead
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title Reid Vandewiele commented on PUP-11163 Re: Prevent thundering herd via deterministic run scheduling 100% of customers run into the problem. A lesser percentage notice it. If the customer is small (say, less than a thousand nodes), the available Puppet compute power will eclipse the total agent workload enough to make the thundering herd issue not a problem. Small herd, small thunder. The issue becomes a problem when the available compute power starts to become more evenly matched to the workload. Charlie Sharpsteen would have to speak to the percentage of customers that Support needs to advise on enabling our special configuration max-queued-requests option, which is the first tier workaround used to mitigate the issue. For our largest customers, say >10,000 nodes (at a guess) we pretty universally have to talk about max-queued-requests at a minimum up front, and for most we end up evolving into suggesting the second tier workaround, which is reidmv-puppet_run_scheduler or similar—basically, stop using the service, and use cron/scheduled-tasks to run the agent instead. That strategy, even as a pre-implemented module, is uncomfortable because all customers would rather run services daemons than cron/scheduled-tasks. The big ones grumpily end up sacrificing that in exchange for stable performance. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on
Jira (PUP-11189) Add functions to iterate over files/templates in the current module
Title: Message Title
Reid Vandewiele commented on PUP-11189
Re: Add functions to iterate over files/templates in the current module
Thanks for that additional context, Dirk Heinrichs. That's helpful. I'm hearing that the simple case, copy files as-is from module to target, is achievable today with the File resource and recurse => remote. I'm also hearing that for a more complex, probably less common use case involving templates, it would be useful to have a general capability of inspecting module content dynamically from Puppet DSL code. Given the complexity and non-generality of the second use case, I don't think it would make sense to implement a series of content-type-specific functions to enable this. That is, I don't think extending or modifying find_template, find_file, etc. would be generally beneficial. It could make sense to extend a capability similar to the existing module_directory() function. For example, pairing it with a new module_files() function. A function like that could take a list of module-relative glob paths, and return an array of file paths. E.g.
call: module_files('module1/files/*', 'module2/templates/**/*.epp', 'module3/metadata.json')
return: [
"/etc/puppetlabs/code/environments/production/modules/module1/files/file1",
"/etc/puppetlabs/code/environments/production/modules/module1/files/file2",
"/etc/puppetlabs/code/environments/production/modules/module2/templates/tmp1.txt.epp",
"/etc/puppetlabs/code/environments/production/modules/module2/templates/tmp2.txt.epp",
"/etc/puppetlabs/code/environments/production/modules/module2/templates/subdir/tmp3.txt.epp",
"/etc/puppetlabs/code/environments/production/modules/module3/metadata.json",
Jira (PUP-11189) Add functions to iterate over files/templates in the current module
Title: Message Title
Reid Vandewiele commented on PUP-11189
Re: Add functions to iterate over files/templates in the current module
I'm curious to understand the use case better. From my first reading of the desired goal, it seems like you should be able to do this today as follows:
file { 'target-directory':
ensure => directory,
path=> '/path/to/dir',
source => 'puppet:///foo/subdir',
recurse => remote,
}
Dirk Heinrichs could you describe in a little bit more detail what you are trying to do specifically? Curious to understand better. Less the technical how-to of what you've tried; more the use case why/what-for. Hoping that a better understanding will shed some light on whether there's already a recommended way to do what you're looking for today, or if not, exactly what benefits the suggested improvement could provide, and how it looks when solving the target problem(s). Especially if introducing it would change the recommended way of solving the problem from an existing option today.
Add Comment
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title Reid Vandewiele commented on PUP-11163 Re: Prevent thundering herd via deterministic run scheduling Beth Glenfield what information is needed for this ticket? (asking based on ticket status change, I may just not understand what that status means in the workflow) Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.406175.1625677218000.82151.1626307980031%40Atlassian.JIRA.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a previously scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity . This may happen after the special startup run, for example, or if a scheduled run entered into a backoff/retry loop based on server 503 responses We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a certain configured proximity We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run # * Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed within a certain proximity We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random splay offset element , seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately , or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run # Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random splay, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequently subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run # Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use a combination of splayed and , deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random splay, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequently run should be scheduled exactly {{$run_interval}} after of the previously scheduled run # Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11163
Prevent thundering herd via deterministic run scheduling
Change By:
Reid Vandewiele
h2. ProblemOne of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [ determine a thundering herd condition in Puppet |https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21).Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use a combination of splayed and deterministic run start times to evenly distribute agent checkins.Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration.h2. Desired SolutionThe Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode.Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random splay, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock scheduleOn startup, an agent might schedule runs thusly. # Schedule a startup run for{{now() + $splay}} # Schedule the first regular daemon run for{{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequently run should be scheduled exactly {{$run_interval}} after of the previously scheduled run # Optionally, a provision can be made to skip the first scheduled daemon run if it would occur too soon after the startup run We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations.
Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling
Title: Message Title Reid Vandewiele created an issue Puppet / PUP-11163 Prevent thundering herd via deterministic run scheduling Issue Type: Improvement Assignee: Unassigned Created: 2021/07/07 10:00 AM Priority: Normal Reporter: Reid Vandewiele Problem One of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was determine a thundering herd condition in Puppet (OSPA-21). Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use a combination of splayed and deterministic run start times to evenly distribute agent checkins. Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See this module for full automation of the configuration. Desired Solution The Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode. Characteristics: Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. An agent's individual run start time should include a random splay, seeded on the agent's certname setting The agent should schedule and perform a first-start run quickly (immediately or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock schedule On startup, an agent might schedule runs thusly. Schedule a startup run for now() + $splay Schedule the
Jira (PUP-9577) Large numbers of facts cause slow performance
Title: Message Title
Reid Vandewiele commented on PUP-9577
Re: Large numbers of facts cause slow performance
The customer triage that was put in place, and which decisively solved the issue, was to patch the Puppet::Parser::TemplateWrapper class to filter which ivars are set. Filter:
require 'delegate'
class ToHashScopeFilterDelegator < SimpleDelegator
def to_hash(*args)
__getobj__.to_hash(*args).reject { |k, _| k.start_with?('badfact_') }
end
end
Patch:
Puppet::Parser::TemplateWrapper.class_eval do
def scope
ToHashScopeFilterDelegator.new(@__scope__)
end
end
Jira (PUP-9577) Large numbers of facts cause slow performance
Title: Message Title
Reid Vandewiele commented on PUP-9577
Re: Large numbers of facts cause slow performance
Taking a look at this due to an at-scale customer performance issue. From what I'm told it sounds like a JRuby implementation detail may be causing a lot of strife when nodes have large numbers of per-node-unique facts which are never used in templates, but which JRuby slowly adds references to on a per-class basis, as they are encountered, which means data in the Puppet::Parser::TemplateWrapper table sticks around long after individual instances of that class are gone, accumulating and impacting catalog runs. As a triage solution, we talked about modifying how instance variables are added to Puppet::Parser::Template instances such that we only add an instance variable if we do a simple check of the template source material and believe it to reference the variable. Rough idea expressed in Ruby:
ivar_names_regex = %r{(?<=@)[a-z0-9_][a-zA-Z0-9_]*}m
ivars_used = template_source.scan(ivar_names_regex)
ivars_used.each do |name|
instance_variable_set("@#{name}", scope.lookupvar(name))
end
Is there an easy way to extend this approach to make it viable enough for a product patch?
Add Comment
Jira (PUP-11023) Make trlinkin-noop a fully supported feature
Title: Message Title Reid Vandewiele commented on PUP-11023 Re: Make trlinkin-noop a fully supported feature Hey Ciprian Badescu, can you add a comment clarifying what information is needed, or if it is not literally information needed, what next steps / conditions / events the ticket is now waiting on? Thanks! Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.394534.1617818266000.57158.1623781380027%40Atlassian.JIRA.
Jira (PUP-9570) Catalog failure on first run due to pluginsync and environment switch
Title: Message Title Reid Vandewiele commented on PUP-9570 Re: Catalog failure on first run due to pluginsync and environment switch Ciprian Badescu yep, probably. That hypothesis—a fact not being available in all environments—is almost certainly the cause. It usually is. It would correspond to step 9 in this ticket's description. The only way to work around the issue today is for the user to play whack-a-mole with facts when problems are noticed, and create piecemeal dependency entanglements between otherwise unrelated environments. This is highly undesirable and creates friction for a platform team trying to provide Puppet-as-a-Service to other teams in an organization. This is what we need to fix. We don't want users to have to work around issues like this on a case-by-case basis. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300842.1553021716000.38449.1621610640029%40Atlassian.JIRA.
Jira (PUP-11023) Make trlinkin-noop a fully supported feature
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11023
Make trlinkin-noop a fully supported feature
Change By:
Reid Vandewiele
The Forge module [trlinkin-noop|https://forge.puppet.com/modules/trlinkin/noop] has been used heavily by customers for years. However, it is extremely finicky to use due to a bit of a hack of an implementation. This makes it difficult to base higher-level use cases on. See [reidmv-change_risk|https://forge.puppet.com/modules/reidmv/change_risk], which provides some customers with an ability to better control when Puppet is permitted to enforce certain kinds of changes on systems, while continuously enforcing other configuration.We should provide an in-product {{noop()}} function which can be called directly, affecting class scope, or with a block, affecting that block's scope.These implementations should be more predictable and less brittle than the trlinkin-noop module's implementation. *They should work at least as reliably and predictably as tags.* The behavior with regards to propagation relating to { {include()}} is the thing most important to make better compared to trlinkin-noop. { code:java}class example1 { noop() notify { 'I am no-op': }} {code}{code:java} class example2 { noop() || { notify { 'I am no-op': } } notify { 'I am op': }}{code}
Add Comment
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Jira (PUP-11023) Make trlinkin-noop a fully supported feature
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-11023
Make trlinkin-noop a fully supported feature
Change By:
Reid Vandewiele
The Forge module [trlinkin-noop|https://forge.puppet.com/modules/trlinkin/noop] has been used heavily by customers for years. However, it is extremely finicky to use due to a bit of a hack of an implementation. This makes it difficult to base higher-level use cases on. See [reidmv-change_risk|https://forge.puppet.com/modules/reidmv/change_risk], which provides some customers with an ability to better control when Puppet is permitted to enforce certain kinds of changes on systems, while continuously enforcing other configuration.We should provide an in-product {{noop()}} function which can be called directly, affecting class scope, or with a block, affecting that block's scope.These implementations should be more predictable and less brittle than the trlinkin-noop module's implementation. *They should work at least as reliably and predictably as tags.*{code:java} class example1 { noop() notify { 'I am no-op': }} {code}{code:java}class example2 { noop() || { notify { 'I am no- op': } } noop() || { notify { 'I am no- op': } } }{code}
Add Comment
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Jira (PUP-11023) Make trlinkin-noop a fully supported feature
Title: Message Title
Reid Vandewiele created an issue
Puppet / PUP-11023
Make trlinkin-noop a fully supported feature
Issue Type:
New Feature
Assignee:
Unassigned
Created:
2021/04/07 10:57 AM
Priority:
Normal
Reporter:
Reid Vandewiele
The Forge module trlinkin-noop has been used heavily by customers for years. However, it is extremely finicky to use due to a bit of a hack of an implementation. This makes it difficult to base higher-level use cases on. See reidmv-change_risk, which provides some customers with an ability to better control when Puppet is permitted to enforce certain kinds of changes on systems, while continuously enforcing other configuration. We should provide an in-product noop() function which can be called directly, affecting class scope, or with a block, affecting that block's scope. These implementations should be more predictable and less brittle than the trlinkin-noop module's implementation. They should work at least as reliably and predictably as tags.
class example1 {
noop()
notify { 'I am no-op': }
Jira (FACT-3003) Don't split keys for already structured json/yaml external facts
Title: Message Title
Reid Vandewiele commented on FACT-3003
Re: Don't split keys for already structured json/yaml external facts
I did a think-through of this and came up with the following framing, which I think describes how we have implemented this functionality today. Adding a Fact There are three primary ways to add facts to Facter.
Via the Ruby API's Facter.add() method
Via a text-emitting external fact
Via a structured-object emitting external fact
For all three methods, the definition spec is the same. Input: A dot-notation insertion spec indicating where in the Fact tree the new value should appear. Output: The value which should appear at the designated insertion point. Examples Each of the following examples results in the following Fact tree.
{
"tier": "production",
"cmdb" {
"owner": "Jaime",
"raci": {
"responsible": "Jaime",
"informed": "Logan"
}
}
}
Jira (FACT-3000) Dot notation implemented in Facter 4 is incomplete
Title: Message Title
Reid Vandewiele updated an issue
Facter / FACT-3000
Dot notation implemented in Facter 4 is incomplete
Change By:
Reid Vandewiele
Facter 4 introduced compositive fact structuring via dot-notation as a feature. However, the dot-notation implementation used is incomplete, and behaves differently from dot-notation elsewhere in the ecosystem.Specifically, Facter 4 dot-notation splits key segments incorrectly, because it does not recognize the use of double or single quotes to indicate keys which contain literal dot characters.The reference implementation for dot-notation is found in Puppet, [here|https://github.com/puppetlabs/puppet/blob/85bd9ed07e3e6c956f1972624a50be914292e1ab/lib/puppet/pops/lookup/sub_lookup.rb#L6-L44].Full dot-notation should be supported everywhere Facter 4 currently evaluates dot-notation strings to segment structured fact names. This includes external fact output (text, json, yaml), as well as {{Facter.add()}}.*Today's behavior*Consider the following plain-text custom fact file:{code}foo.bar._one_=1foo.bar.two=2foo."i.have.dots"=3"me.too"=4{code}Today, Facter returns the following for these facts.{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"\"i": { "have": {"dots\"": "3" }} }, "\"me": {"too\"": "4" }}{code}This is incorrect.*Correct behavior*Full and correct support for dot-notation SHOULD give:{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"i.have.dots": "3" }, "me.too": "4"}{code}
Add Comment
Jira (FACT-3000) Dot notation implemented in Facter 4 is incomplete
Title: Message Title
Reid Vandewiele updated an issue
Facter / FACT-3000
Dot notation implemented in Facter 4 is incomplete
Change By:
Reid Vandewiele
Facter 4 introduced compositive fact structuring via dot-notation as a feature. However, the dot-notation implementation used is incomplete, and behaves differently from dot-notation elsewhere in the ecosystem.Specifically, Facter 4 dot-notation splits key segments incorrectly, because it does not recognize the use of double or single quotes to indicate keys which contain literal dot characters.The reference implementation for dot-notation is found in Puppet, [here|https://github.com/puppetlabs/puppet/blob/85bd9ed07e3e6c956f1972624a50be914292e1ab/lib/puppet/pops/lookup/sub_lookup.rb#L6-L44]. Full dot-notation should be supported everywhere Facter 4 currently evaluates dot-notation to segment structured fact names. This includes external fact output (text, json, yaml), as well as {{Facter.add()}}. *Today's behavior*Consider the following plain-text custom fact file:{code}foo.bar._one_=1foo.bar.two=2foo."i.have.dots"=3"me.too"=4{code}Today, Facter returns the following for these facts.{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"\"i": { "have": {"dots\"": "3" }} }, "\"me": {"too\"": "4" }}{code}This is incorrect.*Correct behavior*Full and correct support for dot-notation SHOULD give:{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"i.have.dots": "3" }, "me.too": "4"}{code}
Add Comment
Jira (FACT-3000) Dot notation implemented in Facter 4 is incomplete
Title: Message Title
Reid Vandewiele updated an issue
Facter / FACT-3000
Dot notation implemented in Facter 4 is incomplete
Change By:
Reid Vandewiele
Facter 4 introduced compositive fact structuring via dot-notation as a feature. However, the dot-notation implementation used is incomplete, and behaves differently from dot-notation elsewhere in the ecosystem.Specifically, Facter 4 dot-notation splits key segments incorrectly, because it does not recognize the use of double or single quotes to indicate keys which contain literal dot characters.The reference implementation for dot-notation is found in Puppet, [here|https://github.com/puppetlabs/puppet/blob/85bd9ed07e3e6c956f1972624a50be914292e1ab/lib/puppet/pops/lookup/sub_lookup.rb#L6-L44].*Today's behavior*Consider the following plain-text custom fact file:{code}foo.bar._one_=1foo.bar.two=2foo."i.have.dots"=3"me.too"=4{code}Today, Facter returns the following for these facts.{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"\"i": { "have": {"dots\"": "3" }} }, "\" foo me ": {"too\"": "4" }}{code}This is incorrect.*Correct behavior*Full and correct support for dot-notation SHOULD give:{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"i.have.dots": "3" }, "me.too": "4"}{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (FACT-3000) Dot notation implemented in Facter 4 is incomplete
Title: Message Title
Reid Vandewiele updated an issue
Facter / FACT-3000
Dot notation implemented in Facter 4 is incomplete
Change By:
Reid Vandewiele
Facter 4 introduced compositive fact structuring via dot-notation as a feature. However, the dot-notation implementation used is incomplete, and behaves differently from dot-notation elsewhere in the ecosystem.Specifically, Facter 4 dot-notation splits key segments incorrectly, because it does not recognize the use of double or single quotes to indicate keys which contain literal dot characters.The reference implementation for dot-notation is found in Puppet, [here|https://github.com/puppetlabs/puppet/blob/85bd9ed07e3e6c956f1972624a50be914292e1ab/lib/puppet/pops/lookup/sub_lookup.rb#L6-L44].*Today's behavior*Consider the following plain-text custom fact file:{code}foo.bar._one_=1foo.bar.two=2foo." I i . Have have . Dots dots "=3"me.too"=4{code}Today, Facter returns the following for these facts.{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"\"i": { "have": {"dots\"": "3" }} }, "\"foo": {"too\"": "4" }}{code}This is incorrect.*Correct behavior*Full and correct support for dot-notation SHOULD give:{code:json}{ "foo": {"bar": { "one": "1", "two": "2"},"i.have.dots": "3" }, "me.too": "4"}{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (FACT-3000) Dot notation implemented in Facter 4 is incomplete
Title: Message Title Reid Vandewiele created an issue Facter / FACT-3000 Dot notation implemented in Facter 4 is incomplete Issue Type: Bug Assignee: Unassigned Created: 2021/03/26 12:57 PM Priority: Normal Reporter: Reid Vandewiele Facter 4 introduced compositive fact structuring via dot-notation as a feature. However, the dot-notation implementation used is incomplete, and behaves differently from dot-notation elsewhere in the ecosystem. Specifically, Facter 4 dot-notation splits key segments incorrectly, because it does not recognize the use of double or single quotes to indicate keys which contain literal dot characters. The reference implementation for dot-notation is found in Puppet, here. Today's behavior Consider the following plain-text custom fact file: foo.bar._one_=1 foo.bar.two=2 foo."I.Have.Dots"=3 "me.too"=4 Today, Facter returns the following for these facts.
Jira (PUP-10966) Allow full-relative path references in file(), template(), related functions
Title: Message Title Reid Vandewiele commented on PUP-10966 Re: Allow full-relative path references in file(), template(), related functions Josh Cooper the Bolt team have sketched out a pretty solid adoption and deprecation plan for Bolt, with regards to the use of pseudo-relative paths pertaining to Bolt scripts—e.g. run_script()—which used to borrow the idiom. What would it take for Puppet to align with their deprecation plan, albeit with a MUCH longer (probably indefinite) deprecation period for the old pathing? Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.391231.1615913392000.171664.1616516040625%40Atlassian.JIRA.
Jira (PUP-10966) Allow full-relative path references in file(), template(), related functions
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10966
Allow full-relative path references in file(), template(), related functions
Change By:
Reid Vandewiele
One of Puppet's most confusing long-standing idioms is its use of pseudo-relative module file references. Most notably, in {{file()}}, {{template()}}, and related functions.{code:java}# This retrieves content from /modname/files/filenamefile('modname/filename')# This retrieves content from /modname/templates/filenametemplate('modname/filename') {code}This is difficult for new users to understand, and when ported to the command-line it breaks normal Unix behavior in that forward-slash does not reliably indicate a real file path, even relatively.h2. Feature RequestAnywhere these confusing references are used, permit instead full-relative path references, following this convention:{{/full/relative/path}}So, the examples above could be used instead as:{code:java} # This retrieves content from /modname/files/filenamefile('modname/files/filename')# This retrieves content from /modname/templates/filenametemplate('modname/templates/filename') {code}The same way that it has become best practice to always use a DSL option that includes the literal string "facts" when referencing facts for clarity*, this capability would improve intuitiveness, readability, and discoverability for new users.This will also make using a an ecosystem consistent full-relative module path more tenable in CLI applications.* e.g., don't use {{$::osfamily}}. Use {{getvar('facts.osfamily)}} or {{$facts['osfamily']}} instead.
Add Comment
Jira (PUP-10966) Allow full-relative path references in file(), template(), related functions
Title: Message Title
Reid Vandewiele created an issue
Puppet / PUP-10966
Allow full-relative path references in file(), template(), related functions
Issue Type:
Improvement
Assignee:
Unassigned
Created:
2021/03/16 9:49 AM
Priority:
Normal
Reporter:
Reid Vandewiele
One of Puppet's most confusing long-standing idioms is its use of pseudo-relative module file references. Most notably, in file(), template(), and related functions.
# This retrieves content from /modname/files/filename
file('modname/filename')
# This retrieves content from /modname/templates/filename
template('modname/filename')
This is difficult for new users to understand, and when ported to the command-line it breaks normal Unix behavior in that forward-slash does not reliably indicate
Jira (PUP-10093) Add a rewrap(Any, Callable) function to make using Sensitive easier
Title: Message Title Reid Vandewiele commented on PUP-10093 Re: Add a rewrap(Any, Callable) function to make using Sensitive easier I'm curious to see an example of what a user would have to do to get benefit from this. I.e. a user story, and a Puppet manifest or code snippet demonstrating how rewrap() helps. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.328237.157026849.133439.1612460940135%40Atlassian.JIRA.
Jira (PUP-9072) Use repeated_param UX for Deferred()
Title: Message Title Reid Vandewiele commented on PUP-9072 Re: Use repeated_param UX for Deferred() Interested? Totally. Nothing has changed about the UX considerations. Only thing different now is what year it is. I think it would have been best to ship this with the preferred UX on day 1 so there's some diminishment to the value from the non-preferred way having been out there and being learned by people for awhile. It sounded to me like where this got left was an open question about whether it was technically possible to define a constructor that takes String, *args. If the answer is "yes, and it's easy", I think we should totally do it. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.271610.1534439633000.94268.1607475060023%40Atlassian.JIRA.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10491
Allow adding descriptive or administrative information to resources
Change By:
Reid Vandewiele
Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifiers pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration items So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*annotation*". As a metaparam, "annotation" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The annotation metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).Example usage:{code:java}notify { 'example': message => 'this is a notify resource', annotation => 'this is some annotation, a description, about it',}file { '/tmp/example.conf': ensure => file, owner=> 'root', metadata annotation => {'description' => ' Metadata Annotation does not need to be a string','product' => 'Example Product','owner' => 'Kelly', },} {code}h3. What about tags?Tags have two
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10491
Allow adding descriptive or administrative information to resources
Change By:
Reid Vandewiele
Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifiers pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration items So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "* metadata annotation *". As a metaparam, " metadata annotation " would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata annotation metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).Example usage:{code:java} notify { 'example': message => 'this is a notify resource', metadata annotation => 'this is some metadata annotation , a description, about it',}file { '/tmp/example.conf': ensure => file, owner=> 'root', metadata => {'description' => 'Metadata does not need to be a string','product' => 'Example Product','owner' => 'Kelly', },} {code}h3. What
Jira (PUP-10774) Long query time for AD groups
Title: Message Title Reid Vandewiele commented on PUP-10774 Re: Long query time for AD groups Question: on the FFI github page, there is this warning: On Linux systems running with PaX (Gentoo, Alpine, etc.), FFI may trigger mprotect errors. You may need to disable mprotect for ruby (paxctl -m [/path/to/ruby]) for the time being until a solution is found. Would we need to worry about this at all on our supported platforms, if we start using FFI, LIBC, and getgrouplist? I don't know how PaX/mprotect works but it seems like even if a customer did have it installed we might be okay since we're only linking LIBC, and I'm sure that's already being used by Ruby. Figured it was worth asking the question though. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.377144.160449921.74227.1604706300203%40Atlassian.JIRA.
Jira (PUP-8969) Support interpolation of sensitive values in EPP templates
Title: Message Title Reid Vandewiele commented on PUP-8969 Re: Support interpolation of sensitive values in EPP templates Josh Cooper I have never seen a situation where in a template or epp the customer wanted the string "** REDACTED **" (or whatever) to be displayed. I've frequently seen people accidentally do it, and then have to thrash around for awhile trying to figure out that they need to call .unwrap(). Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.262922.1530114614000.52738.1602265740028%40Atlassian.JIRA.
Jira (PUP-7582) Make attempt to export or virtualize a class an error
Title: Message Title
Reid Vandewiele commented on PUP-7582
Re: Make attempt to export or virtualize a class an error
I think we should make it a hard error. The point of the ticket is that when you put a "@" in front of class today, it silently doesn't do what you probably thought it was doing, and that's confusing. If we make it a hard error there could be some code, somewhere, that a user would then need to remove an "@" sign from. I've never seen a user try to do this intentionally. Removing the "@" sign would have no functional change on what their code does anyway, and at first blush I believe it's safe to fix non-clever code with something close to as simple as find . -name '*.pp' | xargs sed -i 's/@class {/class {/'. An error is better than not actually doing what the syntax lets you think you're doing.
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.19.1495458026000.41483.1600883880080%40Atlassian.JIRA.
Jira (PUP-10525) puppet apply should set trusted['external']
Title: Message Title Reid Vandewiele commented on PUP-10525 Re: puppet apply should set trusted['external'] Enis Inan interesting, sounds like that should support the dev/test use case. Either set trusted_external_command in the user/main section, or pass it on the CLI. For the servicenow_integration specifically I don't at first blush believe we should move it from the master section... I kind of feel like it shouldn't happen for manual invocations of apply unless intentionally invoked. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.359456.1590012226000.68066.1590103440023%40Atlassian.JIRA.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10491
Allow adding descriptive or administrative information to resources
Change By:
Reid Vandewiele
Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifiers pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration itemsSo that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases. h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be). Example usage:{code:java}notify { 'example': message => 'this is a notify resource', metadata => 'this is some metadata, a description, about it',}file { '/tmp/example.conf': ensure => file, owner=> 'root', metadata => {'description' => 'Metadata does not need to be a string','product' => 'Example Product','owner' => 'Kelly', },} {code} h3. What about tags?Tags have two characteristics that make them unsuitable to
Jira (PUP-9570) Catalog failure on first run due to pluginsync and environment switch
Title: Message Title Reid Vandewiele commented on PUP-9570 Re: Catalog failure on first run due to pluginsync and environment switch Josh Cooper Any update on this? Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300842.1553021716000.65495.1589904060027%40Atlassian.JIRA.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele commented on PUP-10491 Re: Allow adding descriptive or administrative information to resources I've managed to come up with one possible reason to prefer "annotation" over "metadata". Recording it here for consideration. In the specific context of Puppet, we've already used "meta-" to describe the general space that a parameter like this exists in. It is literally a meta parameter. In a sense, all of the metaparameters (tag, notify, loglevel) are metadata; just different kinds of metadata. Perhaps the name "annotation" should be preferred as it is a clarification of what metadata is being assigned, and that it is metadata is already communicated by way of the set of parameters, metaparameters, to which it belongs. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.357687.1588699805000.63415.1589555760039%40Atlassian.JIRA.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele commented on PUP-10491 Re: Allow adding descriptive or administrative information to resources I've updated the description quite a bit, and changed the suggested name to "metadata" (reasoning given). Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.357687.1588699805000.63102.1589497020111%40Atlassian.JIRA.
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifiers pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration itemsSo that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be). h2 h3 . What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifier pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration itemsSo that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for • Business-process identifier identifiers pertaining to configuration items • Human-readable descriptions of configuration items • Responsible party for configuration itemsSo that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for * • Business-process identifier pertaining to configuration items * • Human-readable descriptions of configuration items * • Responsible party for configuration items So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for* Business-process identifier pertaining to configuration items * Human-readable descriptions of configuration items * Responsible party for configuration items So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the propagation
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for * Business-process identifier pertaining to configuration items * Human-readable descriptions of configuration items * Responsible party for configuration items | So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases.|1 |As a Puppet code workflow designer, I want to my teams to create document-as-you-go annotations in their code for * Business-process identifier pertaining to configuration items * Human-readable descriptions of configuration items * Responsible party for configuration items So that later when Puppet is is making changes to a system, I can easily look up who, what, and why.||2|As a Puppet content developer, I want to attach ownership information to resources, So my team can be identified and alerted if Puppet has a problem applying my resources.||3|As a change deployment approver, When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to consume a high-level summation for the _meaning_ of changes applied to a system, So I can quickly get a sense of what's being done, Without needing to read low-level Puppet implementation code.||4| As a customer receiving a Puppet-managed server as a service, I want to review a *human-readable* list of all “stuff” managed by Puppet on my system, So I can have some understanding of what Puppet is doing.|Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.h3. SuggestionA simple way to solve for this would be to add a new, non-operative metaparam, "*metadata*". As a metaparam, "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparam would by itself constitute foundational delivery of this feature.The metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on such a thing to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h2. What about tags?Tags have two characteristics that make them unsuitable to fully address these use cases.# Tags cannot contain text written for human consumption, such as descriptions or text-form comments due to not allowing spaces and other special characters.# Tags propagate. Especially for long-form comments, it may be desirable NOT to propagate them, as duplication and accidental scope escape can be undesirable.For some use cases the propagation
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Consider the following use cases. # |1 | As a Puppet developer code workflow designer , I want to attach comments my teams to resources, create document-as-you-go annotations in their code for * Business-process identifier pertaining to configuration items So I can export a high * Human - level summary readable descriptions of what configuration items * Responsible party for configuration items |So that later when Puppet is being managed by is making changes to a catalog system, I can easily look up who, what, and why.| # |2| As a sysadmin / change reviewer Puppet content developer , I want to easily identify at a high level what is changing on my system attach ownership information to resources , So I don’t need to understand all the low-level my team can be identified and alerted if Puppet implementation code has a problem applying my resources.| # |3| As an auditor a change deployment approver , When reviewing changes made in a lower-tier environment to decide if it's ok to promote, I want to review consume a *human high - readable* list level summation for the _meaning_ of changes made applied to a system in the last month , So I can verify all audit controls are quickly get a sense of what's being satisfied by done, Without needing to read low-level Puppet implementation code.| # |4| As a manager customer receiving a Puppet-managed server as a service , I want to review a *human-readable* list of all “stuff” managed by Puppet on a given my system, So I can understand have some understanding of what Puppet is doing .| Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption. Further, while we have puppet-strings to create self-documenting _code_, we don't have any tooling to create self-documenting _catalogs_ or _reports_—these being examples of artifact outputs produced by the potentially documented code. h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases. h3. Suggestion A very simple way to solve for this would be to add a new, non-operative metaparameter metaparam , " annotation *metadata* ". As a metaparameter metaparam , annotation "metadata" would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparameter metaparam would by itself constitute a fully functional foundational delivery of this feature. The Puppet feature metadata metaparam could be built on by later PE features to produce polished reports, but would not depend on them being delivered such a thing to deliver value today.For a customer
Jira (PUP-10491) Allow adding descriptive or administrative information to resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Allow adding descriptive or administrative information to resources Change By: Reid Vandewiele Summary: Provide annotation capability for Allow adding descriptive or administrative information to resources Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.357687.1588699805000.62943.1589489040035%40Atlassian.JIRA.
Jira (PUP-10491) Provide annotation capability for resources
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10491
Provide annotation capability for resources
Change By:
Reid Vandewiele
Consider the following use cases. # As a Puppet developer, I want to attach comments to resources, So I can export a high-level summary of what is being managed by a catalog # As a sysadmin / change reviewer, I want to easily identify at a high level what is changing on my system, So I don’t need to understand all the low-level Puppet implementation code # As an auditor, I want to review a *human-readable* list of changes made to a system in the last month, So I can verify all audit controls are being satisfied by Puppet # As a manager, I want to review a *human-readable* list of all “stuff” managed by Puppet on a given system, So I can understand what Puppet is doingCurrently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.A very simple way to solve for this would be to add a new, non-operative metaparameter, "annotation". As a metaparameter, annotation would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparameter would constitute a fully functional feature. The Puppet feature could be built on by later PE features to produce polished reports, but would not depend on them being delivered to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h3. Compatibility considerationsAfaik, there are no concerns with regards to backwards compatibility. If an existing class or type already uses a parameter called "annotation", that will mask/override a metaparameter by the same name.Probable Implementation Difficulty: trivial. Improper diff to demonstrate:{code: java diff }diff --git a/lib/puppet/type.rb b/lib/puppet/type.rbindex 8951854ddd..5f230e3cea 100644--- a/lib/puppet/type.rb+++ b/lib/puppet/type.rb@@ -1714,6 +1714,16 @@ class Type+ newmetaparam(:annotation) do+desc "A user-supplied annotation about this resource. The annotation+ is non-operative, but will be visible in the node's catalog and can+ be queried for reporting purposes."+ end+ {code}
Jira (PUP-10491) Provide annotation capability for resources
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10491
Provide annotation capability for resources
Change By:
Reid Vandewiele
Consider the following use cases. # As a Puppet developer, I want to attach comments to resources, So I can export a high-level summary of what is being managed by a catalog # As a sysadmin / change reviewer, I want to easily identify at a high level what is changing on my system, So I don’t need to understand all the low-level Puppet implementation code # As an auditor, I want to review a *human-readable* list of changes made to a system in the last month, So I can verify all audit controls are being satisfied by Puppet # As a manager, I want to review a *human-readable* list of all “stuff” managed by Puppet on a given system, So I can understand what Puppet is doingCurrently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.A very simple way to solve for this would be to add a new, non-operative metaparameter, "annotation". As a metaparameter, annotation would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparameter would constitute a fully functional feature. The Puppet feature could be built on by later PE features to produce polished reports, but would not depend on them being delivered to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be).h3. Compatibility considerationsAfaik, there are no concerns with regards to backwards compatibility. If an existing class or type already uses a parameter called "annotation", that will mask/override a metaparameter by the same name. Probable Implementation Difficulty: trivial. Improper diff to demonstrate:{code:java}diff --git a/lib/puppet/type.rb b/lib/puppet/type.rbindex 8951854ddd..5f230e3cea 100644--- a/lib/puppet/type.rb+++ b/lib/puppet/type.rb@@ -1714,6 +1714,16 @@ class Type+ newmetaparam(:annotation) do+desc "A user-supplied annotation about this resource. The annotation+ is non-operative, but will be visible in the node's catalog and can+ be queried for reporting purposes."+ end+ {code}
Jira (PUP-10499) Out-of-sync no-op resources should cause dependent resources to skip
Title: Message Title
Reid Vandewiele created an issue
Puppet / PUP-10499
Out-of-sync no-op resources should cause dependent resources to skip
Issue Type:
Improvement
Assignee:
Unassigned
Created:
2020/05/11 12:00 PM
Priority:
Normal
Reporter:
Reid Vandewiele
Let resource A be a dependency of resource B Let resource A be no-op Problem Puppet will always consider B's dependency to be satisfied, even if resource A is detected to be out-of-sync. Resource A is in no-op mode so it cannot "fail". Puppet will never skip resource B due to a dependency failure. However, resource A is out-of-sync, and is known to be in an incorrect state. Logically, this should cause resource B to be skipped, just as it would be if resource A were to be marked out-of-sync due to a sync failure. Depending on your dependency chains this behavior can cause problems, when Puppet cannot successfully configure resource B unless or until resource A is in-sync. Example Manifest
file { 'dependency':
path => '/tmp/never-create-me',
ensure => present,
noop => true,
Jira (PUP-10491) Provide annotation capability for resources
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10491 Provide annotation capability for resources Change By: Reid Vandewiele Consider the following use cases. # As a Puppet developer, I want to attach comments to resources, So I can export a high-level summary of what is being managed by a catalog # As a sysadmin / change reviewer, I want to easily identify at a high level what is changing on my system, So I don’t need to understand all the low-level Puppet implementation code # As an auditor, I want to review a *human-readable* list of changes made to a system in the last month, So I can verify all audit controls are being satisfied by Puppet # As a manager, I want to review a *human-readable* list of all “stuff” managed by Puppet on a given system, So I can understand what Puppet is doingCurrently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption.h2. Feature RequestProvide a solve for use case #1 above, in support of the rest of the use cases.A very simple way to solve for this would be to add a new, non-operative metaparameter, "annotation". As a metaparameter, annotation would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the addition of the metaparameter would constitute a fully functional feature. The Puppet feature could be built on by later PE features to produce polished reports, but would not depend on them being delivered to deliver value today.For a customer engagement, a proof-of-concept module has been implemented [here|https://github.com/reidmv/reidmv-annotation]. (Note that the module is more complicated than a built-in integration would be). h3. Compatibility considerationsAfaik, there are no concerns with regards to backwards compatibility. If an existing class or type already uses a parameter called "annotation", that will mask/override a metaparameter by the same name. Add Comment
Jira (PUP-10491) Provide annotation capability for resources
Title: Message Title Reid Vandewiele created an issue Puppet / PUP-10491 Provide annotation capability for resources Issue Type: New Feature Assignee: Unassigned Created: 2020/05/05 10:30 AM Priority: Normal Reporter: Reid Vandewiele Consider the following use cases. As a Puppet developer, I want to attach comments to resources, So I can export a high-level summary of what is being managed by a catalog As a sysadmin / change reviewer, I want to easily identify at a high level what is changing on my system, So I don’t need to understand all the low-level Puppet implementation code As an auditor, I want to review a human-readable list of changes made to a system in the last month, So I can verify all audit controls are being satisfied by Puppet As a manager, I want to review a human-readable list of all “stuff” managed by Puppet on a given system, So I can understand what Puppet is doing Currently, we don't have a way to attach any kind of self-documenting information to Puppet content, or produce dynamic reports about what Puppet is doing on a target system based on documentation or annotations written for human consumption. Feature Request Provide a solve for use case #1 above, in support of the rest of the use cases. A very simple way to solve for this would be to add a new, non-operative metaparameter, "annotation". As a metaparameter, annotation would be available to specify on any Puppet resource or class. Because it is non-operative and for reporting purposes only, the
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title Reid Vandewiele commented on PUP-10478 Re: Add getvar Hiera backend hiera-eyaml is built in. Works out-of-box, no plugins or additional installation required. The primary use case motivating the discussion around providing it out-of-box (I don’t care if it’s technically core or technically one of the modules we ship with core) is: Provide a way to supply Hiera data through trusted_external_command. Like for eyaml, the ideal solution to this use case should not require users to identify and install extra plugins. It so happens that there are several other ways users inject data into Puppet for which the same motivation (I want this data to go to Hiera) may also exist. It seems like we can solve our primary use case through a generalized backend. Alternative approaches could be to modify the definition of how trusted_external_command works. I haven't explored any design ideas for that yet. We would need to be careful not to overcomplicate the interface if we want to jump that way. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.357322.1588196554000.52235.1588276140032%40Atlassian.JIRA.
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title Reid Vandewiele commented on PUP-10478 Re: Add getvar Hiera backend Hey Henrik! There was a long discussion in Slack about this that didn't get carried into the ticket description. Re: “makes the result of lookup depend on order of evaluation”: You're right, and we are aware of that concern. We had a long Slack discussion about best practices, user awareness of this kind of issue, what appropriate guard-rails should be, or how aggressively we should protect against users doing this. There is nowhere else in Hiera that programmatically protects against the use of local-scope variables in the hierarchy as far as I know, so the suggested default option topscope_variables_only=true and associated error message actually provides more protection for the proposed getvar backend than we have anywhere else. We considered permitting only $facts or $trusted to be used here by default as well. The current suggestion of any top-scope variable without opting in to what Puppet tells you is a bad idea is because it's usually a safe bet that top-scope variables are set before classes are evaluated, and therefore reasonably permissible. Re: “Don’t however try to make `paths` be anything other than file paths” That sounds like a valuable implementation note to be aware of. For now, updated the description to use "uris". Re: “you can write your own backend function […] that is super simple to do in a custom backend function” This is where we started, actually, and was one of the options considered and discussed in the Slack thread. The main counter-point is disagreement that for users writing a customer back-end function is that simple. It's simple to write a function, but more complicated to know that you can write a backend, choose an appropriate backend type, and put it together. Not that hard, but definitely a barrier. The original impetus for this ticket was implementing an integration with ServiceNow through the trusted_external_command feature. If we want to do this, and only do it ourselves, for ServiceNow, then we just write a servicenow bakcend. But if we want users to be able to do this more generally, including for in-house use of trusted_external_command, we should provide a generalized backend for this in Puppet. Add Comment
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code:yaml} hierarchy: - name: "Node-specific Hiera data, via trusted and via fact"data_hash: getvar paths uris : - "trusted.external.servicenow.hieradata" - "trusted.external.unicorn.hieradata" - "facts.hieradata" - name: "Some yaml stuff"data_hash: yamlpaths: - "environments/%{trusted.extensions.pp_environment}.yaml" - "common.yaml"{code}where for the {{getvar}} Hiera layer, {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys. For example:{code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"}{code}To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}}, because per Hiera best practices, only variables not subject to evaluation-order dependencies or calling scope should be referenced in hiera.yaml. Thus, the following hierarchy configuration should fail:{code:yaml}---hierarchy: - name: "Example failure"data_hash: getvar path uri : "class_scope_variable.hieradata"{code}The error emitted should look something like:{panel}{{ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml: getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.}}{panel}Configuring the backend to allow the use of non-topscope variables would look like this:{code:yaml}---hierarchy: - name: "Example failure (bypassed)"data_hash: getvar path uri : "class_scope_variable.hieradata"options: topscope_variables_only: false{code}The backend type should be {{data_hash}}. The most technically appropriate type might be {{lookup_key}}, but there are UX considerations here and the name {{data_hash}} helps communicate to the user how this works—the data returned should be like what you find in a yaml file, which is type {{data_hash}}.
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code:yaml}hierarchy: - name: "Node-specific Hiera data, via trusted and via fact" lookup_key data_hash : getvarpaths: - "trusted.external.servicenow.hieradata" - "trusted.external.unicorn.hieradata" - "facts.hieradata" - name: "Some yaml stuff"data_hash: yamlpaths: - "environments/%{trusted.extensions.pp_environment}.yaml" - "common.yaml"{code}where for the {{getvar}} Hiera layer, {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys. For example:{code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"}{code}To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}}, because per Hiera best practices, only variables not subject to evaluation-order dependencies or calling scope should be referenced in hiera.yaml. Thus, the following hierarchy configuration should fail:{code:yaml}---hierarchy: - name: "Example failure" lookup_key data_hash : getvarpath: "class_scope_variable.hieradata"{code}The error emitted should look something like:{panel}{{ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml: getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.}} {panel}Configuring the backend to allow the use of non-topscope variables would look like this:{code:yaml}---hierarchy: - name: "Example failure (bypassed)" lookup_key data_hash : getvarpath: "class_scope_variable.hieradata"options: topscope_variables_only: false{code} The backend type should be {{data_hash}}. The most technically appropriate type might be {{lookup_key}}, but there are UX considerations here and the name {{data_hash}} helps communicate to the user how this works—the data returned should be like what you find in a yaml file, which is type {{data_hash}}.
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like {code:yaml} hierarchy: - name: "Node-specific Hiera data, via trusted and via fact"lookup_key: getvarpaths: - "trusted.external.servicenow.hieradata" - "trusted.external.unicorn.hieradata" - "facts.hieradata" - name: "Some yaml stuff"data_hash: yamlpaths: - "environments/%{trusted.extensions.pp_environment}.yaml" - "common.yaml"{code} where for the {{getvar}} Hiera layer, {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys. For example: {code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"}{code} To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}}, because per Hiera best practices, only variables not subject to evaluation-order dependencies or calling scope should be referenced in hiera.yaml. Thus, the following hierarchy configuration should fail: {code:yaml} ---hierarchy: - name: "Example failure"lookup_key: getvarpath: "class_scope_variable.hieradata"{code} The error emitted should look something like: { code panel } {{ ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml : getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false. }} { code panel } Configuring the backend to allow the use of non-topscope variables would look like this: {code:yaml} ---hierarchy: - name: "Example failure (bypassed)"lookup_key: getvarpath: "class_scope_variable.hieradata"options: topscope_variables_only: false{code}
Add
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code:yaml}hierarchy: - name: "Node-specific Hiera data, via trusted and via fact"lookup_key: getvarpaths: - "trusted.external.servicenow.hieradata" - "trusted.external.unicorn.hieradata" - "facts.hieradata" - name: "Some yaml stuff"data_hash: yamlpaths: - "environments/%{trusted.extensions.pp_environment}.yaml" - "common.yaml"{code}where for the {{getvar}} Hiera layer, {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys. For example:{code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"}{code}To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}}, because per Hiera best practices, only variables not subject to evaluation-order dependencies or calling scope should be referenced in hiera.yaml. Thus, the following hierarchy configuration should fail:{code:yaml}---hierarchy: - name: "Example failure"lookup_key: getvarpath: "class_scope_variable.hieradata"{code}The error emitted should look something like:{code}ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.{code}Configuring the backend to allow the use of non-topscope variables would look like this:{code:yaml}---hierarchy: - name: "Example failure (bypassed) "lookup_key: getvarpath: "class_scope_variable.hieradata"options: topscope_variables_only: false{code}
Add Comment
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like {code:yaml} --- hierarchy: - name: " ServiceNow hiera Node-specific Hiera data , via trusted and via fact "lookup_key: getvar path paths : - " trusted.external.servicenow. hiera_data hieradata" - "trusted.external.unicorn.hieradata" - "facts.hieradata" - name: " User fact Hiera data Some yaml stuff " lookup_key data_hash : getvar yaml path paths : facts - "environments/%{trusted . hiera extensions . data pp_environment}.yaml" - "common.yaml" {code} where for the {{ getvar}} Hiera layer, {{ path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys , e . g. For example: {code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"} {code} To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}} since those are the top-level , because per Hiera best practices, only variables that we've deemed safe not subject to use evaluation-order dependencies or calling scope should be referenced in hiera . All other paths will raise an error yaml . Thus, the following hierarchy configuration should fail: People who want to use other, unsafe top {code:yaml} - level variables can do so via an --hierarchy: - name: "Example failure"lookup_key: getvarpath: "class_scope_variable.hieradata" { code}The error emitted should look something like: { unsafe code } } ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option , which dictates that variables consulted must exist and be in topscope . If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.{code} Configuring that the backend to allow the use of non-topscope variables would look something like this: {code :yaml } - --hierarchy: - name: " unsafe hiera data Example failure " lookup_key: getvar path: foo "class_scope_variable . bar hieradata" unsafe options : true topscope_variables_only: false {code}
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code:yaml}---hierarchy: - name: "ServiceNow hiera data"lookup_key: getvarpath: trusted.external.servicenow.hiera_data - name: "User fact Hiera data" lookup_key: getvar path: facts.hiera.data{code}where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g.{code:json}{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"} {code} {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error.People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like{code}- name: "unsafe hiera data" lookup_key: getvar path: foo.bar unsafe: true{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code :yaml } --- hierarchy: - name: "ServiceNow hiera data"lookup_key: getvarpath: trusted.external.servicenow.hiera_data - name: "User fact Hiera data"lookup_key: getvarpath: facts.hiera.data {code}where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g.{code: java json }{ "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"} {code} {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error.People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like{code}- name: "unsafe hiera data" lookup_key: getvar path: foo.bar unsafe: true{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like{code}- --hierarchy: - name: "ServiceNow hiera data" lookup_key: getvar path: trusted.external.servicenow.hiera_data - name: "User fact Hiera data"lookup_key: getvarpath: facts.hiera.data {code}where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g.{code:java} { "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"} {code} {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error.People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like{code}- name: "unsafe hiera data" lookup_key: getvar path: foo.bar unsafe: true{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10478) Add getvar Hiera backend
Title: Message Title
Reid Vandewiele updated an issue
Puppet / PUP-10478
Add getvar Hiera backend
Change By:
Reid Vandewiele
Configuring this would look something like {code} - name: "ServiceNow hiera data" lookup_key: getvar path: trusted.external.servicenow.hiera_data{code} where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value whose key you're looking up containing hiera lookup keys, e . g. { code:java} { "cygwin::enable": false, "ntp::servers": ["time.nist.gov","clock.sjc.he.net" ], "ntp::restrict": ["127.0.0.1" ], "ntp::disable_monitor": true, "resource_api::agent::api_version": "1.4.2", "resource_api::server::api_version": "1.4.2", "syslog::server": "rsyslog.ops.puppetlabs.net"} {code} {{ path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error.People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like {code} - name: "unsafe hiera data" lookup_key: getvar path: foo.bar unsafe: true{code}
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10292) external trusted data is executed for file_content requests
Title: Message Title Reid Vandewiele assigned an issue to Unassigned Puppet / PUP-10292 external trusted data is executed for file_content requests Change By: Reid Vandewiele Assignee: Reid Vandewiele Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.346028.1581515567000.50984.1588193281023%40Atlassian.JIRA.
Jira (PUP-10364) Add new ppRegCertExt OID "pp_owner"
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10364 Add new ppRegCertExt OID "pp_owner" Change By: Reid Vandewiele Release Notes: Enhancement Release Notes Summary: Provide a new certificate extension OID and short-name, pp_owner Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.349188.1583867523000.38307.1586901420032%40Atlassian.JIRA.
Jira (PUP-10368) Checksums not validated when downloading file http(s):// sources
Title: Message Title Reid Vandewiele commented on PUP-10368 Re: Checksums not validated when downloading file http(s):// sources This bug doesn't involve redirects or HEAD requests (manually supplied checksum), so I think it's probably not related to PUP-8300. The HTTP(S) epic does look like a good home for it though. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.349628.1584115737000.12338.1584118560033%40Atlassian.JIRA.
Jira (PUP-10368) Checksums not validated when downloading file http(s):// sources
Title: Message Title
Reid Vandewiele created an issue
Puppet / PUP-10368
Checksums not validated when downloading file http(s):// sources
Issue Type:
Bug
Assignee:
Unassigned
Created:
2020/03/13 9:08 AM
Priority:
Normal
Reporter:
Reid Vandewiele
Puppet Version: all OS Name/Version: all When using a File resource with an http(s) source type and a checksum_value, Puppet does not validate that the content it downloads and puts into place matches the mandated checksum. This can cause Puppet to repeatedly re-download the file on every run, constantly reporting success, when in fact it is failing to retrieve the expected content. Assume the following manifest.
file { '/tmp/file':
ensure => file,
source => 'http://httpstat.us/200',
checksum => 'sha256',
checksum_value => 'ea8fac7c65fb589b0d53560f5251f74f9e9b243478dcb6b3ea79b5e36449c8d9',
Jira (PUP-10364) Add new ppRegCertExt OID "pp_owner"
Title: Message Title Reid Vandewiele created an issue Puppet / PUP-10364 Add new ppRegCertExt OID "pp_owner" Issue Type: Improvement Assignee: Unassigned Created: 2020/03/10 12:12 PM Priority: Normal Reporter: Reid Vandewiele In Puppet-as-a-Service use cases it is common for individual nodes to be owned by various different tenant, or customer, teams. None of the registered ppRegCertExt OIDs we currently provide cleanly allow generic specification of an owner in this context. Provide a new OID, pp_owner, which generically fits this use case. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10216) Failure to fetch node definition may result in bad pluginsync and cascading failure
Title: Message Title Reid Vandewiele updated an issue Puppet / PUP-10216 Failure to fetch node definition may result in bad pluginsync and cascading failure Change By: Reid Vandewiele Method Found: Needs Assessment Customer Feedback Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.341077.1578089491000.11988.1578089580239%40Atlassian.JIRA.
Jira (PUP-10216) Failure to fetch node definition may result in bad pluginsync and cascading failure
Title: Message Title Reid Vandewiele created an issue Puppet / PUP-10216 Failure to fetch node definition may result in bad pluginsync and cascading failure Issue Type: Bug Assignee: Unassigned Created: 2020/01/03 2:11 PM Priority: Normal Reporter: Reid Vandewiele Puppet Version: all Early on in the agent run process, Puppet calls an indirector to choose an environment to pluginsync against. If anything goes wrong with that call, pretty much anything at all, Puppet will pluginsync against the agent-configured environment (usually "production") and spit out a warning. Warning: Unable to fetch my node definition, but the agent run will continue Desired Behavior: Puppet should pluginsync against the last-used environment, or alternatively, fail fast. At customer sites where "production" is not used, pluginsyncing against what is probably an empty environment has been observed to add ten minutes+ of time to a Puppet agent run, as it first deletes all of its plugins, and then has to re-sync them fresh from the correct environment after it switches over. This can negatively affect the delivery of changes that need to be performed within a change control window. Actual Behavior: Puppet pluginsyncs against production, or the environment value set with --environment or in puppet.conf. Reproduction: Create an environment "r1" and "production" Pin a test agent to "r1"
Jira (PUP-9570) Catalog failure on first run due to pluginsync and environment switch
Title: Message Title
Reid Vandewiele commented on PUP-9570
Re: Catalog failure on first run due to pluginsync and environment switch
Just for fun, I tried to use the example code to create a workable pluginsync_environment fact. It doesn't seem to quite work though. I created a simple site.pp with this code in it:
if $facts['pluginsync_environment'] != $server_facts['environment'] {
notify { 'STOP':
message => "Now what? pluginsync_environment=${facts['pluginsync_environment']}, server_facts.environment=${server_facts['environment']}",
}
}
else {
notify { 'PROCEED':
message => "Safe to compile catalog. pluginsync_environment=${facts['pluginsync_environment']}, server_facts.environment=${server_facts['environment']}",
}
}
Then, I observed what happened when trying to switch from a starting environment of env_1 to a new target environment of env_2.
Jira (BOLT-864) Always upload code from bolt when running a task via Orchestrator
Title: Message Title Reid Vandewiele commented on BOLT-864 Re: Always upload code from bolt when running a task via Orchestrator Opened issue in Github as https://github.com/puppetlabs/bolt/issues/1190. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.277376.1537894192000.80082.1567118100248%40Atlassian.JIRA.
Jira (PUP-7814) HTTPS file sources with non-puppet-trusted certs can't be used
Title: Message Title Reid Vandewiele commented on PUP-7814 Re: HTTPS file sources with non-puppet-trusted certs can't be used This would be specific to the File resource provider's use of Net::HTTP, and would not extend to any of the Puppet agent's core PKI trust configuration, yes? That is, we wouldn't end up being able to trust and download a catalog from a "master" with a cert signed by Let's Encrypt or anything like that Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.203766.1501710671000.47922.1565270400333%40Atlassian.JIRA.
Jira (BOLT-1382) wait_until_available() incorrect results using Orchestrator
Title: Message Title Reid Vandewiele commented on BOLT-1382 Re: wait_until_available() incorrect results using Orchestrator Filed a starter PR at https://github.com/puppetlabs/bolt/pull/1031. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.312114.1560285784000.42975.1560290280518%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (BOLT-1382) wait_until_available() incorrect results using Orchestrator
Title: Message Title Reid Vandewiele commented on BOLT-1382 Re: wait_until_available() incorrect results using Orchestrator It kinda looks like we operate in batches, and don't have any concept of some targets in a batch being unavailable while others in the same batch are. https://github.com/puppetlabs/bolt/blob/377350ecf2a9e193b1013fb05554cd12e7418245/lib/bolt/executor.rb#L284-L300 Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.312114.1560285784000.42927.1560287940220%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (BOLT-1382) wait_until_available() incorrect results using Orchestrator
Title: Message Title
Reid Vandewiele created an issue
Puppet Task Runner / BOLT-1382
wait_until_available() incorrect results using Orchestrator
Issue Type:
Bug
Assignee:
Unassigned
Created:
2019/06/11 1:43 PM
Priority:
Normal
Reporter:
Reid Vandewiele
The wait_until_available() function seems to all-or-nothing succeed/fail when using --transport pcp. See below for an example. The same test plan will return the expected results when run over ssh.
[root@pe-xl-core-0 bolt-project]# cat site-modules/util/plans/test.pp
plan util::test (
TargetSpec $nodes,
) {
$connection_results = wait_until_available(get_targets($nodes),
wait_time => 0,
Jira (BOLT-1366) Allow plans to return uniform Type, including on error
Title: Message Title
Reid Vandewiele updated an issue
Puppet Task Runner / BOLT-1366
Allow plans to return uniform Type, including on error
Change By:
Reid Vandewiele
It isn't possible today for a plan author to formally signal to callers that the plan failed (surfacing an error without requiring the caller to provide distinct (and not bound by convention) code paths for "ok" vs. "error". E.g.{code}$results = case ($plan_result = run_plan('mymodule::myplan', '_catch_errors' => true,)) { Error: { $plan_result.details['result_set'] } default: { $plan_result }}{code}Ideally, a caller should be able to expect a consistent return type regardless of whether or not the plan failed; just like a caller can always expect an object of type ResultSet to be returned from {{run_task}}.Allow for a moment that a plan _always_ returns an object of type PlanResult. If this is true, we can generalize success/failure based on {{$pr.ok}}, just as ResultSet handling can be based on {{$rs.ok}}. If a PlanResult is not ok (it failed), we can generalize access to the error raised via e.g. {{$pr.error}}. The PlanResult object can define a standard interface to data returned by a plan, such as defining a {{.value}} method, and the index method ({{.[]}}) for easy access to returned data keys.A caller could then simplify their code to e.g.{code}$result = run_plan('mymodule::myplan', _catch_errors => true,)run_task('mymodule::next_step, $result.value.ok_set){code}OR{code}$result['summary']{code}OR{code}if !$result.ok { # handle it}{code}...etc., depending on what the plan author chooses to return.Being able to rely on the datatype returned by {{run_plan}} greatly simplifies how plan authors can call and deal with the results from running other plans. One way of doing this might be to create a new type, PlanResult, which akin to ResultSet would come with a standard set of methods to interact with it. We could additionally define PlanResult::Error, which is still a PlanResult, but adds a {{.error}} method to access an included Error object, and would be compatible with code like {{case $result \{ Error: \{ ... \} \}}}.We could extend {{fail_plan}} to return a PlanResult::Error, if given a PlanResult return value and an Error. Plan authors could then choose to create well-formed plans in an opt-in way.If we didn't have to deal with existing plans I'd say just always return a PlanResult, with a little more automation around it and less effort from the plan author. Maybe it's a better enough experience to make a change in 2.0.
Jira (BOLT-1366) Allow plans to return uniform Type, including on error
Title: Message Title Reid Vandewiele updated an issue Puppet Task Runner / BOLT-1366 Allow plans to return uniform Type, including on error Change By: Reid Vandewiele Summary: Overhaul user experience handling plan errors Allow plans to return uniform Type, including on error Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.311455.155977910.37100.1559840700328%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (BOLT-1366) Overhaul user experience handling plan errors
Title: Message Title
Reid Vandewiele created an issue
Puppet Task Runner / BOLT-1366
Overhaul user experience handling plan errors
Issue Type:
Improvement
Assignee:
Unassigned
Created:
2019/06/05 4:58 PM
Priority:
Normal
Reporter:
Reid Vandewiele
It isn't possible today for a plan author to formally signal to callers that the plan failed (surfacing an error without requiring the caller to provide distinct (and not bound by convention) code paths for "ok" vs. "error". E.g.
$results = case ($plan_result = run_plan('mymodule::myplan',
'_catch_errors' => true,
)) {
Error: { $plan_result.details['result_set'] }
default: { $plan_result }
}
Jira (BOLT-1363) Singular version of TargetSpec data type
Title: Message Title Reid Vandewiele created an issue Puppet Task Runner / BOLT-1363 Singular version of TargetSpec data type Issue Type: Improvement Assignee: Unassigned Created: 2019/06/04 2:52 PM Priority: Normal Reporter: Reid Vandewiele TargetSpec accepts a single Variant[String, Target] OR an Array of the same, and is understood to best be treated as if it could refer to multiple endpoints. Sometimes a plan (or task) parameter should accept a single target only. At present, there is no built-in data type to support that. Create and provide such a data type. SingleTargetSpec ? Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Jira (BOLT-1337) Make .filter_set called on a ResultSet return a ResultSet
Title: Message Title Reid Vandewiele updated an issue Puppet Task Runner / BOLT-1337 Make .filter_set called on a ResultSet return a ResultSet Change By: Reid Vandewiele Summary: Make . filter filter_set called on a ResultSet return a ResultSet Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.309878.1558622066000.23897.1559083920063%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (BOLT-1337) Make .filter called on a ResultSet return a ResultSet
Title: Message Title
Reid Vandewiele updated an issue
Puppet Task Runner / BOLT-1337
Make .filter called on a ResultSet return a ResultSet
Change By:
Reid Vandewiele
This should work. It doesn’t, because filter called on a ResultSet returns an array.{code}# Identify those nodes that need to be rebooted.$reboot_targets = $patch_results.ok_set.filter |$result| { $result['reboot_flag'] == true}.targets{code}Make a new method, {{. filter filter_set }} , which when called on a ResultSet return returns a ResultSet. Making a new method instead of just fixing {{.filter}} because of concerns over breaking / backwards incompatible changes.
Add Comment
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this
Jira (BOLT-1337) Make .filter called on a ResultSet return a ResultSet
Title: Message Title
Reid Vandewiele created an issue
Puppet Task Runner / BOLT-1337
Make .filter called on a ResultSet return a ResultSet
Issue Type:
Improvement
Assignee:
Unassigned
Created:
2019/05/23 7:34 AM
Priority:
Normal
Reporter:
Reid Vandewiele
This should work. It doesn’t, because filter called on a ResultSet returns an array.
# Identify those nodes that need to be rebooted.
$reboot_targets = $patch_results.ok_set.filter |$result| {
$result['reboot_flag'] == true
}.targets
Make .filter called on a ResultSet return a ResultSet.
Jira (PUP-9570) Catalog failure on first run due to pluginsync and environment switch
Title: Message Title Reid Vandewiele commented on PUP-9570 Re: Catalog failure on first run due to pluginsync and environment switch Josh Cooper I spent a fair amount of time trying to figure out how to build a pluginsync_environment fact with no luck. Any ideas or pointers? As near as I can tell, what environment a pluginsync is actually performed against is not saved or referenced anywhere a normal fact could get at it, nor is the information obviously retrievable from the configurer object (which I don't know if can be accessed by a fact). Talking to Branan Riley and Nick Lewis last night, there are a number of ideas about how to fix it as a bug, but nothing reasonably accessible to a user. One idea Nick had (I think it was Nick) was for the configurer to do something more intelligent with a failed compilation error. However, at the moment a failure doesn't convey an environment, so to go that route we'd need to pass more data back (e.g. an optional environment key) in the HTTP error response. Seems like it would be cleanest though to just bake in a pluginsync environment value to the agent as you suggest above. We would then want to follow it up by short-circuiting catalog compilation if the pluginsync environment doesn't match the compilation environment. Seems like that's all Puppet work and not user fixable. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send
Jira (PUP-9254) Agent Functions - add eval() function
Title: Message Title Reid Vandewiele commented on PUP-9254 Re: Agent Functions - add eval() function Another use case for this: plans. Allowing users to supply Puppet code to a plan to be part of an apply block. Opens up the ability to create and use powerful content-development plan-based tools. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
