Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title Nacho Barrientos commented on PDB-4830 Re: Big performance impact when upgrading to PuppetDB5/6 It would have been nice if, after three years of basically zero communication and finally closing this issue as "won't fix", (at least) some pointers to the "current work" had been left in the ticket. I think it's the bare minimum. Add Comment This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.367361.1595856224000.1062.1687198740037%40Atlassian.JIRA.
Jira (FACT-3115) virt-what-1.22-1 changes virtual=kvm to virtual=redhat
Title: Message Title Nacho Barrientos commented on FACT-3115 Re: virt-what-1.22-1 changes virtual=kvm to virtual=redhat This is also affecting CentOS Stream 8 now. I've contributed a patch: https://github.com/puppetlabs/facter/pull/2485 Add Comment This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.444600.1650614566000.16844.1650873900038%40Atlassian.JIRA.
Jira (PUP-11451) Make collecting exported resources optional
Title: Message Title
Nacho Barrientos commented on PUP-11451
Re: Make collecting exported resources optional
Thinking twice for us it'd be even better to stop the compilation than just collecting "nothing". Something like this seems to work:
require 'puppet/indirector/none'
class Puppet::Resource::Fail < Puppet::Indirector::None
def find(request)
raise Puppet::Indirector::ValidationError, _("Collecting exported resources is disabled.")
end
def search(request)
raise Puppet::Indirector::ValidationError, _("Collecting exported resources is disabled.")
end
end
which would result in something like:
Jira (PUP-11451) Make collecting exported resources optional
Title: Message Title Nacho Barrientos commented on PUP-11451 Re: Make collecting exported resources optional Hi Charlie, That's great. As long as this does not prevent catalogs and resources from being stored in PuppetDB it could definitely fly for us. Would you be willing to distribute puppet/indirector/resource/none.rb as part of Puppet? Add Comment This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.434328.1643983687000.80.1648126920067%40Atlassian.JIRA.
Jira (PUP-11451) Make collecting exported resources optional
Title: Message Title Nacho Barrientos commented on PUP-11451 Re: Make collecting exported resources optional Hi Charlie, Ben, Thanks for your input. Our use case is a multi tenant configuration management infrastructure based on Puppet where somebody could accidentally or maliciously export a resource to be collected by somebody else that could lead to broken or malicious configuration being applied. In our case, the usage of exported resources can normally easily be replaced by PuppetDB queries so we'd like to disable resource collection to reduce the risks. Replying to your comment Charlie, unfortunately we as admins don't have full control on the code that our masters compile and our users are not enforced (only encouraged) to implement lint checks. Ben, I understand your concerns but I don't think bad communication from the admins justify not having the functionality. In my opinion, as long as the defaults are not modified and the setting well documented, it should be okay. We cannot influence how configuration changes on the masters are communicated everywhere Anyway, thanks for your input and please let us know at your earliest convenience if you'd like us to contribute a change request. Otherwise, we'll most likely carry a local patch to disable exported resources collection. Thanks again. /cc traylenator Add Comment This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the
Jira (PUP-11451) Make collecting exported resources optional
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-11451 Make collecting exported resources optional Issue Type: New Feature Assignee: Unassigned Created: 2022/02/04 6:08 AM Priority: Normal Reporter: Nacho Barrientos According to the documentation, storing catalogs and facts in PuppetDB and using exported resources are two actions that are governed by the same configuration option (storeconfigs): Whether to store each client's configuration, including catalogs, facts, and related data. This also enables the import and export of resources in the Puppet language - a mechanism for exchange resources between nodes. https://puppet.com/docs/puppet/7/configuration.html#storeconfigs It'd be useful for us in our deployment if those two things were not tied to each other. In other words, if it was possible to keep "storing configs" but exported resource collectors were ignored. This is rather cheap to achieve by introducing a new configuration option and patching the exported resources collector, something like: @@ -18,7 +18,7 @@ class Puppet::Pops::Evaluator::Collectors::ExportedCollector < Puppet::Pops::Eva # Ensures that storeconfigs is present before calling AbstractCollector's # evaluate method
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title Nacho Barrientos commented on PDB-4830 Re: Big performance impact when upgrading to PuppetDB5/6 Not at all, we're stuck in PuppetDB4 due to this Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.367361.1595856224000.155793.1614756300023%40Atlassian.JIRA.
Jira (FACT-2134) Facter should distinguish between CentOS 8 and CentOS 8 Stream
Title: Message Title Nacho Barrientos commented on FACT-2134 Re: Facter should distinguish between CentOS 8 and CentOS 8 Stream A (community) proposal https://github.com/puppetlabs/facter/pull/2291 (only facter-ng) Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.336204.1574351826000.146263.1613733300080%40Atlassian.JIRA.
Jira (PUP-10855) Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog
Title: Message Title Nacho Barrientos commented on PUP-10855 Re: Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog Thanks for confirming the loaders bit and for the hint about the catalog retrieval. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.383328.1610703917000.118120.1611075840184%40Atlassian.JIRA.
Jira (PUP-10855) Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-10855 Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog Issue Type: Bug Assignee: Unassigned Created: 2021/01/15 1:45 AM Priority: Normal Reporter: Nacho Barrientos Puppet Version: 6.15.0 Puppet Server Version: Does not apply OS Name/Version: CentOS8 Hi, Opening this for discussion mainly, it might not be a bug. We have an in-house debugging tool that downloads a catalog from a master and persists it to disk (serialised as JSON) to further inspection later on by another tool. Inspired by this, the code is something like this: require 'puppet' require 'puppet/face' Puppet.initialize_settings Puppet.initialize_facts Puppet.settings.preferred_run_mode = :agent
Jira (PUP-10654) Supplying data of mixed data types (Variant) via Hiera (YAML backend)
Title: Message Title Nacho Barrientos commented on PUP-10654 Re: Supplying data of mixed data types (Variant) via Hiera (YAML backend) Thanks both for your input. I agree it makes sense to respect the native data types of the data backend and use a richer backend for these cases. Technically speaking is the correct path to take, however in our environment introducing an extra backend would add complexity for our users, more training to do and more things to maintain so I think we're just going to split the parameter in this particular case. It's a pity given the existence of a type system but it's the simplest approach at the moment so we can carry on with the upgrade. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.370870.1598950851000.29844.1599118620029%40Atlassian.JIRA.
Jira (PUP-10654) Supplying data of mixed data types (Variant) via Hiera (YAML backend)
Title: Message Title Nacho Barrientos commented on PUP-10654 Re: Supplying data of mixed data types (Variant) via Hiera (YAML backend) Thanks for you input! I moved to PUP project as all of Hiera5 is implemented in the puppet project. Sorry about that, I was not sure Can you confirm that adding Regexp to https://github.com/puppetlabs/puppet/blob/17b2735731f6468aef92a5540b35292e056726e3/lib/puppet/functions/yaml_data.rb#L25 solves the issue? Yes, the catalog compiles and the catalog diff is empty Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.370870.1598950851000.28877.1599035040037%40Atlassian.JIRA.
Jira (HI-618) Supplying data of mixed data types (Variant) via Hiera (YAML backend)
Title: Message Title Nacho Barrientos created an issue Hiera / HI-618 Supplying data of mixed data types (Variant) via Hiera (YAML backend) Issue Type: Story Assignee: Unassigned Created: 2020/09/01 2:00 AM Priority: Normal Reporter: Nacho Barrientos Puppet Version: 6.15.0 Puppet Server Version: 6.11.1 OS Name/Version: CentOS7 Hi, We're upgrading to Puppet6 and we encountered an interesting situation we'd like to discuss with you. Please note that as explained below this is not a blocker for us but we believe it's worth discussing it to know what's the best path to take in terms of patching our code. We've got this in our code base: class foo::bar ( Array[Variant[String, Regexp]] $my_param, ... Allowing us to use a single parameter to later on apply configuration depending on the type of the element of the collection: $_directories =
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title
Nacho Barrientos commented on PDB-4830
Re: Big performance impact when upgrading to PuppetDB5/6
Thanks once again for your time.
Is it important for your usecase that only the queries like /pdb/query/v4/facts// are performant, or do you also make use of queries that don't provide the value, like /pdb/query/v4/facts/?
It'd be a good starting point but to be honest it would not be enough. Most of our queries (generated by the module above) talk directly to /pdb/query/v4/facts passing a query, for example: "Get the value of the ipaddress fact for nodes whose fact fact1 has x as value, fact fact2 has y and fact fact3 has z": via the query_nodes() function:
query_nodes('fact1="x" and fact2="y" and fact3="z"', ipaddress)
translates into:
GET pdb/query/v4/facts --query
'["extract",["value"],["and",["and",["in","certname",["extract","certname",["select_fact_contents",["and",["=","path",["fact2"]],["=","value","y"],["in","certname",["extract","certname",["select_fact_contents",["and",["=","path",["fact1"]],["=","value","x"],["in","certname",["extract","certname",["select_fact_contents",["and",["=","path",["fact3"]],["=","value","z"]],["or",["=","name","ipaddress"'
# PuppetDB 4
real0m0.471s
# PuppetDB 5
real0m16.302s
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title Nacho Barrientos commented on PDB-4830 Re: Big performance impact when upgrading to PuppetDB5/6 Hi Austin Blatt, Thanks for taking a look at the ticket and for coming back to me. For the use-case of retrieving arbitrary fact values for simple fact queries, using the inventory is indeed much faster: # PuppetDB 5 GET raw pdb/query/v4/facts/fqdn/node1.example.org "node1.example.org" 0m8.358s # PuppetDB 5 GET raw pdb/query/v4/inventory --query 'query=["=", "facts.fqdn", "node1.example.org"]' | jq '.[].certname' "node1.example.org" 0m0.236s which is comparable to the performance we have in production:
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title
Nacho Barrientos updated an issue
PuppetDB / PDB-4830
Big performance impact when upgrading to PuppetDB5/6
Change By:
Nacho Barrientos
Hi,We're preparing the upgrade of a rather big PuppetDB instance (~40k hosts) from PuppetDB4 to PuppetDB5.There's a type of query that we do rather often (directly to the API or via the PuppetDB query module when compiling catalogs) which is to select _certnames_ for which a certain fact has a given value. The simplest way to reproduce this scenario is to issue something like:{noformat}GET pdb/query/v4/facts/factname/factvalue{noformat}In the 'old' PuppetDB world (<=PuppetDB4), where the fact values were normalised in the _facts_ table we added an index like this:{noformat}"facts_value_string_idx" btree (value_string){noformat}which for a query on the database the size above for nodes with the fact 'fqdn' _fqdn_ set to 'node1 _node1 .example. org` org_ performed pretty well thanks to the index:{noformat}[ { "certname": "node1.example.org", "environment": "production", "name": "fqdn", "value": "node1.example.org" } ]{noformat}*real 0m0.319s*The index is used avoiding a full table scan and the performance is acceptable. We normally query for more complex fact combinations using several string fact values but for illustration purposes the above is good enough.However, in the new world (PuppetDB5 and beyond) it seems that the _facts_ table is gone and now the fact values are in two columns in the _factsets_ table (_stable_ and _volatile_) of type _jsonb_.With the same HW setup, PostgreSQL configuration, OS configuration and a copy of the data (we're populating both instances at the same time using an extra _submit_only_server_urls_ in the terminus), the query above produces the same results as expected but it is much slower:*real 0m7.403s*We could easily reproduce the (bad) ratio PDB4/PDB5 with queries of different complexities.Looking at the possible indices that this query could use we found this one:{noformat}"idx_factsets_jsonb_merged" gin ((stable || volatile) jsonb_path_ops){noformat}However, taking a look at the generated SQL, PuppetDB seems to be using _jsonb_extract_path()_ to then compare to the value being looked for. However, this approach cannot benefit from the index above.I have the whole query if needed but to illustrate the issue what I did was to analyze the query and extract the juiciest subquery that could show the problem which is:{noformat}explain analyze SELECT certname FROM factsets WHERE jsonb_extract_path(stable||volatile, 'fqdn') = '"node1.example.org"'::jsonb; QUERY PLAN-- Seq Scan on factsets (cost=0.00...63 rows=182 width=20) (actual time=2708.687..5292.499 rows=1 loops=1) Filter: (jsonb_extract_path((stable || volatile), VARIADIC '\{fqdn}'::text[]) = '"node1.example.org"'::jsonb) Rows Removed by Filter: 36492 Execution Time: 5292.528 ms{noformat}As you can see it's a full table scan.
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title
Nacho Barrientos updated an issue
PuppetDB / PDB-4830
Big performance impact when upgrading to PuppetDB5/6
Change By:
Nacho Barrientos
Hi,We're preparing the upgrade of a rather big PuppetDB instance (~40k hosts) from PuppetDB4 to PuppetDB5.There's a type of query that we do rather often (directly to the API or via the PuppetDB query module when compiling catalogs) which is to select _certnames_ for which a certain fact has a given value. The simplest way to reproduce this scenario is to issue something like:{noformat}GET pdb/query/v4/facts/factname/factvalue{noformat}In the 'old' PuppetDB world (<=PuppetDB4), where the fact values were normalised in the _facts_ table we added an index like this:{noformat}"facts_value_string_idx" btree (value_string){noformat}which for a query on the database the size above for nodes with the fact 'fqdn' set to 'node1.example.org` performed pretty well thanks to the index:{noformat}[ { "certname": "node1. cern example . ch org ", "environment": "production", "name": "fqdn", "value": "node1. cern example . ch org " } ]{noformat}*real 0m0.319s*The index is used avoiding a full table scan and the performance is acceptable. We normally query for more complex fact combinations using several string fact values but for illustration purposes the above is good enough.However, in the new world (PuppetDB5 and beyond) it seems that the _facts_ table is gone and now the fact values are in two columns in the _factsets_ table (_stable_ and _volatile_) of type _jsonb_.With the same HW setup, PostgreSQL configuration, OS configuration and a copy of the data (we're populating both instances at the same time using an extra _submit_only_server_urls_ in the terminus), the query above produces the same results as expected but it is much slower:*real 0m7.403s*We could easily reproduce the (bad) ratio PDB4/PDB5 with queries of different complexities.Looking at the possible indices that this query could use we found this one:{noformat}"idx_factsets_jsonb_merged" gin ((stable || volatile) jsonb_path_ops){noformat}However, taking a look at the generated SQL, PuppetDB seems to be using _jsonb_extract_path()_ to then compare to the value being looked for. However, this approach cannot benefit from the index above.I have the whole query if needed but to illustrate the issue what I did was to analyze the query and extract the juiciest subquery that could show the problem which is:{noformat}explain analyze SELECT certname FROM factsets WHERE jsonb_extract_path(stable||volatile, 'fqdn') = '"node1.example.org"'::jsonb; QUERY PLAN-- Seq Scan on factsets (cost=0.00...63 rows=182 width=20) (actual time=2708.687..5292.499 rows=1 loops=1) Filter: (jsonb_extract_path((stable || volatile), VARIADIC '\{fqdn}'::text[]) = '"node1.example.org"'::jsonb) Rows Removed by Filter: 36492 Execution Time: 5292.528 ms{noformat}As you can see it's a full table scan. However,
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title
Nacho Barrientos updated an issue
PuppetDB / PDB-4830
Big performance impact when upgrading to PuppetDB5/6
Change By:
Nacho Barrientos
Hi,We're preparing the upgrade of a rather big PuppetDB instance (~40k hosts) from PuppetDB4 to PuppetDB5.There's a type of query that we do rather often (directly to the API or via the PuppetDB query module when compiling catalogs) which is to select _certnames_ for which a certain fact has a given value. The simplest way to reproduce this scenario is to issue something like:{noformat}GET pdb/query/v4/facts/factname/factvalue{noformat}In the 'old' PuppetDB world (<=PuppetDB4), where the fact values were normalised in the _facts_ table we added an index like this:{noformat}"facts_value_string_idx" btree (value_string){noformat}which for a query on the database the size above for nodes with the fact 'fqdn' set to 'node1.example.org` performed pretty well thanks to the index:{noformat}[ { "certname": "node1.cern.ch", "environment": "production", "name": "fqdn", "value": "node1.cern.ch" } ]{noformat}*real 0m0.319s*The index is used avoiding a full table scan and the performance is acceptable. We normally query for more complex fact combinations using several string fact values but for illustration purposes the above is good enough.However, in the new world (PuppetDB5 and beyond) it seems that the _facts_ table is gone and now the fact values are in two columns in the _factsets_ table (_stable_ and _volatile_) of type _jsonb_.With the same HW setup, PostgreSQL configuration, OS configuration and a copy of the data (we're populating both instances at the same time using an extra _submit_only_server_urls_ in the terminus), the query above produces the same results as expected but it is much slower:*real 0m7.403s*We could easily reproduce the (bad) ratio PDB4/PDB5 with queries of different complexities.Looking at the possible indices that this query could use we found this one:{noformat}"idx_factsets_jsonb_merged" gin ((stable || volatile) jsonb_path_ops){noformat}However, taking a look at the generated SQL, PuppetDB seems to be using _jsonb_extract_path()_ to then compare to the value being looked for. However, this approach cannot benefit from the index above.I have the whole query if needed but to illustrate the issue what I did was to analyze the query and extract the juiciest subquery that could show the problem which is:{noformat}explain analyze SELECT certname FROM factsets WHERE jsonb_extract_path(stable||volatile, 'fqdn') = '"node1.example.org"'::jsonb; QUERY PLAN-- Seq Scan on factsets (cost=0.00...63 rows=182 width=20) (actual time=2708.687..5292.499 rows=1 loops=1) Filter: (jsonb_extract_path((stable || volatile), VARIADIC '\{fqdn}'::text[]) = '"node1.example.org"'::jsonb) Rows Removed by Filter: 36492 Execution Time: 5292.528 ms{noformat}As you can see it's a full table scan. However, if the query
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title
Nacho Barrientos updated an issue
PuppetDB / PDB-4830
Big performance impact when upgrading to PuppetDB5/6
Change By:
Nacho Barrientos
Hi,We're preparing the upgrade of a rather big PuppetDB instance (~40k hosts) from PuppetDB4 to PuppetDB5.There's a type of query that we do rather often (directly to the API or via the PuppeDB PuppetDB query module when compiling catalogs) which is to select _certnames_ forwhich a certain fact has a given value. The simplest way to reproduce this scenario is to issue something like:{noformat}GET pdb/query/v4/facts/factname/factvalue{noformat}In the 'old' PuppetDB world (<=PuppetDB4), where the fact values were normalised in the _facts_ table we added an index like this:{noformat}"facts_value_string_idx" btree (value_string){noformat}which for a query on the database the size above for nodes with the fact 'fqdn' set to 'node1.example.org` performed pretty well thanks to the index:{noformat}[ { "certname": "node1.cern.ch", "environment": "production", "name": "fqdn", "value": "node1.cern.ch" } ]{noformat}*real 0m0.319s*The index is used avoiding a full table scan and the performance is acceptable. We normally query for more complex fact combinations using several string fact values but for illustration purposes the above is good enough.However, in the new world (PuppetDB5 and beyond) it seems that the _facts_ table is gone and now the fact values are in two columns in the _factsets_ table (_stable_ and _volatile_) of type _jsonb_.With the same HW setup, PostgreSQL configuration, OS configuration and a copy of the data (we're populating both instances at the same time using an extra _submit_only_server_urls_ in the terminus), the query above produces the same results as expected but it is much slower:*real 0m7.403s*We could easily reproduce the (bad) ratio PDB4/PDB5 with queries of different complexities.Looking at the possible indices that this query could use we found this one:{noformat}"idx_factsets_jsonb_merged" gin ((stable || volatile) jsonb_path_ops){noformat}However, taking a look at the generated SQL, PuppetDB seems to be using _jsonb_extract_path()_ to then compare to the value being looked for. However, this approach cannot benefit from the index above.I have the whole query if needed but to illustrate the issue what I did was to analyze the query and extract the juiciest subquery that could show the problem which is:{noformat}explain analyze SELECT certname FROM factsets WHERE jsonb_extract_path(stable||volatile, 'fqdn') = '"node1.example.org"'::jsonb; QUERY PLAN-- Seq Scan on factsets (cost=0.00...63 rows=182 width=20) (actual time=2708.687..5292.499 rows=1 loops=1) Filter: (jsonb_extract_path((stable || volatile), VARIADIC '\{fqdn}'::text[]) = '"node1.example.org"'::jsonb) Rows Removed by Filter: 36492 Execution Time: 5292.528 ms{noformat}As you can see it's a full table scan. However, if
Jira (PDB-4830) Big performance impact when upgrading to PuppetDB5/6
Title: Message Title Nacho Barrientos created an issue PuppetDB / PDB-4830 Big performance impact when upgrading to PuppetDB5/6 Issue Type: Bug Assignee: Unassigned Created: 2020/07/27 6:23 AM Priority: Normal Reporter: Nacho Barrientos Hi, We're preparing the upgrade of a rather big PuppetDB instance (~40k hosts) from PuppetDB4 to PuppetDB5. There's a type of query that we do rather often (directly to the API or via the PuppeDB query module when compiling catalogs) which is to select certnames for which a certain fact has a given value. The simplest way to reproduce this scenario is to issue something like: GET pdb/query/v4/facts/factname/factvalue In the 'old' PuppetDB world (<=PuppetDB4), where the fact values were normalised in the facts table we added an index like this: "facts_value_string_idx" btree (value_string) which for a query on the database the size above for nodes with the fact 'fqdn' set to 'node1.example.org` performed pretty well thanks to the index:
Jira (PUP-9566) Allow to send extra headers when requesting a catalog compilation
Title: Message Title Nacho Barrientos commented on PUP-9566 Re: Allow to send extra headers when requesting a catalog compilation Hi, For info we've had our proposed patch in production for a while now. We'll see if we can de-fork in the future, depending on how you implement the feature. Thanks! Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300606.1552923936000.113267.1569394800493%40Atlassian.JIRA.
Jira (PUP-5005) Puppet agent should support requiring a message when disabling
Title: Message Title Nacho Barrientos commented on PUP-5005 Re: Puppet agent should support requiring a message when disabling It still sounds like a good idea Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.92272.1438824678000.60382.1566208440155%40Atlassian.JIRA.
Jira (PUP-9566) Allow to send extra headers when requesting a catalog compilation
Title: Message Title Nacho Barrientos commented on PUP-9566 Re: Allow to send extra headers when requesting a catalog compilation Hi Rob, Thanks for coming back to us. In our case actually, as explained by David, we only need to add custom headers to the requests sent by the agents when they chat to the Puppet servers. It'd be perhaps interesting to configure the HTTP client to restrict what type of requests will be modified (by path, for instance) but indeed it's not worth to spend time on this due to how the client is implemented. This is totally an agent-oriented change proposal. In our case our agents only talk to the Puppet servers so to make use of this new feature the only thing that we had to do configuration-wise was to add the new setting to the [agent] section of the puppet.conf available on the agents and that was it. Puppet servers were left untouched as we don't need them to send any extra headers to any endpoint. I personally don't see a risk if the default behavior is to not to send anything extra so nothing changes, that indeed combined with a good description in the documentation. Obviously administrators could send anything they wanted but that's their choice. On top of that, there's nothing harmful that you could naively do (like when using a switch), users must explicitly configure what data to send in those arbitrary headers. Overloading the User-Agent could be a solution but I think it's far from being elegant. Actually we're already using that header for something else and adding non-standard data there could be painful as it'd require doing some post-processing on our monitoring pipelines to remove undesired stuff. We cannot provide much more information than what has been described in previous posts, not because we can't share more details but because there isn't much more really. We want to send metadata from the agent side that will be consumed by the ingress layer to make routing decisions. As Puppet talks HTTP we think that HTTP headers is a good fit for this. Using extended attributes in the client's certificate is unfortunately not an option for us at the moment. This new feature was born in the context of deploying our masters on Kubernetes clusters, however it could be useful in many more situations outside of the containers world. Add Comment
Jira (PUP-9566) Allow to send extra headers when requesting a catalog compilation
Title: Message Title Nacho Barrientos commented on PUP-9566 Re: Allow to send extra headers when requesting a catalog compilation Hi Rob, Thanks for your input. Does that mean that you're not going to consider the proposed patch? If so, please let us know as soon as possible as in that case unfortunately we'll have to fork. Thanks. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9566) Allow to send extra headers when requesting a catalog compilation
Title: Message Title Nacho Barrientos commented on PUP-9566 Re: Allow to send extra headers when requesting a catalog compilation I agree with David that in our case using headers is much more lightweight in terms of operational cost and maintenance. As he mentioned, we're planning on using an agent-generated header to give a "hint" to the load balancing layer and then verify on the backend that the node is actually authorised to make such request doing an RPC to a trusted source of information. Apart from the issue when changing the hostgroup, persuading the managers of the external CA to sign CSRs containing the extra header (that must be validated) might be a PITA for us. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7848) Capitals in variable names in EPP templates hang threads forever
Title: Message Title Nacho Barrientos commented on PUP-7848 Re: Capitals in variable names in EPP templates hang threads forever Thanks for (re-)fixing this. Unfortunately my e-mail to puppet-dev has not generated much interest :/ Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7848) Capitals in variable names in EPP templates hang threads forever
Title: Message Title Nacho Barrientos commented on PUP-7848 Re: Capitals in variable names in EPP templates hang threads forever BTW, I'd like to take the opportunity to ask something. Is there any kind of watchdog that can be configured at puppetserver level to automatically destroy instances that are misbehaving like these ones (perhaps based on the CPU wall time, age...)? We're already using over here max-requests-per-instance but for obvious reasons it's not useful in this case. Bugs like this one are quite dangerous and can quickly degrade the performance of a production system. The more agents exercising the bad code and triggering the issue, the faster the load goes up. There should be a way to tell Puppetserver how to protect itself. Should I open a separate ticket for this? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7848) Capitals in variable names in EPP templates hang threads forever
Title: Message Title Nacho Barrientos commented on PUP-7848 Re: Capitals in variable names in EPP templates hang threads forever BTW, I'd like to take the opportunity to ask something. Is there any kind of watchdog that can be configured at puppetserver level to automatically destroy instances that are misbehaving like these ones (perhaps based on the CPU wall time, age...)? We're already using over here max-requests-per-instance but for obvious reasons it's not useful in this case. Bugs like this one are quite dangerous and can quickly degrade the performance of a production system. The more agents exercising the bad code and triggering the issue, the faster the load goes up. There should be a way to tell Puppetserver how to protect itself. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures I looked into it, and I don't think it's that bad actually. What happens is that when the type is created, it loads all providers for that type. So although it's a "load all", it's not a "load all types". Yes, correct. This behaviour is mentioned here. Once a type is needed it's read from disk. After that, the next step in newtype() is to loadall() all the providers available for it (example – second formatted block). If the provider does not work in the master (for instance, missing dependencies) the type loading error is cached (the action, not the error itself) and the compilation aborts (unable to load provider...). The next compilation by the same thread succeeds if the failure in loading the provider is not critical for the type definition to be used to validate what's in the catalog. That's exactly what's happening with the specific example on this ticket, the Nova_network type. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Thomas Hallgren, there you go: [qtp1249723194-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/nova/lib/puppet/type/nova_network.rb inc1382019 [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:209:in `load' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:173:in `type' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:389:in `resource_type' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:377:in `resource_type' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:318:in `initialize' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/collectors/catalog_collector.rb:13:in `initialize' [qtp1249723194-67] [puppetserver] Puppet CERN - load_file:
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Ok, so to summarise, even if we managed to get generate_types to work and that prevented the type loader from loading all the providers when compiling a catalog, it'd be something that we could not deploy inmmediatly as we'd have to think how to integrate it with the system that generates our Puppet environments. After seeing how the master behaves when that feature is not used (the type loader loads always all providers for all types – quite arguable) I'm going to propose to my colleagues to patch the nova module locally to require the deps by path. -require 'puppet/provider/openstack' -require 'puppet/provider/openstack/auth' -require 'puppet/provider/openstack/credentials' +require File.join(File.dirname(__FILE__), '..','..','..', '..', 'openstacklib/lib/puppet/provider/openstack') +require File.join(File.dirname(__FILE__), '..','..','..', '..', 'openstacklib/lib/puppet/provider/openstack/auth') +require File.join(File.dirname(__FILE__), '..','..','..', '..', 'openstacklib/lib/puppet/provider/openstack/credentials') This seems to work and the catalog always compiles
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Hi, Thanks for your suggestion. 6156a6ba4640f617b5df42bea011b5ef548ad542 applied on top of 4.9.4, .resource_types generated for the environment of the host requesting the catalog, master restarted but I get the same result, the Ruby types are loaded: 2017-06-15 17:41:06,277 WARN [qtp1435036242-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/nova/lib/puppet/type/nova_network.rb inc1382019 2017-06-15 17:41:06,280 WARN [qtp1435036242-67] [puppetserver] Puppet CERN - autoloader: loadall puppet/provider/nova_network inc1382019 2017-06-15 17:41:06,288 WARN [qtp1435036242-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/nova/lib/puppet/provider/nova_network/nova.rb inc1382019 $ head /var/lib/puppet/environments/inc1382019/.resource_types/nova_network.pp # This file was automatically generated on 2017-06-15 17:38:52 +0200. # Use the 'puppet generate types' command to regenerate this file.
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures After following the stack when loadall is executed, it seems to come from here: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/metatype/manager.rb#L126 Puppet CERN - autoloader: loadall puppet/provider/nova_network inc1382019 ["/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:212:in `loadall'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:126:in `newtype'", "/var/lib/puppet/environments/inc1382019/modules/nova/lib/puppet/type/nova_network.rb: That makes sense on the agent perhaps, but not on the master side, does it? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Well, it happens too with core types: 2017-06-15 14:26:47,074 WARN [qtp1205135533-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/package.rb inc1382019 2017-06-15 14:26:47,083 WARN [qtp1205135533-67] [puppetserver] Puppet CERN - autoloader: loadall puppet/provider/package inc1382019 2017-06-15 14:26:47,135 WARN [qtp1205135533-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/chocolatey/lib/puppet/provider/package/chocolatey.rb inc1382019 ... Also when the master starts it loads a bunch of things, eg: 2017-06-15 14:21:51,422 WARN [clojure-agent-send-pool-0] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/file.rb production 2017-06-15 14:21:51,450 WARN [clojure-agent-send-pool-0] [puppetserver] Puppet CERN - autoloader: loadall puppet/provider/file production 2017-06-15 14:21:51,462 WARN [clojure-agent-send-pool-0] [puppetserver] Puppet CERN -
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Is it expected than when a type is loaded by the master Puppet::Util::Autoload.loadall is called on all the providers? For example: 2017-06-15 14:06:27,486 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/network/lib/puppet/type/network_config.rb inc1382019 2017-06-15 14:06:27,488 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: loadall puppet/provider/network_config inc1382019 2017-06-15 14:06:27,495 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/network/lib/puppet/provider/network_config/interfaces.rb inc1382019 2017-06-15 14:06:27,503 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded): /var/lib/puppet/environments/inc1382019/modules/network/lib/puppet/provider/network_config/interfaces.rb inc1382019 2017-06-15 14:06:27,505 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/network/lib/puppet/provider/network_config/redhat.rb inc1382019 2017-06-15 14:06:27,508 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded): /var/lib/puppet/environments/inc1382019/modules/network/lib/puppet/provider/network_config/redhat.rb inc1382019 2017-06-15 14:06:27,509 WARN [qtp529923418-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded):
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Then I must be doing something wrong: Notice: Generating '/var/lib/puppet/environments/inc1382019/.resource_types/nova_network.pp' using 'pcore' format. ... bash-4.2$ pwd /var/lib/puppet/environments/inc1382019 bash-4.2$ stat .resource_types/nova_network.pp File: ‘.resource_types/nova_network.pp’ Size: 2622 Blocks: 8 IO Block: 4096 regular file The environmentpath for the master is /var/lib/puppet/environments. The type seems to be loaded by the autoloader from the environment itself: 2017-06-15 13:44:09,685 WARN [qtp1992813264-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): /var/lib/puppet/environments/inc1382019/modules/nova/lib/puppet/type/nova_network.rb inc1382019
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures You can try to use the generate types solution that is used to overcome environment isolation problems. I'm afraid that this does not make any difference . After generating .resource_types for the environment of the node requesting the catalog the behaviour is the same: the first compilation fails (the master tries to load puppet/provider/nova_network/nova.rb, the failure is cached and the compilation succeeds after trying again.) Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Yes, I totally agree. The prime goal should be to understand why the master is trying to load puppet/provider/nova_network/nova when looking up puppet/type/nova_network I've added some debug statements to util/autoload.rb's load_file method to see what and when files are requested to be loaded. What I'm seeing is that during the first catalog request after puppetserver is restarted plenty of providers are loaded: 2017-06-15 11:29:17,497 WARN [qtp1376959403-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded): puppet/provider/service/launchd 2017-06-15 11:29:17,499 WARN [qtp1376959403-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): puppet/provider/service/openbsd 2017-06-15 11:29:17,503 WARN [qtp1376959403-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded): puppet/provider/service/openbsd 2017-06-15 11:29:17,504 WARN [qtp1376959403-67] [puppetserver] Puppet CERN - autoloader: load_file (trying to load): puppet/provider/service/openrc 2017-06-15 11:29:17,506 WARN [qtp1376959403-67] [puppetserver] Puppet CERN - autoloader: load_file (loaded): puppet/provider/service/openrc ... Until it attempts to load puppet/provider/nova_network/nova when it fails for the reasons that we seem to understand (see my previous comments)
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Henrik Lindberg just responding to the ping from earlier. It seems like in order for the file containing the Puppet::Provider::Nova class to be able to resolve the puppet/provider/openstack require without using a fully-qualified path that a plugin sync of the openstack module would have needed to happen. The pluginsync might explain why in this case that the problem only seemed to happen during the first agent run. Our masters never pluginsync the openstacklib module (the one that ships puppet/provider/openstack) therefore if that was the issue, shouldn't all compilations be failing? # puppetserver irb irb(main):001:0> require 'puppet' => true irb(main):002:0> require 'puppet/provider/firewall' => true irb(main):003:0> require 'puppet/provider/openstack' LoadError: no such file to load -- puppet/provider/openstack Perhaps the failure is cached? On the other hand this would be bad as that code would be taken from the environment of the master, not the environment of
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Hi, ok, I will close this issue as there does not seem to be anything that is wrong with puppet. I tend to disagree, we don't know that yet for sure until we figure out why the provider is loaded by the master. It's not clear to me if that's the module's or the master's fault, is it to you? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos commented on PUP-7657 Re: Providers requiring files in other modules might cause intermittent compilation failures Hi Henrik, It's nice to chat to you again. ...a provider should not be loaded during compilation - that could indicate a bad implementation in that type. Yes, indeed, that was the first thing that I thought that was not clicking in. I've managed to find an older version of the module with which the reproducer above does not trigger the issue under the same circumstances. The puzzling part is that the diff of the lib directory does not reveal any change made to the type in question, although the only provider for that type is renamed and modified. $ g diff good..bad --stat -- lib/ lib/facter/libvirt_uuid.rb | 11 +++ lib/puppet/provider/nova.rb| 140 +--- lib/puppet/provider/nova_aggregate/nova.rb | 173 - lib/puppet/provider/nova_aggregate/openstack.rb| 107 +++ lib/puppet/provider/nova_config/ini_setting.rb | 10 --
Jira (PUP-7657) Providers requiring files in other modules might cause intermittent compilation failures
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-7657 Providers requiring files in other modules might cause intermittent compilation failures Issue Type: Bug Assignee: Unassigned Components: Puppet Server Created: 2017/06/14 12:39 AM Priority: Normal Reporter: Nacho Barrientos Hello, Please take this report for the moment as a call for help debugging although it might end up becoming a bug report. The Puppet version over here is 4.9.4. We're experiencing intermittent compilation failures in nodes making using of the following modules: https://github.com/openstack/puppet-openstacklib/tree/stable/newton https://github.com/openstack/puppet-nova/tree/stable/newton The snippet to reproduce the issue is pretty simple: Package<| tag ==
Jira (PUP-7302) Puppet HTTP Node API: make fact_merge optional
Title: Message Title Nacho Barrientos commented on PUP-7302 Re: Puppet HTTP Node API: make fact_merge optional Hi Josh Cooper. Are you sure that that patch addresses what's described on this ticket? I have not read the patch itself, but the description of the MR does not seem to cover this issue. Will that patch prevent puppetserver from requesting facts from PDB when a request to /node is done? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7302) Puppet HTTP Node API: make fact_merge optional
Title: Message Title Nacho Barrientos commented on PUP-7302 Re: Puppet HTTP Node API: make fact_merge optional Hello? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7302) Puppet HTTP Node API: make fact_merge optional
Title: Message Title Nacho Barrientos commented on PUP-7302 Re: Puppet HTTP Node API: make fact_merge optional Something like this: https://github.com/puppetlabs/puppet/pull/5689 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7302) Puppet HTTP Node API: make fact_merge optional
Title: Message Title
Nacho Barrientos updated an issue
Puppet / PUP-7302
Puppet HTTP Node API: make fact_merge optional
Change By:
Nacho Barrientos
Hi,In a Puppet installation with PuppetDB available, each call of the agents to the [/node entrypoint|https://docs.puppet.com/puppet/latest/http_api/http_node.html] of the Puppet HTTP API generates a call to PuppetDB's _/vX/nodes/foo.example.com/facts_ if the list of facts is not cached on the master. This is very likely in our case because we have plenty of masters serving our agents and also because the TTL is 30 ~20 minutes, which is lower than our default Puppet run interval.This generates queries to PuppetDB which, unless I was missing something, are not necessary in our deployment and we'd rather to use those CPU cycles for something else. I fail to see how getting the list of last known facts as parameters are useful to the agent when calling _/node_ (first request during a Puppet run). OTOH, we do need variables coming from the ENC like the environment, of course.We managed to avoid those queries in a development master by doing this hack:{noformat}--- routes.yaml.orig2017-03-04 11:19:52.164820233 +0100+++ routes.yaml 2017-03-04 11:12:18.021435083 +0100@@ -6,6 +6,6 @@ ---master: facts: - terminus: puppetdb+ terminus: yaml cache: yaml{noformat}The Puppet agent run went fine, however this configuration prevents later on the masters from sending commands to PuppetDB to replace facts which is something we'd like to keep.Would you take into consideration a patch to, behind a setting, make [this behaviour|https://github.com/puppetlabs/puppet/blob/ebd96213cab43bb2a8071b7ac0206c3ed0be8e58/lib/puppet/node.rb#L116] optional?Thank you!
Add Comment
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Jira (PUP-7302) Puppet HTTP Node API: make fact_merge optional
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-7302 Puppet HTTP Node API: make fact_merge optional Issue Type: Bug Assignee: Unassigned Created: 2017/03/04 3:31 AM Priority: Normal Reporter: Nacho Barrientos Hi, In a Puppet installation with PuppetDB available, each call of the agents to the /node entrypoint of the Puppet HTTP API generates a call to PuppetDB's /vX/nodes/foo.example.com/facts if the list of facts is not cached on the master. This is very likely in our case because we have plenty of masters serving our agents and also because the TTL is 30 minutes, which is lower than our default Puppet run interval. This generates queries to PuppetDB which, unless I was missing something, are not necessary in our deployment and we'd rather to use those CPU cycles for something else. I fail to see how getting the list of last known facts as parameters are useful to the agent when calling /node (first request during a Puppet run). OTOH, we do need variables coming from the ENC like the environment, of course. We managed to avoid those queries in a development master by doing this hack: --- routes.yaml.orig2017-03-04 11:19:52.164820233 +0100 +++ routes.yaml
Jira (PUP-5482) Should a not found type be marked as such?
Title: Message Title Nacho Barrientos commented on PUP-5482 Re: Should a not found type be marked as such? +1 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5482) Should a not found type be marked as such?
Title: Message Title Nacho Barrientos commented on PUP-5482 Re: Should a not found type be marked as such? https://github.com/puppetlabs/puppet/pull/4427 Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5482) Should a not found type be marked as such?
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-5482 Should a not found type be marked as such? Issue Type: Improvement Assignee: Unassigned Created: 2015/11/09 7:33 AM Priority: Normal Reporter: Nacho Barrientos If a Ruby type is not available in the file system, every time it is found by the parser, the Autoloader will scan all over again the directories in search_directories for it. This generates tons of stat()s that could potentially be saved. I have a patch to propose already, please watch this space. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Jira (PUP-5005) Puppet agent should support requiring a message when disabling
Title: Message Title Nacho Barrientos commented on PUP-5005 Re: Puppet agent should support requiring a message when disabling Sounds like a good idea Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-3727) Notify: provide a way to log only the message itself at 'notice' level
Title: Message Title Nacho Barrientos created an issue Puppet / PUP-3727 Notify: provide a way to log only the message itself at 'notice' level Issue Type: Improvement Assignee: Unassigned Created: 2014/12/03 12:40 AM Priority: Minor Reporter: Nacho Barrientos Hi, I've already asked in the users' mailing list but judging from the number of responses I got it seems that what I want to do is not possible so please consider this a very low priority feature request. Of course, feel free to shut this as won't fix if there's indeed a way to do it When using the Notify resource type, is there any way to hide the log message that is produced by the definition itself and only keep the actual message string on the output generated by Puppet agent? In this example: Notice: hello Notice: /Stage[main]/Foo::Bar/Notify[baz]/message: defined 'message' as 'hello' The idea would be to tell Puppet somehow to only print the following: Notice: hello We've tried passing audit = [] as an attribute to the resource declaration but it didn't produce any effect log-wise. Also we changed the loglevel to 'debug' at resource level and it hid both. We're (still) using Puppet 3.4.3 so maybe there's something available to cover this use case in newer versions. Thanks in advance
Jira (PUP-1592) Puppet excessively stats the filesystem when looking for defined types
Title: Message Title Nacho Barrientos commented on an issue Re: Puppet excessively stats the filesystem when looking for defined types Yeah, manifests are shared across all the masters via a NFS share. Add Comment Puppet / PUP-1592 Puppet excessively stats the filesystem when looking for defined types [~masterzen] and I realized that when puppet tries to resolve a defined type, it always tries to first find the defined type within all the plugin directories as it tries to first load it as a resource type and then as a defined type. We should be able to avoid a lot of stats on the filesystem by simply changing the lookup order. [~masterzen] has so... This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
