Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Claire Cadman updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Claire Cadman Labels: doc_reviewed Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.53360.1602508680088%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title
Josh Cooper commented on PUP-10628
Re: Performance regression with large hashes using lookup
This also fixes the issue where the hash is validated multiple times for each level of the hash. Previously PTypeSetType#instance? was three times with the following object o:
{'hash2' => {'hash3' => { 'key1' => ..., ... } } } {'hash3' => { 'key1' => ..., ... } } {'key1' => ... }
With the new code, the instance? is still called three times, but it returns immediately due to the object not being an instance of PTypeSetType.
Add Comment
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper commented on PUP-10628 Re: Performance regression with large hashes using lookup Passed CI in 3b720e6e2e Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.30186.1599154200101%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Release Notes: Bug Fix Release Notes Summary: Fixes a performance regression in 5.5.x and up which affected hiera lookups in large hashes. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.28458.1598984520026%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper commented on PUP-10628 Re: Performance regression with large hashes using lookup Follow up merged to 5.5.x https://github.com/puppetlabs/puppet/commit/6f168e901fae4a3b8593a342f4df54044fed22dd. We now verify the object is an instance of (or subclass) of PTypeSetType. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.28450.1598984340023%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Sprint: Platform Core KANBAN Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.25679.1598569800038%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper assigned an issue to Josh Cooper Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Assignee: Josh Cooper Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.25677.1598569740027%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper commented on PUP-10628 Re: Performance regression with large hashes using lookup I modified TypeSetType#instance? so inference is skipped if the object isn't a hash or if the pcore_version key is missing. The was merged to 5.5.x in https://github.com/puppetlabs/puppet/commit/d00f2a6a23f6a98daf2b954aea2e1bf01c020c17. I'm going to file a separate ticket to remove TypeSetType from RichData to be fixed in Puppet 7. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.25634.1598568780053%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Team: Coremunity Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.25635.1598568780096%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Fix Version/s: PUP 6.19.0 Fix Version/s: PUP 5.5.22 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.25636.1598568780139%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Henrik Lindberg commented on PUP-10628 Re: Performance regression with large hashes using lookup Josh Cooper I think it is may be as easy as checking if it has the key "pcore_uri" - which should be present and signifies that "this is something that has to do with pcore" - but it could be that actual usage is too relaxed with the formalities of declaring this (or the pcore_version for that matter). Apart from Puppet core and possibly Bolt, I don't think there is any use of `TypeSet` in the wild - if so, the worst case would be that manually authored typeset files would need to be updated with the pcore_uri. Would need to ask Thomas Hallgren to get all the details. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.21909.1598120880088%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper commented on PUP-10628 Re: Performance regression with large hashes using lookup Tests pass for me locally when Type and TypeSet are removed, so I submitted a PR https://github.com/puppetlabs/puppet/pull/8294. It looks like TypeSet#instance? could be optimized to reject the value if the required pcore_version key is missing, but I'm not sure which keys are actually required based on the way __pcore_init_hash optionally parses the hash: https://github.com/puppetlabs/puppet/blob/main/lib/puppet/pops/types/p_type_set_type.rb#L174-L180. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.21865.1598052180030%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Henrik Lindberg commented on PUP-10628 Re: Performance regression with large hashes using lookup An optimization that was done in other places were that instead of inferring the type of the value and then matching the inferred against the desired you ask if it is an instance of the desired type - not sure if there is something like that going on here, or if also asking if it is an instance requires a deep recursion. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.20605.1597951620156%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Henrik Lindberg commented on PUP-10628 Re: Performance regression with large hashes using lookup RichData as a format allows for types to be embedded in the serialization of data - that way, a serialization can be self describing. In serialization it is optional to use this, or to simply include types as reference via their name (i.e. "hoping" that the other side has the same definition of the type). Early on in the implementation of rich-data support in the catalog we did include the serialized types - but this proved to be too bulky. There is currently no other data type (alias) defined for RichData without the Type and Type set types, but one could be created. Don't see it is of much practical use to support Type/TypeSet in hiera data, you would get the same if types were returned as strings and then turned into types dynamically - rather than a hiera backend returning an instance of Type/TypeSet. Not sure if dropping them from RichData would cause problems - probably not that difficult to test. Speeding up the inference is still of value - there may be some obvious things that can be done to make it faster to discriminate if a hash could possibly represent a type or typeset. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper commented on PUP-10628 Re: Performance regression with large hashes using lookup Excluding TypeSet from the list of possible data provider values types is about 300x faster: diff --git a/lib/puppet/pops/loader/static_loader.rb b/lib/puppet/pops/loader/static_loader.rb index 3e6b78ce4c..8faca4e84f 100644 --- a/lib/puppet/pops/loader/static_loader.rb +++ b/lib/puppet/pops/loader/static_loader.rb @@ -29,6 +29,7 @@ class StaticLoader < Loader 'Data' => 'Variant[ScalarData,Undef,Hash[String,Data],Array[Data]]', 'RichDataKey' => 'Variant[String,Numeric]', 'RichData' => 'Variant[Scalar,SemVerRange,Binary,Sensitive,Type,TypeSet,URI,Object,Undef,Default,Hash[RichDataKey,RichData],Array[RichData]]', +'LookupDataValue' => 'Variant[Scalar,SemVerRange,Binary,Sensitive,Type,URI,Object,Undef,Default,Hash[RichDataKey,LookupDataValue],Array[LookupDataValue]]', # Backward compatible aliases. 'Puppet::LookupKey' => 'RichDataKey',
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title
Josh Cooper commented on PUP-10628
Re: Performance regression with large hashes using lookup
This can be reproduced using:
require 'puppet'
require 'puppet/pops'
Puppet.initialize_settings
Puppet::Util::Log.newdestination(:console)
values = 0.upto(2000).inject({}) do |memo, i|
memo["key#{i}"] = "value#{i}"
memo
end
hash = { 'hash2' => { 'hash3' => values } }
Puppet::Util.benchmark(:notice, "Inferred hash is an instance of #{Puppet::Pops::Lookup::DataProvider.value_type.name} in %{seconds}") do
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title
Oleksandr Shtepa commented on PUP-10628
Re: Performance regression with large hashes using lookup
Looks like same problem exists for facter code. On some node I have 270 IP addresses. --profile shows:
[puppetserver] Puppet functions -> net::ipaddresses4_list: 6.164248 s (5 calls)
On a node with 3 IPs:
[puppetserver] Puppet functions -> net::ipaddresses4_list: 0.075847 s (5 calls)
net::ipaddresses4_list only works with fact $::networking:
function net::ipaddresses4_list() >>Array[String] {
$::networking['interfaces'].filter|$v|{ 'bindings' in $v[1] }.reduce([])|$memo, $v|{ $memo + $v[1]['bindings'].map()|$i|{ $i['address']} }
}
Add Comment
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title
Josh Cooper commented on PUP-10628
Re: Performance regression with large hashes using lookup
Thanks Oleksandr Shtepa! I was able to reproduce using your instructions:
# puppet --version
6.17.0
# cd /etc/puppetlabs/puppet
# cat hiera.yaml
---
version: 5
defaults:
datadir: .
data_hash: yaml_data
hierarchy:
- name: test
path: test.yaml
# echo -e 'hash1:\n hash2:\nhash3:' > test.yaml
# for i in {1..2000}; do echo " key$i: value$i" >> test.yaml ; done
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Josh Cooper updated an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Change By: Josh Cooper Attachment: rbspy-2020-08-20-gAPdCc51RI.flamegraph.svg Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.369684.1597848926000.19923.1597902420037%40Atlassian.JIRA.
Jira (PUP-10628) Performance regression with large hashes using lookup
Title: Message Title Oleksandr Shtepa created an issue Puppet / PUP-10628 Performance regression with large hashes using lookup Issue Type: Bug Affects Versions: PUP 6.17.0 Assignee: Unassigned Components: Hiera & Lookup Created: 2020/08/19 7:55 AM Environment: Test configuration (2000 items hash): $ cat > hiera.yaml < --- version: 5 defaults: datadir: . data_hash: yaml_data hierarchy: - name: test
