Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Claire Cadman updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Claire Cadman Labels: doc-reviewed Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.151781.1633968060084%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Ciprian Badescu updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Ciprian Badescu Fix Version/s: PUP 6.26.0 Fix Version/s: PUP 6.25.1 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.151319.1633934280045%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions Passed CI in 8112a59212a Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.151204.1633747680095%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions This was cherry-picked to the release branch for 7.12.0: d3a2ea9f5d825037ec3e5f72c26f6199ad121bd5 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.151037.1633729200025%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Fix Version/s: PUP 7.13.0 Fix Version/s: PUP 7.12.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.151036.1633729140348%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Summary: Type checking arguments passed to a block (reduce puppet language functions are 11 , merge, etc) is very slow 350x slower than ruby functions Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.150120.1633629180267%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Release Notes: Enhancement Release Notes Summary: Greatly speeds up the amount of time taken to type check arguments passed to blocks of iterative functions, such as reduce and merge. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.149476.1633558560026%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions Merged to 6.x in https://github.com/puppetlabs/puppet/commit/5bbb5cf1795418a36b368bca274035bbcdcdaada Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.149464.1633558200030%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Epic Link: PUP- 5773 11073 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.147309.1633450680260%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Epic Link: PUP-5773 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.147308.1633450680214%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Fix Version/s: PUP 7.13.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.145389.1633115280143%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Fix Version/s: PUP 6.26.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.145390.1633115280189%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Sprint: Coremunity Kanban Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.142809.1632962880089%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Team: Froyo Coremunity Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.142808.1632962880044%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Josh Cooper assigned an issue to Josh Cooper Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Josh Cooper Assignee: Josh Cooper Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.142807.1632962820048%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Josh Cooper commented on PUP-9561
Re: puppet language functions are 11,350x slower than ruby functions
Actually the type inference is much worse than I expected. Here are 3 manifests, using reduce, merge and merge2 which is the same as merge except the memo isn't passed to the block.
❯ cat reduce.pp
$test = generate('/bin/sh', '-c', 'seq 1 4000').split("\n")
$result = $test.reduce({}) |$memo,$value| { $memo + { "key $value" => "foo" } }
❯ cat merge.pp
$test = generate('/bin/sh', '-c', 'seq 1 4000').split("\n")
$result = merge($test) |$memo,$value| { { "key $value" => "foo" } }
❯ cat merge2.pp
$test = generate('/bin/sh', '-c', 'seq 1 4000').split("\n")
$result = merge($test) |$value| { { "key $value" => "foo" } }
Using puppet#3828aabe8d32368faa5cdc1a428189d0cb117e52, I get:
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Josh Cooper commented on PUP-9561
Re: puppet language functions are 11,350x slower than ruby functions
Bartosz Blizniak When using the merge function, the block just needs to return the value for that iteration, so you'll want to do:
$test = generate('/bin/seq', '1', '4000').split("\n")
$result = $test.merge |$memo,$value| { { "key $value" => "foo" } }
That said, it is still slow due to the amount of time puppet spends inferring the data type of the unused $memo parameter. I think there's a simple fix to the merge function for the typical case where the $memo is not needed/wanted:
$result = $test.merge |$value| { { "key $value" => "foo" } }
Doing that drops the time to merge 4000 elements from 13.2 sec to 3.8 sec:
❯ time bx puppet apply merge.pp
Notice: Compiled catalog for localhost in environment production in 9.48 seconds
Notice: Applied catalog in 0.01 seconds
bundle exec puppet apply merge.pp 12.90s user 0.24s system 99% cpu 13.244 total
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Bartosz Blizniak commented on PUP-9561
Re: puppet language functions are 11,350x slower than ruby functions
Hi, we are currently seeing some performance issues with the "reduce" function performance and "merge". A customer provided sample code:
$test = generate('/bin/seq', '1', '2000').split("\n") $result = $test.reduce({}) |$memo,$value| { $memo + { "key $value" => "foo" } }
This took around 3 seconds. Increasing the generate line to 4000 will increase the runtime to 12 seconds which is more than double. Is this something that we can improve the performance on or is this expected?
Add Comment
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.140854.1632828180028%40Atlassian.JIRA.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Henrik Lindberg commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions Sean Millichamp Since the feature for merge is now added to stdlib, maybe you can use that. And on that note, I am closing this ticket - it shows enough alternatives how to avoid exponential running time for hash construction. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.300276.1552587373000.28823.1557409260412%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Henrik Lindberg commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions And PUP-9568 was changed to become MODULES-8760 (for which there is now a PR to stdlib). Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Henrik Lindberg commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions I added PUP-9568 as an idea for how we can add a hash builder function (by vamping the existing `merge` function) in stdlib. Maybe do the work there so it can be used with older puppet version. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Jorie Tappa updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Jorie Tappa Team: Server Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Jorie Tappa updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Jorie Tappa Sub-team: Language Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Henrik Lindberg commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions Sean Millichamp You are correct - copying each time in the loop, and each time the hash is a little bigger is what kills performance. It is a common use case, and for small sized hashes it is not too bad. For bigger hashes, one of the techniques I showed work much faster. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Sean Millichamp commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions Thanks for the suggestions. Given that I can't use group_by yet (on 5.5) I'll probably stick with the Ruby I'm using, but Henrik's suggestions were clever and interesting to read. Would you mind clarifying what part of the original is responsible for the exponential running time? Is it the last line in the reduce that merges the results? Is that because each time through the loop requires allocating a new hash and copying all of the elements in the previous hash, before doing the merge? I believe that is a fairly common pattern I use in reduce, often to manipulate a data structure I can't otherwise modify (such as in an each construct, for example) because of immutable variables. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Henrik Lindberg commented on PUP-9561
Re: puppet language functions are 11,350x slower than ruby functions
If being on a version where group_by does not exist the function can be written like this:
$keys = $system_info.map |$x| { $x['system_id'] }.sort
$duplicate_ids = $keys.map |$i, $v| { if $v == $keys[$i+1] { $v } }.unique.filter |$x| { $x != undef}
This algorithm sorts they key, and for any duplicate key there will be at least one entry in the mapped result with this key (the rest are undef and filtered out). This variant is slower than the group_by version by 2x but it is at least linear.
Add Comment
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it,
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Henrik Lindberg commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions The original has exponential running time for the length of the input data, and the rewritten function is linear. For smaller number of entries (I tested with 200) the difference between the two algorithms is that Ben's version is 7x faster. For 500 entries it is 48x faster, and so on. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Ben Ford commented on PUP-9561
Re: puppet language functions are 11,350x slower than ruby functions
After Puppet 6.3, this could be rewritten as something like (eyeball compiled only)
$system_info.group_by |$rec| { $rec['system_id'] }
.filter |$k, $v| { $v.size > 1}
.keys
Add Comment
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this gro
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Ben Ford commented on PUP-9561 Re: puppet language functions are 11,350x slower than ruby functions I've had similar conversations with two other people in recent weeks. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title Ben Ford updated an issue Puppet / PUP-9561 puppet language functions are 11,350x slower than ruby functions Change By: Ben Ford Reporter: Ben Ford Sean Millichamp Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9561) puppet language functions are 11,350x slower than ruby functions
Title: Message Title
Ben Ford created an issue
Puppet / PUP-9561
puppet language functions are 11,350x slower than ruby functions
Issue Type:
Task
Assignee:
Unassigned
Components:
Language
Created:
2019/03/14 11:16 AM
Priority:
Normal
Reporter:
Ben Ford
From Slack: https://puppetcommunity.slack.com/archives/CFD8Z9A4T/p1552586549522900 I wrote a Puppet language function to, given an array of hashes, return a hash that counted duplicate instances of one of the keys in the original hash and then returned all of the values where the count was > 1. this was the code:
# Find count of records with duplicates:
$id_count = $system_info.reduce({}) |$results, $rec| {
$id = $rec['system_id']
$current = $results[$id].lest || { 0 }
$results + {$id => $current + 1}
