I set it to ignore the offending interface names, but the issue of increasing Nic::Managed count was still happening. I found that the offending hosts had hundreds of duplicate interfaces. I wrote a little hammer script to delete all interfaces from the hosts. Now, all appears to be well.
Thanks for all the help! Justin On Tuesday, November 1, 2016 at 12:12:04 PM UTC-4, Josh wrote: > > See the 'ignore interfaces with matching identifier' option under Settings > -> Provisioning. > > I had the same problem with Docker network interfaces. > > Josh > > On Tue, Nov 1, 2016 at 12:06 PM, Justin Foreman <[email protected] > <javascript:>> wrote: > >> Okay, now we're getting somewhere. This is an environment with five oVirt >> nodes in two clusters. >> Cluster1: 2 nodes >> Cluster2: 3 nodes >> >> Each have a handful of VMs, some manually installed, and some provisioned >> by Foreman spanning both clusters. >> >> The oVirt nodes each appear to have maybe 10-50 nics (mostly VLAN >> interfaces and bridges for VMs). When I run the puppet agent on any of the >> three nodes in Cluster2, the Nic::Managed count shoots through the roof. >> Even if I kill the puppet agent, the count continues to rise. >> >> Here's a list of the NICs on one of the offending hosts: >> >> http://pastebin.com/DxZup68B >> >> Honestly, the NIC information for these hosts aren't very useful. As a >> temporary workaround, is there a way to exclude gathering NIC information >> during this process? >> >> Thanks! >> Justin >> >> On Tuesday, November 1, 2016 at 8:56:50 AM UTC-4, Lukas Zapletal wrote: >>> >>> Ok one minute is fine, the counters will reset in 5 minutes anyway. >>> >>> Ok, the problem is in setup_clone / setup_object_clone method which >>> creates a deep copy of each record for comparison. But I wonder how is >>> possible you have 20k calls of this clone after just 1 minute. >>> >>> Tell me more about your infrastructure. How many hosts? What is the >>> everage count of NICs associated with a host? Don't you have some kind >>> of broken host with 20k NICs associated? Remember, Puppet fact upload >>> will cause creation of NIC record for each NIC reported, so you could >>> have some broken host reporting "ethXYZ_address" each puppet run >>> causing the NIC table to grow. >>> >>> Also, can you tell the 100% CPU utilization is by Ruby process itself, >>> or is that caused by swapping? If this is really the Ruby process >>> doing some work, please also run >>> >>> foreman-tracer rails calls >>> >>> For a minute or two to see where is it looping in. Then pastebin again, >>> thanks. >>> >>> LZ >>> >>> >>> On Tue, Nov 1, 2016 at 12:47 PM, Justin Foreman <[email protected]> >>> wrote: >>> > I wasn't sure how long to run them, so I ran each for 60 seconds. >>> > >>> > foreman-tracer rails objects-total >>> > http://pastebin.com/QdZePcWQ >>> > >>> > foreman-tracer rails objects >>> > http://pastebin.com/jVbDRm3c >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "Foreman users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an >>> > email to [email protected]. >>> > To post to this group, send email to [email protected]. >>> > Visit this group at https://groups.google.com/group/foreman-users. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> Later, >>> Lukas @lzap Zapletal >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
