foreman-tracer rails calls: http://pastebin.com/80ziGqXr
On Tuesday, November 1, 2016 at 12:06:45 PM UTC-4, Justin Foreman wrote: > > Okay, now we're getting somewhere. This is an environment with five oVirt > nodes in two clusters. > Cluster1: 2 nodes > Cluster2: 3 nodes > > Each have a handful of VMs, some manually installed, and some provisioned > by Foreman spanning both clusters. > > The oVirt nodes each appear to have maybe 10-50 nics (mostly VLAN > interfaces and bridges for VMs). When I run the puppet agent on any of the > three nodes in Cluster2, the Nic::Managed count shoots through the roof. > Even if I kill the puppet agent, the count continues to rise. > > Here's a list of the NICs on one of the offending hosts: > > http://pastebin.com/DxZup68B > > Honestly, the NIC information for these hosts aren't very useful. As a > temporary workaround, is there a way to exclude gathering NIC information > during this process? > > Thanks! > Justin > > On Tuesday, November 1, 2016 at 8:56:50 AM UTC-4, Lukas Zapletal wrote: >> >> Ok one minute is fine, the counters will reset in 5 minutes anyway. >> >> Ok, the problem is in setup_clone / setup_object_clone method which >> creates a deep copy of each record for comparison. But I wonder how is >> possible you have 20k calls of this clone after just 1 minute. >> >> Tell me more about your infrastructure. How many hosts? What is the >> everage count of NICs associated with a host? Don't you have some kind >> of broken host with 20k NICs associated? Remember, Puppet fact upload >> will cause creation of NIC record for each NIC reported, so you could >> have some broken host reporting "ethXYZ_address" each puppet run >> causing the NIC table to grow. >> >> Also, can you tell the 100% CPU utilization is by Ruby process itself, >> or is that caused by swapping? If this is really the Ruby process >> doing some work, please also run >> >> foreman-tracer rails calls >> >> For a minute or two to see where is it looping in. Then pastebin again, >> thanks. >> >> LZ >> >> >> On Tue, Nov 1, 2016 at 12:47 PM, Justin Foreman <[email protected]> >> wrote: >> > I wasn't sure how long to run them, so I ran each for 60 seconds. >> > >> > foreman-tracer rails objects-total >> > http://pastebin.com/QdZePcWQ >> > >> > foreman-tracer rails objects >> > http://pastebin.com/jVbDRm3c >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Foreman users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > To post to this group, send email to [email protected]. >> > Visit this group at https://groups.google.com/group/foreman-users. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Later, >> Lukas @lzap Zapletal >> > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
