Does "zenoss" mean Xen virtualization? PVM or HVM? How many forests are on each host?
You could simply try upgrading from 6.0-3.2 to the latest release, 6.0-4, and see if that helps. But if it were me I would want to know which query or queries caused the problem. Even though you aren't sending queries directly to that busy host, it's resolving index lookups as requested by the eval hosts. So it's still important to look at long-running queries, as these are the ones likely driving the load on your busy host. You also want to have a reproducible test case, and the best way to build that is to isolate a query that recreates the high load. At the same time, dig into how utilization is measured and exactly what the numbers are. It's not enough to say that a host is "maxed out": you need to understand which subsystem is the bottleneck. It's quite difficult to drive a 16-core or 32-core host to 0% idle, especially if the workload is mixed between network, disk, and CPU activity. You really want to know how much of each is involved, to better understand what "maxed out" really means. For example 'iostat -mxz 15' is a good way to monitor current activity, or if sysstat is collecting data then sar can display it. Just to illustrate the point, here are some low-utilization numbers from a system I happen to have handy. 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:05:01 AM all 6.24 0.30 0.50 0.24 0.09 92.64 12:15:01 AM all 2.79 0.00 0.12 0.09 0.05 96.95 12:25:01 AM all 3.39 0.00 0.16 0.10 0.07 96.27 12:35:01 AM all 2.80 0.00 0.13 0.06 0.06 96.96 If this host were "maxed out", that could appear as high %user, or %nice, or %system, or %iowait, or %steal - or any mix of those. That, in turn, would tell you something about why the host is busy. If it turns out to be high %system or %iowait, take a look at the :8001/host-status.xqy page for the host in question. At the bottom you'll see a table of rates and loads, which will tell you something about where the host is spending its time. -- Mike On 29 Sep 2013, at 11:57 , indar verma <[email protected]> wrote: > One more thing to add, > > We are sending requests to Ml1 to Ml3 in round robin fashion from the > application end. so Ml4 & Ml5 are not accepting any direct request from the > front end app. > > we are ingesting data through these two ml4 &Ml5 newly added nodes. > > Thanks, > JJ > > From: indar verma <[email protected]> > To: "[email protected]" <[email protected]> > Cc: "[email protected]" > <[email protected]> > Sent: Monday, 30 September 2013 12:18 AM > Subject: Re: One node CPU utilization maxed out but others not in a 5 node > cluster once load increases > > Thanks Michael!!! > > Which version of MarkLogic is this? - MarkLogic 6.0-3.2 > > > > How many CPU cores are on each host? ML1 to ML4 - 2CPUs & 16 cores and ML5 - > 2 CPUs & 32 cores > > Local Disk failover enabled between ML1 to Ml3 and Ml4 to Ml5 (as ml4 & Ml5 > have been added recently) > > > How much RAM is on each host? - 64GB > > > How are you measuring CPU utilization? - Linux commands once response time > increases and infra team tell us. it seems thye are using zenoss > > > My concern is why ML4 CPU is maxed out even it is not having much data nor > xQuery is being executed on this node. > > Could it be the reason as Ml5 is highly configured than Ml4 and fail over is > enabled between both? > > Offcourse xQuery execution can be one of the case, these will be taking time > however Heavy load & data is on Ml1 to Ml3 even their CPU usages is around > 30%. > > Thanks, > > JJ > > > From: indar verma <[email protected]> > To: "[email protected]" <[email protected]> > Sent: Sunday, 29 September 2013 11:18 PM > Subject: One node CPU utilization maxed out but others not in a 5 node > cluster once load increases > > Hi All, > > We have a 5 nodes clustered environment and data is in TBs. Ml1 to Ml3 are > evaluator nodes & data nodes and Ml4 & Ml5 are data nodes only for now. > > Once loads increase the Ml4 CPU utilization going to become maximum and other > 4 nodes are normal around 30% utilization. > > Due to Ml4 CPU maxed out, page is not rendering, some queries are giving time > out. > > Also one thing, more data processing is not being happen in Ml4 than other > nodes. > > Could you please somebody help me to resolve this? what can be the root cause > for this problem? what should we improve? > > Thanking in advance!!! > > JJ > > > > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
