What is the latency between your nodes?
Have you restarted the scheduler after changing that setting?
Are you using CherryPy ?
I don't think it's benefitial in having too many schedulers unless you have
a pretty good retention between them set up. I'd recommend two plus one
spare for a setup your size.
On 12 May 2015 at 18:27, David Good <dg...@willingminds.com> wrote:
> Does it hurt to have more than I need? All of the Shinken servers have
> plenty of resources to spare. I'd rather they have more threads than they
> need than too few. Unless that creates other problems.
>
> I've removed the extra schedulers and pollers I had added thinking it
> would help (now a total of 4 each plus a spare) and raised the thread pool
> size on all daemons to 50 and I'm still having issues with timeouts.
>
>
>
> On 5/12/15 2:34 AM, Felipe openglx wrote:
>
> Those timeout issues happened to me on a similar sized structure.
> One of my realms was never getting stable and I discovered it wasn't
> timeout but actually number of threads on Cherrypy due number of other
> Shinken components.
>
> Try allowing more threads on your scheduler (as it is the most central
> component):
>
> daemon_thread_pool_size=12
>
>
> Twelve threads was good enough for me but your number will be higher. So
> you have:
>
> * 2 "master": 1 arbiter, 1 broker, 1 reactionner, 1 receiver, 2
> schedulers, 2 pollers
> * 3 nodes: 2 schedulers, 2 pollers
>
>
> That means that you have 2*8 + 3*4 = 28 daemons. Maybe bring your thread
> pool size to 30 ?
>
>
> I'd suspect root cause is your setup having way too many schedulers for
> no apparent reason. If you are having them distributed please take some
> read on setting up some realms.
>
>
> Regards
>
>
>
> On 12 May 2015 at 05:29, David Good <dg...@willingminds.com> wrote:
>
>>
>> We're still running 2.2 currently (we're on a tight schedule to get
>> Shinken into production). We just revamped our host and service
>> configuration, adding about 1000 hosts (currently 3351 hosts, 1839
>> hostgroups, 30404 services). Since loading that configuration we've had
>> trouble with connections from the arbiter timing out so it's constantly
>> redispatching the configuration only to have the daemon that timed out
>> recover. I've increased ping timeouts to 6 seconds and data timeouts to
>> 120 seconds but it still times out. I though maybe adding more
>> schedulers and pollers would help as it would cut the config into
>> smaller pieces that could be processed more quickly, but it's still
>> constantly dispatching and re-dispatching configurations.
>>
>> We're running on 5 servers, each with 40 CPUs and 64 GB of RAM. One of
>> the servers is the master and runs all daemons plus an extra poller and
>> scheduler. Three other servers are running two schedulers and two
>> pollers each. The last is a spare setup the same as the master. None
>> of the servers are showing significant CPU, I/O, Memory or network usage.
>>
>> Any ideas? Would upgrading to 2.4 help?
>>
>>
>> ------------------------------------------------------------------------------
>> One dashboard for servers and applications across Physical-Virtual-Cloud
>> Widest out-of-the-box monitoring support with 50+ applications
>> Performance metrics, stats and reports that give you Actionable Insights
>> Deep dive visibility with transaction tracing using APM Insight.
>> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
>> _______________________________________________
>> Shinken-devel mailing list
>> Shinken-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/shinken-devel
>>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM
> Insight.http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
>
>
>
> _______________________________________________
> Shinken-devel mailing
> listShinken-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/shinken-devel
>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Shinken-devel mailing list
> Shinken-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/shinken-devel
>
>
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Shinken-devel mailing list
Shinken-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shinken-devel
