Re: [ClusterLabs] Antw: Re: Resources start serial, not parralel
There is in source code:
https://github.com/yuusuke/pacemaker/blob/efe2d6ebc55be39b8be43de38e7662f039b61dec/crmd/throttle.c
switch(r->mode) {
case throttle_extreme:
case throttle_high:
jobs = 1; /* At least one job must always be allowed */
break;
and
throttle_handle_load(float load, const char *desc)
{
if(load > THROTTLE_FACTOR_HIGH * throttle_load_target) {
crm_notice("High %s detected: %f", desc, load);
return throttle_high;
There are my log files:
Dec 16 04:51:09 server1 crmd[29851]: notice: throttle_handle_load: High
CPU load detected: 8.01
Dec 16 04:51:39 server1 crmd[29851]: notice: throttle_handle_load: High
CPU load detected: 7.16
Dec 16 04:52:09 server1 crmd[29851]: notice: throttle_handle_load: High
CPU load detected: 6.98
Dec 16 04:52:39 server1 crmd[29851]: notice: throttle_handle_load: High
CPU load detected: 6.47
So, main point of my issue is jobs = 1
Please, does it possibly to increase number of jobs through throttle high?
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: Resources start serial, not parralel
Hi guys, have you any ideas for root cause of trouble? I will appreciate any clues for investigation. 2015-12-14 17:40 GMT+09:00 Oleg Ilyin : > Hi Ulrich, > > thank you for your answer, > > what about which limits do you talk? > > The pacemaker software runs by root user, so limits should be increased > for root. > There is output from one of my server: > > -bash-4.1# id > uid=0(root) gid=0(root) groups=0(root) > > -bash-4.1# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 124801 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size(512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 124801 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > -bash-4.1# ps -ef |grep pacemaker > root 17947 1 0 Dec11 ?00:00:09 pacemakerd > 189 17953 17947 0 Dec11 ?00:00:11 /usr/libexec/pacemaker/cib > root 17954 17947 0 Dec11 ?00:00:14 > /usr/libexec/pacemaker/stonithd > root 17955 17947 0 Dec11 ?00:00:11 /usr/libexec/pacemaker/lrmd > 189 17956 17947 0 Dec11 ?00:00:09 > /usr/libexec/pacemaker/attrd > 189 17957 17947 0 Dec11 ?00:00:09 > /usr/libexec/pacemaker/pengine > root 17958 17947 0 Dec11 ?00:00:16 /usr/libexec/pacemaker/crmd > > > With settings in pacemaker or system can be changed for start heavy > application ( java ) at the same time? > > > > > 2015-12-14 16:29 GMT+09:00 Ulrich Windl >: > >> Hi! >> >> There is one feature in Linux that may affect you: If processes block on >> I/O (NFS also), the load increases, and the load is the _sum_, and not the >> _average_ of all CPUs. So if you have many CPUs, your abservable load will >> typically increase. Recently we had a load of 60, but nobody actually >> noticed ;-) >> >> So maybe you just need to adjust the limits for pacemaker... >> >> Regards, >> Ulrich >> >> >>> Oleg Ilyin schrieb am 13.12.2015 um 15:00 in >> Nachricht >> : >> > There are errors in my /var/log/messages >> > >> > >> > grep -e crmd\\[ -e crmd: /var/log/messages >> > Dec 13 00:01:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.29 >> > Dec 13 00:01:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.43 >> > Dec 13 00:02:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.90 >> > Dec 13 00:02:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.81 >> > Dec 13 00:04:25 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ >> > TIMER_POPPED origin=crm_timer_popped ] >> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: run_graph: >> Transition >> > 185166 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou >> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): >> > Complete >> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c >> > ause=C_FSA_INTERNAL origin=notify_crmd ] >> > Dec 13 00:08:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.48 >> > Dec 13 00:09:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.82 >> > Dec 13 00:09:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.13 >> > Dec 13 00:10:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.65 >> > Dec 13 00:10:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 6.00 >> > Dec 13 00:11:09 server_name_1 crmd[9941]: notice: >> throttle_handle_
Re: [ClusterLabs] Antw: Re: Resources start serial, not parralel
Hi Ulrich, thank you for your answer, what about which limits do you talk? The pacemaker software runs by root user, so limits should be increased for root. There is output from one of my server: -bash-4.1# id uid=0(root) gid=0(root) groups=0(root) -bash-4.1# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 124801 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 124801 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -bash-4.1# ps -ef |grep pacemaker root 17947 1 0 Dec11 ?00:00:09 pacemakerd 189 17953 17947 0 Dec11 ?00:00:11 /usr/libexec/pacemaker/cib root 17954 17947 0 Dec11 ?00:00:14 /usr/libexec/pacemaker/stonithd root 17955 17947 0 Dec11 ?00:00:11 /usr/libexec/pacemaker/lrmd 189 17956 17947 0 Dec11 ?00:00:09 /usr/libexec/pacemaker/attrd 189 17957 17947 0 Dec11 ?00:00:09 /usr/libexec/pacemaker/pengine root 17958 17947 0 Dec11 ?00:00:16 /usr/libexec/pacemaker/crmd With settings in pacemaker or system can be changed for start heavy application ( java ) at the same time? 2015-12-14 16:29 GMT+09:00 Ulrich Windl : > Hi! > > There is one feature in Linux that may affect you: If processes block on > I/O (NFS also), the load increases, and the load is the _sum_, and not the > _average_ of all CPUs. So if you have many CPUs, your abservable load will > typically increase. Recently we had a load of 60, but nobody actually > noticed ;-) > > So maybe you just need to adjust the limits for pacemaker... > > Regards, > Ulrich > > >>> Oleg Ilyin schrieb am 13.12.2015 um 15:00 in > Nachricht > : > > There are errors in my /var/log/messages > > > > > > grep -e crmd\\[ -e crmd: /var/log/messages > > Dec 13 00:01:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.29 > > Dec 13 00:01:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 4.43 > > Dec 13 00:02:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.90 > > Dec 13 00:02:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.81 > > Dec 13 00:04:25 server_name_1 crmd[9941]: notice: do_state_transition: > > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ > > TIMER_POPPED origin=crm_timer_popped ] > > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: run_graph: Transition > > 185166 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou > > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): > > Complete > > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: do_state_transition: > > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c > > ause=C_FSA_INTERNAL origin=notify_crmd ] > > Dec 13 00:08:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.48 > > Dec 13 00:09:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 5.82 > > Dec 13 00:09:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 4.13 > > Dec 13 00:10:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 4.65 > > Dec 13 00:10:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 6.00 > > Dec 13 00:11:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 5.27 > > Dec 13 00:11:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 4.26 > > Dec 13 00:12:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.60 > > Dec 13 00:12:39 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.97 > > Dec 13 00:13:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.93 > > Dec 13 00:14:09 server_name_1 crmd[9941]: notice: throttle_handle_load: > > High CPU load detected: 3.59 > &g
Re: [ClusterLabs] Resources start serial, not parralel
8 Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ TIMER_POPPED origin=crm_timer_popped ] Dec 13 00:34:26 server_name_1 crmd[9941]: notice: run_graph: Transition 185168 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): Complete Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c ause=C_FSA_INTERNAL origin=notify_crmd ] Dec 13 00:34:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.58 Dec 13 00:35:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.35 Dec 13 00:37:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.38 Dec 13 00:38:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.40 Dec 13 00:38:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.02 Dec 13 00:39:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.08 Dec 13 00:40:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.41 Dec 13 00:40:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.49 Dec 13 00:41:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.41 Dec 13 00:41:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.12 Dec 13 00:42:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 8.12 Dec 13 00:42:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 9.04 Dec 13 00:43:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 6.90 Dec 13 00:43:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.83 Dec 13 00:44:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 6.28 Dec 13 00:44:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.54 Dec 13 00:45:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.05 Dec 13 00:45:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.17 Dec 13 00:46:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.93 Dec 13 00:46:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.52 Dec 13 00:47:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.68 Dec 13 00:47:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 5.05 Dec 13 00:48:09 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 4.75 Dec 13 00:48:39 server_name_1 crmd[9941]: notice: throttle_handle_load: High CPU load detected: 3.37 Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ TIMER_POPPED origin=crm_timer_popped ] Dec 13 00:49:26 server_name_1 crmd[9941]: notice: run_graph: Transition 185169 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): Complete Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c ause=C_FSA_INTERNAL origin=notify_crmd ] 2015-12-13 21:11 GMT+09:00 Oleg Ilyin : > Hello, > could you help please with issue on my environment > RHEL 6.6 > corosync-1.4.7 > pacemaker-1.1.12 > > When I start several resources in the same time by " pcs resource enable " > command, > resource doesn't start parralel, they start serial. > > If I start resources by one, thay start parralel. I didn't do any special > settings. > > Please, > which parameters can affect to this behavior of cluster? > ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Resources start serial, not parralel
Hello, could you help please with issue on my environment RHEL 6.6 corosync-1.4.7 pacemaker-1.1.12 When I start several resources in the same time by " pcs resource enable " command, resource doesn't start parralel, they start serial. If I start resources by one, thay start parralel. I didn't do any special settings. Please, which parameters can affect to this behavior of cluster? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
