Hi guys, have you any ideas for root cause of trouble? I will appreciate any clues for investigation.
2015-12-14 17:40 GMT+09:00 Oleg Ilyin <[email protected]>: > Hi Ulrich, > > thank you for your answer, > > what about which limits do you talk? > > The pacemaker software runs by root user, so limits should be increased > for root. > There is output from one of my server: > > -bash-4.1# id > uid=0(root) gid=0(root) groups=0(root) > > -bash-4.1# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 124801 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 124801 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > -bash-4.1# ps -ef |grep pacemaker > root 17947 1 0 Dec11 ? 00:00:09 pacemakerd > 189 17953 17947 0 Dec11 ? 00:00:11 /usr/libexec/pacemaker/cib > root 17954 17947 0 Dec11 ? 00:00:14 > /usr/libexec/pacemaker/stonithd > root 17955 17947 0 Dec11 ? 00:00:11 /usr/libexec/pacemaker/lrmd > 189 17956 17947 0 Dec11 ? 00:00:09 > /usr/libexec/pacemaker/attrd > 189 17957 17947 0 Dec11 ? 00:00:09 > /usr/libexec/pacemaker/pengine > root 17958 17947 0 Dec11 ? 00:00:16 /usr/libexec/pacemaker/crmd > > > With settings in pacemaker or system can be changed for start heavy > application ( java ) at the same time? > > > > > 2015-12-14 16:29 GMT+09:00 Ulrich Windl <[email protected] > >: > >> Hi! >> >> There is one feature in Linux that may affect you: If processes block on >> I/O (NFS also), the load increases, and the load is the _sum_, and not the >> _average_ of all CPUs. So if you have many CPUs, your abservable load will >> typically increase. Recently we had a load of 60, but nobody actually >> noticed ;-) >> >> So maybe you just need to adjust the limits for pacemaker... >> >> Regards, >> Ulrich >> >> >>> Oleg Ilyin <[email protected]> schrieb am 13.12.2015 um 15:00 in >> Nachricht >> <CANLWoXrpvLAbcgCOE073iAOJz3zL7RAUB4UQC=0ipfgytvn...@mail.gmail.com>: >> > There are errors in my /var/log/messages >> > >> > >> > grep -e crmd\\[ -e crmd: /var/log/messages >> > Dec 13 00:01:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.290000 >> > Dec 13 00:01:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.430000 >> > Dec 13 00:02:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.900000 >> > Dec 13 00:02:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.810000 >> > Dec 13 00:04:25 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ >> > TIMER_POPPED origin=crm_timer_popped ] >> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: run_graph: >> Transition >> > 185166 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou >> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): >> > Complete >> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c >> > ause=C_FSA_INTERNAL origin=notify_crmd ] >> > Dec 13 00:08:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.480000 >> > Dec 13 00:09:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.820000 >> > Dec 13 00:09:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.130000 >> > Dec 13 00:10:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.650000 >> > Dec 13 00:10:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 6.000000 >> > Dec 13 00:11:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.270000 >> > Dec 13 00:11:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.260000 >> > Dec 13 00:12:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.600000 >> > Dec 13 00:12:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.970000 >> > Dec 13 00:13:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.930000 >> > Dec 13 00:14:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.590000 >> > Dec 13 00:17:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.350000 >> > Dec 13 00:18:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.220000 >> > Dec 13 00:18:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.830000 >> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ >> > TIMER_POPPED origin=crm_timer_popped ] >> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: run_graph: >> Transition >> > 185167 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou >> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): >> > Complete >> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c >> > ause=C_FSA_INTERNAL origin=notify_crmd ] >> > Dec 13 00:24:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.770000 >> > Dec 13 00:24:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.360000 >> > Dec 13 00:25:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.790000 >> > Dec 13 00:26:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.470000 >> > Dec 13 00:27:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.890000 >> > Dec 13 00:27:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.290000 >> > Dec 13 00:28:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.040000 >> > Dec 13 00:29:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.340000 >> > Dec 13 00:29:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.250000 >> > Dec 13 00:30:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.310000 >> > Dec 13 00:31:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.880000 >> > Dec 13 00:31:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.890000 >> > Dec 13 00:32:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.320000 >> > Dec 13 00:32:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.380000 >> > Dec 13 00:33:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.430000 >> > Dec 13 00:33:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.480000 >> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ >> > TIMER_POPPED origin=crm_timer_popped ] >> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: run_graph: >> Transition >> > 185168 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou >> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): >> > Complete >> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c >> > ause=C_FSA_INTERNAL origin=notify_crmd ] >> > Dec 13 00:34:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.580000 >> > Dec 13 00:35:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.350000 >> > Dec 13 00:37:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.380000 >> > Dec 13 00:38:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.400000 >> > Dec 13 00:38:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.020000 >> > Dec 13 00:39:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.080000 >> > Dec 13 00:40:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.410000 >> > Dec 13 00:40:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.490000 >> > Dec 13 00:41:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.410000 >> > Dec 13 00:41:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.120000 >> > Dec 13 00:42:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 8.120000 >> > Dec 13 00:42:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 9.040000 >> > Dec 13 00:43:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 6.900000 >> > Dec 13 00:43:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.830000 >> > Dec 13 00:44:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 6.280000 >> > Dec 13 00:44:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.540000 >> > Dec 13 00:45:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.050000 >> > Dec 13 00:45:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.170000 >> > Dec 13 00:46:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.930000 >> > Dec 13 00:46:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.520000 >> > Dec 13 00:47:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.680000 >> > Dec 13 00:47:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 5.050000 >> > Dec 13 00:48:09 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 4.750000 >> > Dec 13 00:48:39 server_name_1 crmd[9941]: notice: >> throttle_handle_load: >> > High CPU load detected: 3.370000 >> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ >> > TIMER_POPPED origin=crm_timer_popped ] >> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: run_graph: >> Transition >> > 185169 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou >> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2): >> > Complete >> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition: >> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c >> > ause=C_FSA_INTERNAL origin=notify_crmd ] >> > >> > >> > 2015-12-13 21:11 GMT+09:00 Oleg Ilyin <[email protected]>: >> > >> >> Hello, >> >> could you help please with issue on my environment >> >> RHEL 6.6 >> >> corosync-1.4.7 >> >> pacemaker-1.1.12 >> >> >> >> When I start several resources in the same time by " pcs resource >> enable " >> >> command, >> >> resource doesn't start parralel, they start serial. >> >> >> >> If I start resources by one, thay start parralel. I didn't do any >> special >> >> settings. >> >> >> >> Please, >> >> which parameters can affect to this behavior of cluster? >> >> >> >> >> >> >> _______________________________________________ >> Users mailing list: [email protected] >> http://clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> > >
_______________________________________________ Users mailing list: [email protected] http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
