On Wed, Jul 19, 2023 at 8:33 PM Priyanka Balotra <priyanka.14balo...@gmail.com> wrote: > > Sure, > Here are the logs: > > > 63138:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (post_cache_update) debug: Updated cache after membership event 44. > 63139:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x200000000 > (A_ELECTION_CHECK) for controller set by post_cache_update:81 > 63140:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action) for > controller cleared by do_fsa_action:108 > 63141:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started) > info: Delaying start, Config not read (0000000000000040) > 63142:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (register_fsa_input_adv) debug: Stalling the FSA pending further input: > source=do_started cause=C_FSA_INTERNAL data=(nil) queue=0 > 63143:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000002 (with_actions) > for controller set by register_fsa_input_adv:88 > 63144:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (s_crmd_fsa) > debug: Exiting the FSA: queue=0, fsa_actions=0x200000002, stalled=true > 63145:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (config_query_callback) debug: Call 3 : Parsing CIB options > 63146:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (config_query_callback) debug: Shutdown escalation occurs if DC has not > responded to request in 1200000ms > 63147:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (config_query_callback) debug: Re-run scheduler after 900000ms of > inactivity > 63148:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pe_unpack_alerts) debug: Alert pf-ha-alert: > path=/usr/lib/ocf/resource.d/pacemaker/pf_ha_alert.sh timeout=30000ms > tstamp-format='%H:%M:%S.%06N' 0 vars > 63149:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action) for > controller cleared by do_fsa_action:108 > 63150:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started) > debug: Init server comms > 63151:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (qb_ipcs_us_publish) info: server name: crmd > 63152:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started) > notice: Pacemaker controller successfully started and accepting connections > 63153:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action) > for controller cleared by do_fsa_action:108 > 63154:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (do_election_check) debug: Ignoring election check because we are not > in an election > 63155:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000100100 > (new_actions) for controller set by s_crmd_fsa:198 > 63156:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (s_crmd_fsa) > debug: Processing I_PENDING: [ state=S_STARTING cause=C_FSA_INTERNAL > origin=do_started ] > 63157:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000 > (an_action) for controller cleared by do_fsa_action:108 > 63158:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_log) info: > Input I_PENDING received in state S_STARTING from do_started > 63159:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (do_state_transition) notice: State transition S_STARTING -> S_PENDING | > input=I_PENDING cause=C_FSA_INTERNAL origin=do_started > 63160:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000020 > (A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559 > 63161:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000080 > (A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565 > 63162:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action) for > controller cleared by do_fsa_action:108 > 63163:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action) for > controller cleared by do_fsa_action:108 > 63164:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00100000 (an_action) for > controller cleared by do_fsa_action:108 > 63165:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (do_cl_join_query) debug: Querying for a DC > 63166:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000100 (an_action) for > controller cleared by do_fsa_action:108 > 63167:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (controld_start_timer) debug: Started Election Trigger (inject > I_DC_TIMEOUT if pops after 20000ms, source=18) > 63168:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (stonith_api_signon) debug: Attempting fencer connection by > pacemaker-controld with mainloop > 63175:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:131085; real_size:135168; rb->word_size:33792 > 63176:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:131085; real_size:135168; rb->word_size:33792 > 63177:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:131085; real_size:135168; rb->word_size:33792 > 63178:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processing register 8 from client > pacemaker-controld.15962 with call options 0x00000000 > 63179:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processed register from client > pacemaker-controld.15962: OK (rc=0) > 63180:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (stonith_api_signon) debug: Connection to fencer by pacemaker-controld > succeeded (registration token: 5552b1b4-f725-46ac-b239-e404cadd8d94) > 63181:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processing st_notify 9 from client > pacemaker-controld.15962 with call options 0x00000000 > 63182:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] (handle_request) > debug: Enabling st_notify_disconnect callbacks for client > pacemaker-controld.15962 > 63183:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processed st_notify from client > pacemaker-controld.15962: OK (rc=0) > 63184:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processing st_notify 10 from client > pacemaker-controld.15962 with call options 0x00000000 > 63185:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] (handle_request) > debug: Enabling st_notify_fence callbacks for client > pacemaker-controld.15962 > 63186:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processed st_notify from client > pacemaker-controld.15962: OK (rc=0) > 63187:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processing st_notify 11 from client > pacemaker-controld.15962 with call options 0x00000000 > 63188:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] (handle_request) > debug: Enabling st_notify_history_synced callbacks for client > pacemaker-controld.15962 > 63189:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processed st_notify from client > pacemaker-controld.15962: OK (rc=0) > 63190:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (te_trigger_stonith_history_sync) info: Fence history will be synchronized > cluster-wide within 30 seconds > 63191:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] > (te_connect_stonith) notice: Fencer successfully connected > 63192:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) info: Quorum retained | membership=48 members=5 > 63193:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) debug: Member[0] 2 > 63194:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) debug: Member[1] 4 > 63195:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63196:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63197:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63198:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63199:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-e4qK7U/qb-request-cmap-header > 63200:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-e4qK7U/qb-response-cmap-header > 63201:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-e4qK7U/qb-event-cmap-header > 63202:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 4 > 63203:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 4 > 63204:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63205:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63206:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63209:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63210:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-YYxILU/qb-request-cmap-header > 63211:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-YYxILU/qb-response-cmap-header > 63212:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-YYxILU/qb-event-cmap-header > 63213:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 4 > 63214:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) info: Obtaining name for new node 4 > 63218:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63222:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63225:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63240:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63241:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-request-cmap-header > 63242:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-response-cmap-header > 63243:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-event-cmap-header > 63244:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 4 > 63245:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 4 > 63246:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) debug: Member[2] 3 > 63259:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63265:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63267:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63298:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63299:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-0DHKhX/qb-request-cmap-header > 63300:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-0DHKhX/qb-response-cmap-header > 63301:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-0DHKhX/qb-event-cmap-header > 63302:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 3 > 63303:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 3 > 63307:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63313:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63320:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63351:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63352:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-V0bQlV/qb-request-cmap-header > 63353:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-V0bQlV/qb-response-cmap-header > 63355:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-V0bQlV/qb-event-cmap-header > 63356:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 3 > 63357:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) info: Obtaining name for new node 3 > 63365:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63372:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63374:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63415:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63416:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-EAFzTX/qb-request-cmap-header > 63417:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-EAFzTX/qb-response-cmap-header > 63418:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-34-EAFzTX/qb-event-cmap-header > 63419:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 3 > 63420:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 3 > 63421:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) debug: Member[3] 6 > 63425:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63426:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63427:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63479:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63480:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-33-q3mFYU/qb-request-cmap-header > 63481:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-33-q3mFYU/qb-response-cmap-header > 63482:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-33-q3mFYU/qb-event-cmap-header > 63483:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 6 > 63484:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 6 > 63485:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63486:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63487:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63490:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63491:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-EcEbfV/qb-request-cmap-header > 63492:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-EcEbfV/qb-response-cmap-header > 63493:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-EcEbfV/qb-event-cmap-header > 63494:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 6 > 63495:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) info: Obtaining name for new node 6 > 63499:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63502:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63505:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63508:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63509:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-fLk4xW/qb-request-cmap-header > 63510:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-fLk4xW/qb-response-cmap-header > 63511:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-fLk4xW/qb-event-cmap-header > 63512:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 6 > 63513:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 6 > 63514:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) debug: Member[4] 5 > 63517:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63518:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63521:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63528:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63529:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-ushXmW/qb-request-cmap-header > 63530:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-ushXmW/qb-response-cmap-header > 63531:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-ushXmW/qb-event-cmap-header > 63532:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 5 > 63533:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 5 > 63534:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63535:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63536:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63537:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63538:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-x3qVkW/qb-request-cmap-header > 63539:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-x3qVkW/qb-response-cmap-header > 63540:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-x3qVkW/qb-event-cmap-header > 63541:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 5 > 63542:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] > (quorum_notification_cb) info: Obtaining name for new node 5 > 63543:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63544:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63545:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63546:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63547:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-gUNSFU/qb-request-cmap-header > 63548:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-gUNSFU/qb-response-cmap-header > 63549:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-gUNSFU/qb-event-cmap-header > 63550:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 5 > 63551:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 5 > 63552:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (update_peer_state_iter) notice: Node (null) state is now lost | nodeid=1 > previous=member source=pcmk__reap_unseen_nodes > 63553:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (post_cache_update) debug: Updated cache after membership event 48. > 63554:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x200000000 > (A_ELECTION_CHECK) for controller set by post_cache_update:81 > 63555:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action) > for controller cleared by do_fsa_action:108 > 63556:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (do_election_check) debug: Ignoring election check because we are not > in an election > 63557:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: node 2 pid 15962 joined > via cpg_join > 63558:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: FILE-2 (node 2 pid > 15962) is member > 63559:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63560:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63561:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63564:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63565:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-5PH1gV/qb-request-cmap-header > 63566:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-5PH1gV/qb-response-cmap-header > 63567:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-5PH1gV/qb-event-cmap-header > 63568:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 3 > 63569:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 3 > 63570:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 3 pid > 19250) is member > 63571:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[3] - > corosync-cpg is now online > 63572:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] > (peer_update_callback) debug: Sending hello to node 3 so that it learns > our node name > 63573:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63574:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63575:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63576:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63577:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-QATDEV/qb-request-cmap-header > 63578:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-QATDEV/qb-response-cmap-header > 63579:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-QATDEV/qb-event-cmap-header > 63580:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 4 > 63581:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 4 > 63582:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 4 pid > 19122) is member > 63583:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[4] - > corosync-cpg is now online > 63584:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] > (peer_update_callback) debug: Sending hello to node 4 so that it learns > our node name > 63585:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63586:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63587:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63588:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63589:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-TVzR1T/qb-request-cmap-header > 63590:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-TVzR1T/qb-response-cmap-header > 63591:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-TVzR1T/qb-event-cmap-header > 63592:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 5 > 63593:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 5 > 63594:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 5 pid > 19273) is member > 63595:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[5] - > corosync-cpg is now online > 63596:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] > (peer_update_callback) debug: Sending hello to node 5 so that it learns > our node name > 63597:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63598:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63599:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2) > debug: shm size:1048589; real_size:1052672; rb->word_size:263168 > 63600:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect() > 63601:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-8LRaoV/qb-request-cmap-header > 63602:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-8LRaoV/qb-response-cmap-header > 63603:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (qb_rb_close_helper) debug: Closing ringbuffer: > /dev/shm/qb-13142-15962-31-8LRaoV/qb-event-cmap-header > 63604:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (pcmk__corosync_name) info: Unable to get node name for nodeid 6 > 63605:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (get_node_name) > notice: Could not obtain a node name for corosync node with id 6 > 63606:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 6 pid > 19415) is member > 63607:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[6] - > corosync-cpg is now online > 63608:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (peer_update_callback) debug: Sending hello to node 6 so that it learns > our node name > 63609:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (get_xpath_object) debug: No match for //st_notify_history_synced in > /notify > 63610:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (stonith_api_del_notification) debug: Removing callback for > st_notify_history_synced events > 63611:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processing st_notify 12 from client > pacemaker-controld.15962 with call options 0x00000000 > 63612:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958] (handle_request) > debug: Disabling st_notify_history_synced callbacks for client > pacemaker-controld.15962 > 63613:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958] > (stonith_command) debug: Processed st_notify from client > pacemaker-controld.15962: OK (rc=0) > 63614:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (tengine_stonith_history_synced) debug: Fence-history synced - cancel all > timers > 63615:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (crm_get_peer) > info: Node 4 is now known as FILE-4 > 63616:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] > (update_peer_uname) warning: Node names with capitals are discouraged, > consider changing 'FILE-4' > 63617:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] > (peer_update_callback) info: Cluster node FILE-4 is now member > 63618:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] (crm_get_peer) > info: Node 3 is now known as FILE-3 > 63619:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] > (update_peer_uname) warning: Node names with capitals are discouraged, > consider changing 'FILE-3' > 63620:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] > (peer_update_callback) info: Cluster node FILE-3 is now member > 63621:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] (crm_get_peer) > info: Node 5 is now known as FILE-5 > 63622:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] > (update_peer_uname) warning: Node names with capitals are discouraged, > consider changing 'FILE-5' > 63623:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] > (peer_update_callback) info: Cluster node FILE-5 is now member > 63640:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (crm_get_peer) > info: Node 6 is now known as FILE-6 > 63641:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (update_peer_uname) warning: Node names with capitals are discouraged, > consider changing 'FILE-6' > 63642:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (peer_update_callback) info: Cluster node FILE-6 is now member > 63643:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (handle_request) > debug: Raising I_JOIN_OFFER: join-1 > 63644:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00400200 (new_actions) > for controller set by s_crmd_fsa:198 > 63645:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (s_crmd_fsa) > debug: Processing I_JOIN_OFFER: [ state=S_PENDING cause=C_HA_MESSAGE > origin=route_message ] > 63646:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action) for > controller cleared by do_fsa_action:108 > 63647:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00400000 (an_action) for > controller cleared by do_fsa_action:108 > 63648:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (update_dc) > info: Set DC to FILE-6 (3.11.0) > 63649:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__update_peer_expected) info: update_dc: Node FILE-6[6] - expected > state is now member (was (null)) > 63650:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000200 > (A_DC_TIMER_STOP) for controller set by do_cl_join_offer_respond:147 > 63651:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action) for > controller cleared by do_fsa_action:108 > 63788:Jul 17 14:16:32.884 FILE-2 pacemaker-controld [15962] > (do_cib_replaced) debug: Updating the CIB after a replace: DC=false > 63811:Jul 17 14:16:32.892 FILE-2 pacemaker-controld [15962] > (join_query_callback) debug: Respond to join offer join-1 from FILE-6 > 63819:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962] > (pcmk__procfs_pid_of) info: Found pacemaker-based active as process 15957 > 63820:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962] > (throttle_cib_load) debug: Init 6 + 2 ticks at 1689603415 (100 tps) > 63821:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962] (throttle_mode) > debug: Current load is 0.980000 across 10 core(s) > 63822:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962] > (throttle_send_command) info: New throttle mode: negligible load (was > undetermined) > 63823:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962] > (throttle_update) debug: Node FILE-2 has negligible load and supports at > most 20 jobs; new job limit 20 > 63824:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (handle_request) > debug: Raising I_JOIN_RESULT: join-1 > 63825:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00800000 (new_actions) > for controller set by s_crmd_fsa:198 > 63826:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (s_crmd_fsa) > debug: Processing I_JOIN_RESULT: [ state=S_PENDING cause=C_HA_MESSAGE > origin=route_message ] > 63827:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action) for > controller cleared by do_fsa_action:108 > 63828:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (do_cl_join_finalize_respond) debug: Confirming join-1: sending local > operation history to FILE-6 > 63829:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000200 > (new_actions) for controller set by s_crmd_fsa:198 > 63830:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (s_crmd_fsa) > debug: Processing I_NOT_DC: [ state=S_PENDING cause=C_HA_MESSAGE > origin=do_cl_join_finalize_respond ] > 63831:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000 > (an_action) for controller cleared by do_fsa_action:108 > 63832:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (do_log) info: > Input I_NOT_DC received in state S_PENDING from do_cl_join_finalize_respond > 63833:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (do_state_transition) notice: State transition S_PENDING -> S_NOT_DC | > input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond > 63834:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000020 > (A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559 > 63835:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__set_flags_as) debug: FSA action flags 0x00000080 > (A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565 > 63836:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action) for > controller cleared by do_fsa_action:108 > 63837:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action) for > controller cleared by do_fsa_action:108 > 63838:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] > (pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action) for > controller cleared by do_fsa_action:108 > 63863:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962] > (throttle_cib_load) debug: cib load: 0.000667 (2 ticks in 30s) > 63864:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962] (throttle_mode) > debug: Current load is 0.650000 across 10 core(s) > 63865:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962] > (throttle_cib_load) debug: cib load: 0.000333 (1 ticks in 30s) > 63866:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962] (throttle_mode) > debug: Current load is 0.850000 across 10 core(s) > 63868:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958] > (process_remote_stonith_exec) debug: Finalizing action 'reboot' > targeting FILE-2 on behalf of pacemaker-controld.19415@FILE-6: OK | rc=0 > id=4e523b34 > 63869:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958] (remote_op_done) > notice: Operation 'reboot' targeting FILE-2 by FILE-4 for > pacemaker-controld.19415@FILE-6: OK | id=4e523b34 > 63872:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962] > (exec_alert_list) info: Sending fencing alert via pf-ha-alert to (null) > 63875:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962] > (tengine_stonith_notify) crit: We were allegedly just fenced by FILE-4 for > FILE-6! > 63876:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962] > (crm_xml_cleanup) info: Cleaning up memory from libxml2 > 63877:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962] (crm_exit) > info: Exiting pacemaker-controld | with status 100 > 63900:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] > (pcmk_child_exit) warning: Shutting cluster down because > pacemaker-controld[15962] had fatal failure > 63902:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed stopped > 63956:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958] > (process_remote_stonith_exec) debug: Finalizing action 'reboot' > targeting FILE-1 on behalf of pacemaker-controld.19415@FILE-6: OK | rc=0 > id=446afc42 > 63957:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958] (remote_op_done) > notice: Operation 'reboot' targeting FILE-1 by FILE-5 for > pacemaker-controld.19415@FILE-6: OK | id=446afc42> > Thanks > Priyanka
Hi, node FILE-6 requested that node FILE-2 be fenced by node FILE-4. FILE-2's controller daemon received notification that it was being fenced, and it shut down. You'd want to check the logs on FILE-6 to determine why FILE-2 was fenced. > > On Thu, Jul 20, 2023 at 12:07 AM Ken Gaillot <kgail...@redhat.com> wrote: >> >> On Wed, 2023-07-19 at 23:49 +0530, Priyanka Balotra wrote: >> > Hi All, >> > I am using SLES 15 SP4. One of the nodes of the cluster is brought >> > down and boot up after sometime. Pacemaker service came up first but >> > later it faced a fatal shutdown. Due to that crm service is down. >> > >> > The logs from /var/log/pacemaker.pacemaker.log are as follows: >> > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] >> > (pcmk_child_exit) warning: Shutting cluster down because >> > pacemaker-controld[15962] had fatal failure >> >> The interesting messages will be before this. The ones with "pacemaker- >> controld" will be the most relevant, at least initially. >> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] >> > (pcmk_shutdown_worker) notice: Shutting down Pacemaker >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] >> > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed stopped >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] (stop_child) >> > notice: Stopping pacemaker-schedulerd | sent signal 15 to process >> > 15961 >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 >> > (invoking handler) >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] >> > (qb_ipcs_us_withdraw) info: withdrawing server sockets >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] >> > (qb_ipcs_unref) debug: qb_ipcs_unref() - destroying >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] >> > (crm_xml_cleanup) info: Cleaning up memory from libxml2 >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_exit) >> > info: Exiting pacemaker-schedulerd | with status 0 >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957] >> > (qb_ipcs_event_sendv) debug: new_event_notification (/dev/shm/qb- >> > 15957-15962-12-RDPw6O/qb): Broken pipe (32) >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957] >> > (cib_notify_send_one) warning: Could not notify client crmd: >> > Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957] >> > (cib_process_request) info: Completed cib_delete operation for >> > section //node_state[@uname='FILE-2']/*: OK (rc=0, origin=FILE- >> > 6/crmd/74, version=0.24.75) >> > Jul 17 14:18:20.093 FILE-2 pacemaker-fenced [15958] >> > (xml_patch_version_check) debug: Can apply patch 0.24.75 to >> > 0.24.74 >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] >> > (pcmk_child_exit) info: pacemaker-schedulerd[15961] exited >> > with status 0 (OK) >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957] >> > (cib_process_request) info: Completed cib_modify operation for >> > section status: OK (rc=0, origin=FILE-6/crmd/75, version=0.24.75) >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] >> > (pcmk_shutdown_worker) debug: pacemaker-schedulerd confirmed >> > stopped >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] (stop_child) >> > notice: Stopping pacemaker-attrd | sent signal 15 to process 15960 >> > Jul 17 14:18:20.093 FILE-2 pacemaker-attrd [15960] >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 >> > (invoking handler) >> > >> > Could you please help me understand the issue here. >> > >> > Regards >> > Priyanka >> > _______________________________________________ >> > Manage your subscription: >> > https://lists.clusterlabs.org/mailman/listinfo/users >> > >> > ClusterLabs home: https://www.clusterlabs.org/ >> -- >> Ken Gaillot <kgail...@redhat.com> >> >> _______________________________________________ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ > > _______________________________________________ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ -- Regards, Reid Wahl (He/Him) Senior Software Engineer, Red Hat RHEL High Availability - Pacemaker _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
