As requested, the original active was dhoyt-sc-1 and the reboot was issued at 10:52:11. I’ve provided the /var/log/messages from both dhoyt-sc-1 and dhoyt-sc-2. As before, once dhoyt-sc-2 detects that dhoyt-sc-1 is down, dhoyt-sc-2 takes activity, but the consensus takeover_request gets REJECTED.
[2020-07-08 10:51:53 root@dhoyt-sc-1: ~ ]# date; reboot Wed Jul 8 10:52:11 EDT 2020 Connection to 172.23.8.176 closed by remote host. Connection to 172.23.8.176 closed. /var/log/messages: Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c14 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c10 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped target Cloud-init target. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Time & Date Service... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session 872 of user admin. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped target rpc_pipefs.target. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping LVM2 PV scan on device 252:2... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c3 of user root. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c1 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Closed LVM2 poll daemon socket. Jul 8 10:52:11 dhoyt-sc-1 systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Getty on tty1... Jul 8 10:52:11 dhoyt-sc-1 systemd: Unmounting RPC Pipe File System... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c37 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped target Timers. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Daily Cleanup of Temporary Directories. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped target Multi-User System. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping IPMI Driver... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Resets System Activity Logs. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Command Scheduler... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Dynamic System Tuning Daemon... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping OpenSAF daemon... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Entropy Daemon based on the HAVEGE algorithm... Jul 8 10:52:11 dhoyt-sc-1 haveged: haveged: Stopping due to signal 15 Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping The Apache HTTP Server... Jul 8 10:52:11 dhoyt-sc-1 haveged: haveged starting up Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping irqbalance daemon... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Network Time Service... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c9 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c6 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping A lightweight connection pooler for PostgreSQL... Jul 8 10:52:11 dhoyt-sc-1 pgbouncer: 2020-07-08 10:52:11.316 EDT [1497] LOG got SIGINT, shutting down Jul 8 10:52:11 dhoyt-sc-1 avahi-daemon[1058]: Got SIGTERM, quitting. Jul 8 10:52:11 dhoyt-sc-1 avahi-daemon[1058]: Leaving mDNS multicast group on interface eth0.IPv4 with address 172.23.8.176. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Avahi mDNS/DNS-SD Stack... Jul 8 10:52:11 dhoyt-sc-1 avahi-daemon[1058]: avahi-daemon 0.6.31 exiting. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Execute cloud user/final scripts. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping OpenSSH server daemon... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Authorization Manager... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session 865 of user admin. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Serial Getty on ttyS0... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Live Syncing (Mirror) Daemon... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session 967 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session 572 of user root. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Dump dmesg to /var/log/dmesg. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c12 of user insight. Jul 8 10:52:11 dhoyt-sc-1 snmpd[2153]: Received TERM or STOP signal... shutting down... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c13 of user insight. Jul 8 10:52:11 dhoyt-sc-1 opensafd: Stopping OpenSAF Services Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session 873 of user admin. Jul 8 10:52:11 dhoyt-sc-1 osafamfnd[3507]: NO Shutdown initiated Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Apply the settings specified in cloud-config. Jul 8 10:52:11 dhoyt-sc-1 osafamfnd[3507]: NO Removing assignments from AMF components Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Sendmail Mail Transport Client... Jul 8 10:52:11 dhoyt-sc-1 insight-saf-process-monitor: IN CSI remove callback is invoked Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped target Cloud-config availability. Jul 8 10:52:11 dhoyt-sc-1 insight-saf-process-monitor: IN Removing assignment of safCsi=ProcessMonitor,safSi=ProcessMonitor-SI,safApp=Insight from safComp=ProcessMonitor,safSu=SU1,safSg=ProcessMonitor,safApp=Insight Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Session c11 of user insight. Jul 8 10:52:11 dhoyt-sc-1 pgbouncer: 2020-07-08 10:52:11.436 EDT [1497] LOG server connections dropped, exiting Jul 8 10:52:11 dhoyt-sc-1 osafamfnd[3507]: NO Removed 'safSi=ProcessMonitor-SI,safApp=Insight' from 'safSu=SU1,safSg=ProcessMonitor,safApp=Insight' Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Simple Network Management Protocol (SNMP) Daemon.... Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Avahi mDNS/DNS-SD Stack. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped irqbalance daemon. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Authorization Manager. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Entropy Daemon based on the HAVEGE algorithm. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped OpenSSH server daemon. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Command Scheduler. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Serial Getty on ttyS0. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Time & Date Service. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c1 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Sendmail Mail Transport Client. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Getty on tty1. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Network Time Service. Jul 8 10:52:11 dhoyt-sc-1 systemd: lsyncd.service: main process exited, code=exited, status=143/n/a Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Live Syncing (Mirror) Daemon. Jul 8 10:52:11 dhoyt-sc-1 systemd: Unit lsyncd.service entered failed state. Jul 8 10:52:11 dhoyt-sc-1 systemd: lsyncd.service failed. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped LVM2 PV scan on device 252:2. Jul 8 10:52:11 dhoyt-sc-1 systemd: Unmounted RPC Pipe File System. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session 572 of user root. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Simple Network Management Protocol (SNMP) Daemon.. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped A lightweight connection pooler for PostgreSQL. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session 967 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session 873 of user admin. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c37 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Removed slice system-lvm2\x2dpvscan.slice. Jul 8 10:52:11 dhoyt-sc-1 systemd: Starting Show Plymouth Reboot Screen... Jul 8 10:52:11 dhoyt-sc-1 systemd: Starting Restore /run/initramfs... Jul 8 10:52:11 dhoyt-sc-1 systemd: Removed slice system-getty.slice. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopping Sendmail Mail Transport Agent... Jul 8 10:52:11 dhoyt-sc-1 systemd: Removed slice system-serial\x2dgetty.slice. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped /etc/rc.d/rc.local Compatibility. Jul 8 10:52:11 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=DataHandler,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Sendmail Mail Transport Agent. Jul 8 10:52:11 dhoyt-sc-1 systemd-logind: Removed session 873. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped IPMI Driver. Jul 8 10:52:11 dhoyt-sc-1 systemd: Requested transaction contradicts existing jobs: Transaction is destructive. Jul 8 10:52:11 dhoyt-sc-1 systemd-logind: Failed to start session scope session-c1504.scope: Transaction is destructive. Jul 8 10:52:11 dhoyt-sc-1 systemd: Started Show Plymouth Reboot Screen. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c14 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c13 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c11 of user insight. Jul 8 10:52:11 dhoyt-sc-1 dbus[1067]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' Jul 8 10:52:11 dhoyt-sc-1 dbus[1067]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.timedate1.service': Refusing activation, D-Bus is shutting down. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c9 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd: Requested transaction contradicts existing jobs: Transaction is destructive. Jul 8 10:52:11 dhoyt-sc-1 systemd: Stopped Session c10 of user insight. Jul 8 10:52:11 dhoyt-sc-1 systemd-logind: Failed to start session scope session-c1505.scope: Transaction is destructive. Jul 8 10:52:12 dhoyt-sc-1 dbus[1067]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' Jul 8 10:52:12 dhoyt-sc-1 dbus[1067]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.timedate1.service': Refusing activation, D-Bus is shutting down. Jul 8 10:52:12 dhoyt-sc-1 systemd: Stopped Dynamic System Tuning Daemon. Jul 8 10:52:12 dhoyt-sc-1 systemd: Stopping D-Bus System Message Bus... Jul 8 10:52:12 dhoyt-sc-1 systemd: Stopped D-Bus System Message Bus. Jul 8 10:52:12 dhoyt-sc-1 systemd: Stopped The Apache HTTP Server. Jul 8 10:52:12 dhoyt-sc-1 osafamfnd[3507]: NO Healthcheck failed for 'safComp=KafkaConnect,safSu=SU1,safSg=Nway,safApp=Insight' Jul 8 10:52:12 dhoyt-sc-1 osafamfnd[3507]: NO Reason:'Exec of script success, but script exits with non-zero status' Jul 8 10:52:12 dhoyt-sc-1 osafamfnd[3507]: NO Exit code: 1 Jul 8 10:52:12 dhoyt-sc-1 osafamfnd[3507]: NO 'safComp=KafkaConnect,safSu=SU1,safSg=Nway,safApp=Insight' faulted due to 'healthCheckFailed' Jul 8 10:52:12 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=InsightHotDeploy,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:12 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=KafkaConnect,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:12 dhoyt-sc-1 systemd: Stopped Session c3 of user root. Jul 8 10:52:12 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=SBCManager,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:13 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=FtpCollector,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Session c12 of user insight. Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Session c6 of user insight. Jul 8 10:52:13 dhoyt-sc-1 systemd-logind: Failed to abandon session scope: Transport endpoint is not connected Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Session 872 of user admin. Jul 8 10:52:13 dhoyt-sc-1 systemd-logind: Failed to abandon session scope: Transport endpoint is not connected Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Session 865 of user admin. Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopping Permit User Sessions... Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopping Login Service... Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Permit User Sessions. Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Flush Journal to Persistent Storage. Jul 8 10:52:13 dhoyt-sc-1 systemd: Stopped Login Service. Jul 8 10:52:13 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=JmxCollector,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:13 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=SnmpCollector,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:14 dhoyt-sc-1 insight-saf-warmDeploy: IN CSI remove callback is invoked Jul 8 10:52:14 dhoyt-sc-1 insight-saf-warmDeploy: IN Removing assignment of safCsi=InsightWarmDeploy,safSi=Insight-SI,safApp=Insight from safComp=InsightWarmDeploy,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:14 dhoyt-sc-1 insight-saf-warmDeploy: IN Executing /opt/sonus/ems/conf/opensaf/AS/insight_warm_deploy stop Jul 8 10:52:14 dhoyt-sc-1 systemd: Started Restore /run/initramfs. Jul 8 10:52:14 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=FmTrapReceiver,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:16 dhoyt-sc-1 systemd: Received SIGRTMIN+20 from PID 23542 (plymouthd). Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-mirroring: IN CSI remove callback is invoked Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-mirroring: IN Removing assignment of safCsi=KafkaMirroring,safSi=Insight-SI,safApp=Insight from safComp=KafkaMirroring,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-mirroring: IN Executing /opt/sonus/ems/conf/opensaf/AS/kafka_mirroring_actionscript stop Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-schema-registry: IN CSI remove callback is invoked Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-schema-registry: IN Removing assignment of safCsi=KafkaSchemaRegistry,safSi=Insight-SI,safApp=Insight from safComp=KafkaSchemaRegistry,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:20 dhoyt-sc-1 insight-saf-kafka-schema-registry: IN Executing /opt/sonus/ems/conf/opensaf/AS/kafka_schema_registry_actionscript stop Jul 8 10:52:24 dhoyt-sc-1 insight-saf-kafka: IN CSI remove callback is invoked Jul 8 10:52:24 dhoyt-sc-1 insight-saf-kafka: IN Removing assignment of safCsi=Kafka,safSi=Insight-SI,safApp=Insight from safComp=Kafka,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:24 dhoyt-sc-1 insight-saf-kafka: IN Executing /opt/sonus/ems/conf/opensaf/AS/kafka_actionscript stop Jul 8 10:52:24 dhoyt-sc-1 osafamfnd[3507]: NO Healthcheck failed for 'safComp=FileReplication,safSu=SU1,safSg=Nway,safApp=Insight' Jul 8 10:52:24 dhoyt-sc-1 osafamfnd[3507]: NO Reason:'Exec of script success, but script exits with non-zero status' Jul 8 10:52:24 dhoyt-sc-1 osafamfnd[3507]: NO Exit code: 1 Jul 8 10:52:27 dhoyt-sc-1 osafamfnd[3507]: NO 'safComp=NetcoolObjectServer,safSu=SU1,safSg=Nway,safApp=Insight' faulted due to 'healthCheckFailed' Jul 8 10:52:27 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=NetcoolObjectServer,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:28 dhoyt-sc-1 insight-saf-owl: IN CSI remove callback is invoked Jul 8 10:52:28 dhoyt-sc-1 insight-saf-owl: IN Removing assignment of safCsi=Owl,safSi=Insight-SI,safApp=Insight from safComp=Owl,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:28 dhoyt-sc-1 insight-saf-owl: IN Executing /opt/sonus/ems/conf/opensaf/AS/owl_actionscript stop Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vip: IN CSI remove callback is invoked Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vip: IN Removing assignment of safCsi=Vip,safSi=Insight-SI,safApp=Insight from safComp=Vip,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vip: IN Executing /opt/opensaf/osaf-vip stop Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vnfr: IN CSI remove callback is invoked Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vnfr: IN Removing assignment of safCsi=Vnfr,safSi=Insight-SI,safApp=Insight from safComp=Vnfr,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:28 dhoyt-sc-1 insight-saf-vnfr: IN Executing /opt/sonus/ems/conf/opensaf/AS/vnfr_actionscript stop Jul 8 10:52:28 dhoyt-sc-1 insight-saf-agent: IN CSI remove callback is invoked Jul 8 10:52:28 dhoyt-sc-1 insight-saf-agent: IN Removing assignment of safCsi=Agent,safSi=Insight-SI,safApp=Insight from safComp=Agent,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:28 dhoyt-sc-1 insight-saf-agent: IN Executing /opt/sonus/ems/conf/opensaf/AS/agent_actionscript stop Jul 8 10:52:34 dhoyt-sc-1 insight-saf-dbaas: IN CSI remove callback is invoked Jul 8 10:52:34 dhoyt-sc-1 insight-saf-dbaas: IN Removing assignment of safCsi=DBaaS,safSi=Insight-SI,safApp=Insight from safComp=DBaaS,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:34 dhoyt-sc-1 insight-saf-dbaas: IN Executing /opt/sonus/ems/conf/opensaf/osaf-dbaas stop Jul 8 10:52:34 dhoyt-sc-1 su: (to postgres) root on none Jul 8 10:52:34 dhoyt-sc-1 insight-saf-apache: IN CSI remove callback is invoked Jul 8 10:52:34 dhoyt-sc-1 insight-saf-apache: IN Removing assignment of safCsi=Apache,safSi=Insight-SI,safApp=Insight from safComp=Apache,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:34 dhoyt-sc-1 insight-saf-apache: IN Executing /opt/sonus/ems/conf/opensaf/AS/apacheproxy_actionscript stop Jul 8 10:52:36 dhoyt-sc-1 osafamfnd[3507]: NO Healthcheck failed for 'safComp=SonusTrapProbe,safSu=SU1,safSg=Nway,safApp=Insight' Jul 8 10:52:36 dhoyt-sc-1 osafamfnd[3507]: NO Reason:'Exec of script success, but script exits with non-zero status' Jul 8 10:52:36 dhoyt-sc-1 osafamfnd[3507]: NO Exit code: 1 Jul 8 10:52:36 dhoyt-sc-1 osafamfnd[3507]: NO 'safComp=SonusTrapProbe,safSu=SU1,safSg=Nway,safApp=Insight' faulted due to 'healthCheckFailed' Jul 8 10:52:36 dhoyt-sc-1 osaf-insight-npi: Stopping safComp=SonusTrapProbe,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:38 dhoyt-sc-1 insight-saf-zookeeper: IN CSI remove callback is invoked Jul 8 10:52:38 dhoyt-sc-1 insight-saf-zookeeper: IN Removing assignment of safCsi=Zookeeper,safSi=Insight-SI,safApp=Insight from safComp=Zookeeper,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:38 dhoyt-sc-1 insight-saf-zookeeper: IN Executing /opt/sonus/ems/conf/opensaf/AS/zookeeper_actionscript stop Jul 8 10:52:42 dhoyt-sc-1 insight-saf-callTraceListener: IN CSI remove callback is invoked Jul 8 10:52:42 dhoyt-sc-1 insight-saf-callTraceListener: IN Removing assignment of safCsi=CallTraceListener,safSi=Insight-SI,safApp=Insight from safComp=CallTraceListener,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:42 dhoyt-sc-1 insight-saf-callTraceListener: IN Executing /opt/sonus/ems/conf/opensaf/AS/calltracelistener_actionscript stop Jul 8 10:52:42 dhoyt-sc-1 insight-saf-netcoolLicenseMgr: IN CSI remove callback is invoked Jul 8 10:52:42 dhoyt-sc-1 insight-saf-netcoolLicenseMgr: IN Removing assignment of safCsi=NetcoolLicenseManager,safSi=Insight-SI,safApp=Insight from safComp=NetcoolLicenseManager,safSu=SU1,safSg=Nway,safApp=Insight Jul 8 10:52:42 dhoyt-sc-1 insight-saf-netcoolLicenseMgr: IN Executing /opt/sonus/ems/conf/opensaf/AS/netcoolLicenseMgr_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osafamfnd[3507]: NO Removed 'safSi=Insight-SI,safApp=Insight' from 'safSu=SU1,safSg=Nway,safApp=Insight' Jul 8 10:52:47 dhoyt-sc-1 osafamfnd[3507]: NO Removed assignments from AMF components Jul 8 10:52:47 dhoyt-sc-1 osafamfnd[3507]: NO Terminating all AMF components Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-agent Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-dbaas Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-callTraceListener Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-apache Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-netcoolLicenseMgr Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-kafka-schema-registry Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-zookeeper Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-kafka Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-owl Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-vnfr Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-warmDeploy Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-vip Jul 8 10:52:47 dhoyt-sc-1 osaf-insight-pm: cleaning up /opt/opensaf/insight-saf-process-monitor Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: cleaning up /opt/opensaf/insight-saf-kafka-mirroring Jul 8 10:52:47 dhoyt-sc-1 osaffmd[3114]: exiting for shutdown, (sigterm from pid 26736 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafamfwd[4400]: exiting for shutdown, (sigterm from pid 26730 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafsmfnd[4315]: exiting for shutdown, (sigterm from pid 26776 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osaflogd[3431]: exiting for shutdown, (sigterm from pid 26786 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafntfd[3450]: exiting for shutdown, (sigterm from pid 26762 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafsmfd[4357]: exiting for shutdown, (sigterm from pid 26771 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafclmna[3082]: exiting for shutdown, (sigterm from pid 26731 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafclmd[3467]: exiting for shutdown, (sigterm from pid 26728 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafrded[3098]: exiting for shutdown, (sigterm from pid 26785 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer locally disconnected. Marking it as doomed 6 <577, 2010f> (@safSmf_applier1) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer locally disconnected. Marking it as doomed 5 <579, 2010f> (safSmfService) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer locally disconnected. Marking it as doomed 1 <8, 2010f> (safLogService) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer locally disconnected. Marking it as doomed 2 <11, 2010f> (@safLogService_appl) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer locally disconnected. Marking it as doomed 3 <16, 2010f> (safClmService) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer disconnected 6 <577, 2010f> (@safSmf_applier1) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer disconnected 5 <579, 2010f> (safSmfService) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer disconnected 1 <8, 2010f> (safLogService) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer disconnected 2 <11, 2010f> (@safLogService_appl) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: NO Implementer disconnected 3 <16, 2010f> (safClmService) Jul 8 10:52:47 dhoyt-sc-1 osafimmd[3131]: exiting for shutdown, (sigterm from pid 26769 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: WA DISCARD DUPLICATE FEVS message:2781 Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: WA Error code 2 returned for message type 82 - ignoring Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: WA DISCARD DUPLICATE FEVS message:2782 Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: WA Error code 2 returned for message type 82 - ignoring Jul 8 10:52:47 dhoyt-sc-1 osafimmnd[3149]: exiting for shutdown, (sigterm from pid 26999 uid 0) Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/agent_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/osaf-dbaas stop_immediate Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/calltracelistener_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/netcoolLicenseMgr_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/apacheproxy_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/kafka_schema_registry_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/opensaf/osaf-vip stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/kafka_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/kafka_mirroring_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/owl_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/zookeeper_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/insight_warm_deploy stop Jul 8 10:52:47 dhoyt-sc-1 osaf-insight: Executing /opt/sonus/ems/conf/opensaf/AS/vnfr_actionscript stop Jul 8 10:52:47 dhoyt-sc-1 su: (to postgres) root on none Jul 8 10:52:52 dhoyt-sc-1 osafamfnd[3507]: NO Terminated all AMF components Jul 8 10:52:52 dhoyt-sc-1 osafamfnd[3507]: NO Shutdown completed, exiting Jul 8 10:52:52 dhoyt-sc-1 osafamfnd[3507]: exiting for shutdown, (sigterm from pid -1 uid -1) Jul 8 10:52:52 dhoyt-sc-1 osafamfd[3484]: exiting for shutdown, (sigterm from pid -1 uid -1) Jul 8 10:52:53 dhoyt-sc-1 osaftransportd[3077]: exiting for shutdown, (sigterm from pid 28489 uid 0) Jul 8 10:52:54 dhoyt-sc-1 opensafd: OpenSAF services successfully stopped Jul 8 10:52:54 dhoyt-sc-1 opensafd: Stopping OpenSAF Services: [ OK ] Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped OpenSAF daemon. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopping System Logging Service... Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped target Remote File Systems. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped target Remote File Systems (Pre). Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped target NFS client services. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopping GSSAPI Proxy Daemon... Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopping Logout off all iSCSI sessions on shutdown... Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped GSSAPI Proxy Daemon. Jul 8 10:52:54 dhoyt-sc-1 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-52.el7" x-pid="9206" x-info="http://www.rsyslog.com";] exiting on signal 15. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped System Logging Service. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped target Network is Online. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped Initial cloud-init job (metadata service crawler). Jul 8 10:52:54 dhoyt-sc-1 iscsiadm: iscsiadm: No matching sessions found Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped Logout off all iSCSI sessions on shutdown. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped target Network. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopped Rollback uncommitted netcf network config change transactions. Jul 8 10:52:54 dhoyt-sc-1 systemd: Stopping LSB: Bring up/down networking... Jul 8 10:52:54 dhoyt-sc-1 network: Shutting down interface eth0: [FAILED] Jul 8 10:52:55 dhoyt-sc-1 network: Shutting down loopback interface: [ OK ] Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped LSB: Bring up/down networking. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Network (Pre). Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Initial cloud-init job (pre-networking). Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopping IPv6 firewall with ip6tables... Jul 8 10:52:55 dhoyt-sc-1 ip6tables.init: ip6tables: Saving firewall rules to /etc/sysconfig/ip6tables: [ OK ] Jul 8 10:52:55 dhoyt-sc-1 ip6tables.init: ip6tables: Setting chains to policy ACCEPT: nat filter [ OK ] Jul 8 10:52:55 dhoyt-sc-1 ip6tables.init: ip6tables: Flushing firewall rules: [ OK ] Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped IPv6 firewall with ip6tables. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopping IPv4 firewall with iptables... Jul 8 10:52:55 dhoyt-sc-1 iptables.init: iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ] Jul 8 10:52:55 dhoyt-sc-1 iptables.init: iptables: Setting chains to policy ACCEPT: nat filter [ OK ] Jul 8 10:52:55 dhoyt-sc-1 iptables.init: iptables: Flushing firewall rules: [ OK ] Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped IPv4 firewall with iptables. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Basic System. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Paths. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Flexible branding. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Forward Password Requests to Wall Directory Watch. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Forward Password Requests to Plymouth Directory Watch. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Sockets. Jul 8 10:52:55 dhoyt-sc-1 systemd: Closed Open-iSCSI iscsiuio Socket. Jul 8 10:52:55 dhoyt-sc-1 systemd: Closed Avahi mDNS/DNS-SD Stack Activation Socket. Jul 8 10:52:55 dhoyt-sc-1 systemd: Closed D-Bus System Message Bus Socket. Jul 8 10:52:55 dhoyt-sc-1 systemd: Closed Open-iSCSI iscsid Socket. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target System Initialization. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Read and set NIS domainname from /etc/sysconfig/network. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopping Update UTMP about System Boot/Shutdown... Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Set Up Additional Binary Formats. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Apply Kernel Variables. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Local Encrypted Volumes. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Import network configuration from initramfs. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Setup Virtual Console. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopping Load/Save Random Seed... Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Slices. Jul 8 10:52:55 dhoyt-sc-1 systemd: Removed slice User and Session Slice. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Update UTMP about System Boot/Shutdown. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Load/Save Random Seed. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopping Security Auditing Service... Jul 8 10:52:55 dhoyt-sc-1 auditd[1032]: The audit daemon is exiting. Jul 8 10:52:55 dhoyt-sc-1 kernel: type=1305 audit(1594219975.841:14855): audit_pid=0 old=1032 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 Jul 8 10:52:55 dhoyt-sc-1 kernel: type=1131 audit(1594219975.842:14856): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Security Auditing Service. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Local File Systems. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /var/log... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /run/user/2056... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /var/lib/pgsql... Jul 8 10:52:55 dhoyt-sc-1 kernel: type=1131 audit(1594219975.845:14857): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped Configure read-only root support. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting Configuration File System... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /opt/raid... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /run/user/1000... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /boot... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /home... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /tmp... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /run/user/0... Jul 8 10:52:55 dhoyt-sc-1 umount: umount: /var/log: target is busy. Jul 8 10:52:55 dhoyt-sc-1 umount: (In some cases useful info about processes that use Jul 8 10:52:55 dhoyt-sc-1 umount: the device is found by lsof(8) or fuser(1)) Jul 8 10:52:55 dhoyt-sc-1 systemd: var-log.mount mount process exited, code=exited status=32 Jul 8 10:52:55 dhoyt-sc-1 systemd: Failed unmounting /var/log. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/2056. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted Configuration File System. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/1000. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/0. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Swap. Jul 8 10:52:55 dhoyt-sc-1 systemd: Deactivating swap /dev/disk/by-id/dm-uuid-LVM-l3kAh9n8PsfRnpOF6ScpE5OmQoS4rb5kItW7HbyeF2i9VvGaaNiMsVWEuR2NgHsu... Jul 8 10:52:55 dhoyt-sc-1 kernel: XFS (vda1): Unmounting Filesystem Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /home. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /boot. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /var/lib/pgsql. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /var... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /tmp. Jul 8 10:52:55 dhoyt-sc-1 umount: umount: /var: target is busy. Jul 8 10:52:55 dhoyt-sc-1 umount: (In some cases useful info about processes that use Jul 8 10:52:55 dhoyt-sc-1 umount: the device is found by lsof(8) or fuser(1)) Jul 8 10:52:55 dhoyt-sc-1 systemd: var-log.mount mount process exited, code=exited status=32 Jul 8 10:52:55 dhoyt-sc-1 systemd: Failed unmounting /var/log. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/2056. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted Configuration File System. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/1000. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /run/user/0. Jul 8 10:52:55 dhoyt-sc-1 systemd: Stopped target Swap. Jul 8 10:52:55 dhoyt-sc-1 systemd: Deactivating swap /dev/disk/by-id/dm-uuid-LVM-l3kAh9n8PsfRnpOF6ScpE5OmQoS4rb5kItW7HbyeF2i9VvGaaNiMsVWEuR2NgHsu... Jul 8 10:52:55 dhoyt-sc-1 kernel: XFS (vda1): Unmounting Filesystem Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /home. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /boot. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /var/lib/pgsql. Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounting /var... Jul 8 10:52:55 dhoyt-sc-1 systemd: Unmounted /tmp. Jul 8 10:52:55 dhoyt-sc-1 umount: umount: /var: target is busy. Jul 8 10:52:55 dhoyt-sc-1 umount: (In some cases useful info about processes that use Jul 8 10:52:55 dhoyt-sc-1 umount: the device is found by lsof(8) or fuser(1)) Jul 8 10:52:55 dhoyt-sc-1 systemd: var.mount mount process exited, code=exited status=32 Jul 8 10:52:55 dhoyt-sc-1 systemd: Failed unmounting /var. Jul 8 10:52:55 dhoyt-sc-1 systemd: Deactivated swap /dev/vg00/lv.swap. Jul 8 10:52:55 dhoyt-sc-1 systemd: Deactivated swap /dev/dm-1. Jul 8 10:52:55 dhoyt-sc-1 systemd: Deactivated swap /dev/mapper/vg00-lv.swap. Jul 8 10:52:56 dhoyt-sc-1 systemd: Unmounted /opt/raid. Jul 8 10:52:56 dhoyt-sc-1 systemd: Unmounting /opt... Jul 8 10:52:56 dhoyt-sc-1 systemd: Unmounted /opt. Jul 8 10:52:56 dhoyt-sc-1 systemd: Reached target Unmount All Filesystems. Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopped target Local File Systems (Pre). Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopped Remount Root and Kernel File Systems. Jul 8 10:52:56 dhoyt-sc-1 kernel: type=1131 audit(1594219976.632:14858): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopped Create Static Device Nodes in /dev. Jul 8 10:52:56 dhoyt-sc-1 kernel: type=1131 audit(1594219976.633:14859): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:56 dhoyt-sc-1 lvm: 8 logical volume(s) in volume group "vg00" unmonitored Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling. Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopping LVM2 metadata daemon... Jul 8 10:52:56 dhoyt-sc-1 kernel: type=1131 audit(1594219976.666:14860): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-monitor comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:56 dhoyt-sc-1 systemd: Stopped LVM2 metadata daemon. Jul 8 10:52:56 dhoyt-sc-1 systemd: Reached target Shutdown. Jul 8 10:52:56 dhoyt-sc-1 kernel: type=1131 audit(1594219976.668:14861): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-lvmetad comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 8 10:52:56 dhoyt-sc-1 systemd: Reached target Final Step. Jul 8 10:52:56 dhoyt-sc-1 systemd: Starting Reboot... Jul 8 10:52:56 dhoyt-sc-1 systemd: Closed LVM2 metadata daemon socket. Jul 8 10:52:56 dhoyt-sc-1 systemd: Shutting down. Jul 8 10:52:56 dhoyt-sc-1 systemd-shutdown[1]: Syncing filesystems and block devices. Jul 8 10:52:56 dhoyt-sc-1 systemd-shutdown[1]: Sending SIGTERM to remaining processes... Jul 8 10:52:56 dhoyt-sc-1 journal: Journal stopped Meanwhile, on dhoyt-sc-2, the takeover_request is REJECTED: Jul 8 10:52:47 dhoyt-sc-2 osaffmd[7883]: NO FM down on: 2010f Jul 8 10:52:47 dhoyt-sc-2 osafrded[7867]: NO Peer down on node 0x2010f Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: NO Implementer disconnected 6 <0, 2010f> (@safSmf_applier1) Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: NO Implementer disconnected 5 <0, 2010f> (safSmfService) Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: NO Implementer disconnected 1 <0, 2010f> (safLogService) Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: NO Implementer disconnected 2 <0, 2010f> (@safLogService_appl) Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: NO Implementer disconnected 3 <0, 2010f> (safClmService) Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: NO MDS event from svc_id 24 (change:1, dest:13) Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: NO MDS event from svc_id 24 (change:6, dest:13) Jul 8 10:52:47 dhoyt-sc-2 osaffmd[7883]: NO IMMD down on: 2010f Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: WA IMMD lost contact with peer IMMD (NCSMDS_RED_DOWN) Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: WA DISCARD DUPLICATE FEVS message:2781 Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: WA Error code 2 returned for message type 82 - ignoring Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: WA DISCARD DUPLICATE FEVS message:2782 Jul 8 10:52:47 dhoyt-sc-2 osafimmnd[7918]: WA Error code 2 returned for message type 82 - ignoring Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: NO MDS event from svc_id 25 (change:4, dest:564113889561677) Jul 8 10:52:47 dhoyt-sc-2 osaffmd[7883]: NO IMMND down on: 2010f Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: WA IMMND DOWN on active controller 2010f detected at standby immd!! 2020f. Possible failover Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: NO Skipping re-send of fevs message 2781 since it has recently been resent. Jul 8 10:52:47 dhoyt-sc-2 osafimmd[7900]: NO Skipping re-send of fevs message 2782 since it has recently been resent. Jul 8 10:52:52 dhoyt-sc-2 osaffmd[7883]: NO AMFND down on: 2010f Jul 8 10:52:52 dhoyt-sc-2 osafamfd[7987]: NO [D] (Consensus::IsEnabled): ENTERED Jul 8 10:52:52 dhoyt-sc-2 osafamfd[7987]: NO Node 'SC-1' is down. Start failover delay timer Jul 8 10:52:52 dhoyt-sc-2 osafamfd[7987]: NO Start timer for '2010f' Jul 8 10:52:52 dhoyt-sc-2 osafamfd[7987]: NO [D] (Consensus::IsEnabled): ENTERED Jul 8 10:52:52 dhoyt-sc-2 osaffmd[7883]: NO AVD down on: 2010f Jul 8 10:52:52 dhoyt-sc-2 osaffmd[7883]: NO Core services went down on node_id: 2010f Jul 8 10:52:54 dhoyt-sc-2 osafdtmd[7824]: NO Lost contact with 'SC-1' Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: NO Node Down event for node id 2010f: Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: NO Current role: STANDBY Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: Rebooting OpenSAF NodeId = 131343 EE Name = , Reason: Received Node Down for peer controller, OwnNodeId = 131855, SupervisionTime = 60 Jul 8 10:52:54 dhoyt-sc-2 opensaf_reboot: Rebooting remote node in the absence of PLM is outside the scope of OpenSAF Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: NO Controller Failover: Setting role to ACTIVE Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::IsEnabled): ENTERED Jul 8 10:52:54 dhoyt-sc-2 osaffmd[7883]: NO Starting consensus service supervision: 60 s Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::PromoteThisNode): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::CheckForExistingTakeoverRequest): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::CheckForExistingTakeoverRequest): {1} Calling 'ReadTakeoverRequest' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 1 Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read '' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::PromoteThisNode): {1} Calling 'Lock' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Lock): ENTERED Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Lock): Executing command: '/opt/opensaf/osaf-etcd3.plugin lock "SC-2" 0' Jul 8 10:52:58 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin lock "SC-2" 0', returning 1 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO Locked failed: SC-1 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::CurrentActive): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::CurrentActive): {1} Calling 'LockOwner' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::LockOwner): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::LockOwner): Executing command: '/opt/opensaf/osaf-etcd3.plugin lock_owner' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin lock_owner', returning 0 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO Current active controller is SC-1 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::CreateTakeoverRequest): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D-1] (Consensus::CreateTakeoverRequest): {1} Calling 'Create' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Create): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Create): Executing command: '/opt/opensaf/osaf-etcd3.plugin create "takeover_request" "SC-1 SC-2 1 NEW" 60' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin create "takeover_request" "SC-1 SC-2 1 NEW" 60', returning 0 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D-8] (Consensus::CreateTakeoverRequest): max_takeover_retry: 30 Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [0] Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:52:59 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:00 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:00 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:00 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [1] Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:01 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [2] Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:02 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [3] Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:03 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [4] Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:04 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [5] Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [5] Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 NEW' Jul 8 10:53:05 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D-9] (Consensus::CreateTakeoverRequest): {3} Calling 'ReadTakeoverRequest' [6] Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): ENTERED Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ReadTakeoverRequest): Calling 'Get' Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): ENTERED Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Executing command: '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"' Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): ENTERED Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Execute): Executed '/opt/opensaf/osaf-etcd3.plugin get "takeover_request"', returning 0 Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (KeyValue::Get): Read 'SC-1 SC-2 1 REJECTED' Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO [D] (Consensus::ParseTakeoverRequest): ENTERED Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: NO Takeover request rejected Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: WA Takeover request failed (14) Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: ER A controller is already active. We were separated from the cluster? Jul 8 10:53:06 dhoyt-sc-2 osaffmd[7883]: Quick local node rebooting, Reason: A controller is already active. We were separated from the cluster? Jul 8 10:53:06 dhoyt-sc-2 opensaf_reboot: Do quick local node reboot From: Gary Lee <gary....@dektech.com.au> Sent: Wednesday, July 8, 2020 5:38 AM To: Hoyt, David <dh...@rbbn.com>; opensaf-users@lists.sourceforge.net Subject: Re: consensus behaviour ________________________________ NOTICE: This email was received from an EXTERNAL sender ________________________________ Hi David Can you attach the logs from SC-1? ________________________________ ----------------------------------------------------------------------------------------------------------------------- Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments. ----------------------------------------------------------------------------------------------------------------------- _______________________________________________ Opensaf-users mailing list Opensaf-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-users
