Hi, Please,share the syslog and osafimmnd traces available at /var/log/opensaf on all the nodes.
To enable immnd traces uncomment the below line in all the nodes of the cluster at /etc/opensaf/immnd.conf: # Uncomment the next line to enable trace args="--tracemask=0xffffffff" /Neel. On Monday 04 May 2015 03:20 PM, Yao Cheng LIANG wrote: > Nope. All nodes upgraded to 4.6. /Ted > > > Sent from Samsung Mobile > > > -------- Original message -------- > From: Neelakanta Reddy > Date:2015/05/04 5:03 PM (GMT+08:00) > To: [email protected] > Subject: Re: [users] Fw: log > > Hi, > > Two controllers are upgraded to 4.6 and payloads are still have 4.2 . > In general upgrade the nodes must be rolling upgrade. > If some nodes are upgraded manually, and some nodes are still in older > releases then IMM flags need to be toggled accordingly. > > comments below. > > /Neel. > > On Monday 04 May 2015 04:55 AM, Yao Cheng LIANG wrote: > > Dear all, > > > > I recently upgraded my opensaf from 4.2.2 to u4.6.0 for checkpoint > service performance improvement. I have successfully started on both > controllers but I can not do same thing on the payload. From the log > below from playload node, seems Opensaf started successfully but later > on shut itself down for some errors. I am using imm.xml for my 4.2.2 > version. May anyone help? > > > > Thanks. > > > > Ted > > > > Jan 1 08:08:11 (none) user.notice opensafd: Starting OpenSAF > Services (Using TCP) > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: Started > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: Started > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO Established > contact with 'WR20-64_32' > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO Established > contact with 'WR20-64_25' > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER > STATE: IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER > STATE: IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER > STATE: IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_ISOLATED > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_W_AVAILABLE > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO SERVER > STATE: IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_FULLY_AVAILABLE 2578 > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO > RepositoryInitModeT is SA_IMM_INIT_FROM_FILE > > Jan 1 08:08:12 (none) local0.warn osafimmnd[931]: WA IMM Access > Control mode is DISABLED! > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO Epoch set to > 18 in ImmModel > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO SERVER > STATE: IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY > > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: Started > > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: NO > safNode=PL-4,safCluster=myClmCluster Joined cluster, nodeid=a040f > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: Started > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State UNINSTANTIATED > => INSTANTIATING > > Jan 1 08:08:13 (none) local0.notice osafsmfnd[964]: Started > > Jan 1 08:08:13 (none) local0.notice osafmsgnd[974]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[931]: NO Implementer > connected: 38 (MsgQueueService656399) <51, a040f> > IMMND asserted and restarted again, this is because of some information > added in 4.6, which may not be compatible with older releases. > since the cluster is mixed version. > > Please go through osaf/services/saf/immsv/README( particularly Notes on > upgrading from OpenSAF 4.[1,2,3,4,5] to OpenSAF (4.6)). > > once the cluster are upgraded, the flags mentioned needs to be toggled on. > > > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO Fevs count > adjusted to 5871 preLoadPid: 0 > > Jan 1 08:08:13 (none) local0.notice osaflcknd[997]: Started > > Jan 1 08:08:13 (none) local0.notice osafckptnd[1007]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO SERVER > STATE: IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > > Jan 1 08:08:13 (none) local0.notice osafamfwd[1018]: Started > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING > => INSTANTIATED > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO Assigning > 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO Assigned > 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:13 (none) user.notice opensafd: OpenSAF(4.6.0 - > 6467:3561f9d06464) services successfully started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 5 > > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 10 > > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA Resending > introduce-me - problems with MDS ? 10 > > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:23 (none) local0.err osafamfnd[953]: saImmOmInitialize > FAILED, rc = 6 > > Jan 1 08:08:23 (none) local0.alert osafimmnd[986]: AL AMF Node > Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.alert osaflcknd[997]: AL AMF Node > Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.crit osafamfwd[1018]: Rebooting > OpenSAF NodeId = 0 EE Name = No EE Mapped, Reason: AMF unexpectedly > crashed, OwnNodeId = 656399, SupervisionTime = 60 > > Jan 1 08:08:23 (none) local0.notice osaflcknd[997]: exiting for > shutdown > > Jan 1 08:08:23 (none) local0.alert osafsmfnd[964]: AL AMF Node > Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.alert osafckptnd[1007]: AL AMF Node > Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.notice osafsmfnd[964]: exiting for > shutdown > > Jan 1 08:08:23 (none) local0.alert osafmsgnd[974]: AL AMF Node > Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.notice osafckptnd[1007]: exiting for > shutdown > > Jan 1 08:08:23 (none) local0.notice osafmsgnd[974]: exiting for > shutdown > > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: exiting for > shutdown > > Jan 1 08:08:23 (none) local0.notice osafimmnd[931]: NO Implementer > locally disconnected. Marking it as doomed 38 <51, a040f> > (MsgQueueService656399) > > Jan 1 08:08:23 (none) local0.err osafimmnd[931]: ER > immnd_evt_proc_discard_node for *this* node 656399 => Cluster > partitioned ("split brain") - exiting > > Jan 1 08:08:23 (none) user.notice opensaf_reboot: Rebooting local > node; timeout=60 > > > > > ------------------------------------------------------------------------------ > > One dashboard for servers and applications across Physical-Virtual-Cloud > > Widest out-of-the-box monitoring support with 50+ applications > > Performance metrics, stats and reports that give you Actionable Insights > > Deep dive visibility with transaction tracing using APM Insight. > > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > > _______________________________________________ > > Opensaf-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/opensaf-users > > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > _______________________________________________ > Opensaf-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/opensaf-users ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Opensaf-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-users
