Thanks. Please visit link below to get all the logs. Ted
https://www.dropbox.com/s/yqq1idumqyvbrab/log.tar.gz?dl=0 From: Neelakanta Reddy [mailto:[email protected]] Sent: Monday, May 04, 2015 6:07 PM To: Yao Cheng LIANG; [email protected] Subject: Re: [users] Fw: log Hi, Please,share the syslog and osafimmnd traces available at /var/log/opensaf on all the nodes. To enable immnd traces uncomment the below line in all the nodes of the cluster at /etc/opensaf/immnd.conf: # Uncomment the next line to enable trace args="--tracemask=0xffffffff" /Neel. On Monday 04 May 2015 03:20 PM, Yao Cheng LIANG wrote: Nope. All nodes upgraded to 4.6. /Ted Sent from Samsung Mobile -------- Original message -------- From: Neelakanta Reddy Date:2015/05/04 5:03 PM (GMT+08:00) To: [email protected]<mailto:[email protected]> Subject: Re: [users] Fw: log Hi, Two controllers are upgraded to 4.6 and payloads are still have 4.2 . In general upgrade the nodes must be rolling upgrade. If some nodes are upgraded manually, and some nodes are still in older releases then IMM flags need to be toggled accordingly. comments below. /Neel. On Monday 04 May 2015 04:55 AM, Yao Cheng LIANG wrote: > Dear all, > > I recently upgraded my opensaf from 4.2.2 to u4.6.0 for checkpoint service > performance improvement. I have successfully started on both controllers but > I can not do same thing on the payload. From the log below from playload > node, seems Opensaf started successfully but later on shut itself down for > some errors. I am using imm.xml for my 4.2.2 version. May anyone help? > > Thanks. > > Ted > > Jan 1 08:08:11 (none) user.notice opensafd: Starting OpenSAF Services (Using > TCP) > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: Started > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: Started > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO Established contact > with 'WR20-64_32' > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO Established contact > with 'WR20-64_25' > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER STATE: > IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER STATE: > IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO SERVER STATE: > IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_ISOLATED > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_W_AVAILABLE > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO SERVER STATE: > IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE STATE-> > IMM_NODE_FULLY_AVAILABLE 2578 > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO RepositoryInitModeT > is SA_IMM_INIT_FROM_FILE > Jan 1 08:08:12 (none) local0.warn osafimmnd[931]: WA IMM Access Control mode > is DISABLED! > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO Epoch set to 18 in > ImmModel > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO SERVER STATE: > IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: Started > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: NO > safNode=PL-4,safCluster=myClmCluster Joined cluster, nodeid=a040f > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: Started > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State UNINSTANTIATED => > INSTANTIATING > Jan 1 08:08:13 (none) local0.notice osafsmfnd[964]: Started > Jan 1 08:08:13 (none) local0.notice osafmsgnd[974]: Started > Jan 1 08:08:13 (none) local0.notice osafimmnd[931]: NO Implementer > connected: 38 (MsgQueueService656399) <51, a040f> IMMND asserted and restarted again, this is because of some information added in 4.6, which may not be compatible with older releases. since the cluster is mixed version. Please go through osaf/services/saf/immsv/README( particularly Notes on upgrading from OpenSAF 4.[1,2,3,4,5] to OpenSAF (4.6)). once the cluster are upgraded, the flags mentioned needs to be toggled on. > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: Started > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO Fevs count adjusted > to 5871 preLoadPid: 0 > Jan 1 08:08:13 (none) local0.notice osaflcknd[997]: Started > Jan 1 08:08:13 (none) local0.notice osafckptnd[1007]: Started > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO SERVER STATE: > IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > Jan 1 08:08:13 (none) local0.notice osafamfwd[1018]: Started > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING => > INSTANTIATED > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO Assigning > 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO Assigned > 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:13 (none) user.notice opensafd: OpenSAF(4.6.0 - > 6467:3561f9d06464) services successfully started > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 5 > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 10 > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA Resending introduce-me > - problems with MDS ? 10 > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > Jan 1 08:08:23 (none) local0.err osafamfnd[953]: saImmOmInitialize FAILED, > rc = 6 > Jan 1 08:08:23 (none) local0.alert osafimmnd[986]: AL AMF Node Director is > down, terminate this process > Jan 1 08:08:23 (none) local0.alert osaflcknd[997]: AL AMF Node Director is > down, terminate this process > Jan 1 08:08:23 (none) local0.crit osafamfwd[1018]: Rebooting OpenSAF NodeId > = 0 EE Name = No EE Mapped, Reason: AMF unexpectedly crashed, OwnNodeId = > 656399, SupervisionTime = 60 > Jan 1 08:08:23 (none) local0.notice osaflcknd[997]: exiting for shutdown > Jan 1 08:08:23 (none) local0.alert osafsmfnd[964]: AL AMF Node Director is > down, terminate this process > Jan 1 08:08:23 (none) local0.alert osafckptnd[1007]: AL AMF Node Director is > down, terminate this process > Jan 1 08:08:23 (none) local0.notice osafsmfnd[964]: exiting for shutdown > Jan 1 08:08:23 (none) local0.alert osafmsgnd[974]: AL AMF Node Director is > down, terminate this process > Jan 1 08:08:23 (none) local0.notice osafckptnd[1007]: exiting for shutdown > Jan 1 08:08:23 (none) local0.notice osafmsgnd[974]: exiting for shutdown > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: exiting for shutdown > Jan 1 08:08:23 (none) local0.notice osafimmnd[931]: NO Implementer locally > disconnected. Marking it as doomed 38 <51, a040f> (MsgQueueService656399) > Jan 1 08:08:23 (none) local0.err osafimmnd[931]: ER > immnd_evt_proc_discard_node for *this* node 656399 => Cluster partitioned > ("split brain") - exiting > Jan 1 08:08:23 (none) user.notice opensaf_reboot: Rebooting local node; > timeout=60 > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > _______________________________________________ > Opensaf-users mailing list > [email protected]<mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/opensaf-users ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Opensaf-users mailing list [email protected]<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-users ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Opensaf-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-users
