Hi, share , the syslog of active controller.
/Neel. On Monday 04 May 2015 07:23 PM, Yao Cheng LIANG wrote: > This is strange. We tried a few time and it I impossible that there is > link loss or link toggling happened at the same time we start opensafd > on the payload. This payload is running MIPS with Busybox. If I run > payload on another x86 there is no such issue. That payload connects > to controller with the same switch as the mips one. > > Thanks. > > Ted > > > > Sent from Windows Mail > > *From:* 'Neelakanta Reddy' <mailto:reddy.neelaka...@oracle.com> > *Sent:* Monday, May 4, 2015 9:31 PM > *To:* Yao Cheng LIANG <mailto:ycli...@astri.org>, > opensaf-users@lists.sourceforge.net > <mailto:opensaf-users@lists.sourceforge.net> > > Hi, > > comments in-line. > > /Neel. > On Monday 04 May 2015 05:33 PM, Yao Cheng LIANG wrote: > > The error was cause by line below as msq service register with > immnd. Please see this line in immnd log > > Jan 1 8:06:49.884694 osafimmnd [936:immnd_evt.c:0729] WA > immnd_evt_proc_imm_init: PID 0 (1012) for 2040f000003f4, MDS problem? > > > This happens, when the own node receives node down. when there is link > loss or link toggling happened, SC-1 or active controller detected > link loss and sends immnd down message, by the time time message is > sent the link has established again and the PL-3 received the message. > > Verify, the link loss messages in active controller syslog messages. > > *From:*Neelakanta Reddy [mailto:reddy.neelaka...@oracle.com] > *Sent:* Monday, May 04, 2015 6:07 PM > *To:* Yao Cheng LIANG; opensaf-users@lists.sourceforge.net > *Subject:* Re: [users] Fw: log > > Hi, > > Please,share the syslog and osafimmnd traces available at > /var/log/opensaf on all the nodes. > > To enable immnd traces uncomment the below line in all the nodes > of the cluster at /etc/opensaf/immnd.conf: > > # Uncomment the next line to enable trace > args="--tracemask=0xffffffff" > > /Neel. > > On Monday 04 May 2015 03:20 PM, Yao Cheng LIANG wrote: > > Nope. All nodes upgraded to 4.6. /Ted > > Sent from Samsung Mobile > > > > -------- Original message -------- > From: Neelakanta Reddy > Date:2015/05/04 5:03 PM (GMT+08:00) > To: opensaf-users@lists.sourceforge.net > <mailto:opensaf-users@lists.sourceforge.net> > Subject: Re: [users] Fw: log > > Hi, > > Two controllers are upgraded to 4.6 and payloads are still > have 4.2 . > In general upgrade the nodes must be rolling upgrade. > If some nodes are upgraded manually, and some nodes are still > in older > releases then IMM flags need to be toggled accordingly. > > comments below. > > /Neel. > > On Monday 04 May 2015 04:55 AM, Yao Cheng LIANG wrote: > > Dear all, > > > > I recently upgraded my opensaf from 4.2.2 to u4.6.0 for > checkpoint service performance improvement. I have > successfully started on both controllers but I can not do same > thing on the payload. From the log below from playload node, > seems Opensaf started successfully but later on shut itself > down for some errors. I am using imm.xml for my 4.2.2 version. > May anyone help? > > > > Thanks. > > > > Ted > > > > Jan 1 08:08:11 (none) user.notice opensafd: Starting > OpenSAF Services (Using TCP) > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: Started > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: Started > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO > Established contact with 'WR20-64_32' > > Jan 1 08:08:11 (none) local0.notice osafdtmd[914]: NO > Established contact with 'WR20-64_25' > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO > SERVER STATE: IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO > SERVER STATE: IMM_SERVER_CLUSTER_WAITING --> > IMM_SERVER_LOADING_PENDING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO > SERVER STATE: IMM_SERVER_LOADING_PENDING --> > IMM_SERVER_SYNC_PENDING > > Jan 1 08:08:11 (none) local0.notice osafimmnd[931]: NO NODE > STATE-> IMM_NODE_ISOLATED > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE > STATE-> IMM_NODE_W_AVAILABLE > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO > SERVER STATE: IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO NODE > STATE-> IMM_NODE_FULLY_AVAILABLE 2578 > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO > RepositoryInitModeT is SA_IMM_INIT_FROM_FILE > > Jan 1 08:08:12 (none) local0.warn osafimmnd[931]: WA IMM > Access Control mode is DISABLED! > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO > Epoch set to 18 in ImmModel > > Jan 1 08:08:12 (none) local0.notice osafimmnd[931]: NO > SERVER STATE: IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY > > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: Started > > Jan 1 08:08:12 (none) local0.notice osafclmna[943]: NO > safNode=PL-4,safCluster=myClmCluster Joined cluster, nodeid=a040f > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: Started > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State > UNINSTANTIATED => INSTANTIATING > > Jan 1 08:08:13 (none) local0.notice osafsmfnd[964]: Started > > Jan 1 08:08:13 (none) local0.notice osafmsgnd[974]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[931]: NO > Implementer connected: 38 (MsgQueueService656399) <51, a040f> > IMMND asserted and restarted again, this is because of some > information > added in 4.6, which may not be compatible with older releases. > since the cluster is mixed version. > > Please go through osaf/services/saf/immsv/README( particularly > Notes on > upgrading from OpenSAF 4.[1,2,3,4,5] to OpenSAF (4.6)). > > once the cluster are upgraded, the flags mentioned needs to be > toggled on. > > > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO Fevs > count adjusted to 5871 preLoadPid: 0 > > Jan 1 08:08:13 (none) local0.notice osaflcknd[997]: Started > > Jan 1 08:08:13 (none) local0.notice osafckptnd[1007]: Started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > SERVER STATE: IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > > Jan 1 08:08:13 (none) local0.notice osafamfwd[1018]: Started > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' Presence State > INSTANTIATING => INSTANTIATED > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > Assigning 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > > Jan 1 08:08:13 (none) local0.notice osafamfnd[953]: NO > Assigned 'safSi=NoRed10,safApp=OpenSAF' ACTIVE to > 'safSu=PL-4,safSg=NoRed,safApp=OpenSAF' > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:13 (none) user.notice opensafd: OpenSAF(4.6.0 - > 6467:3561f9d06464) services successfully started > > Jan 1 08:08:13 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:14 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:15 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:16 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:17 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:18 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 5 > > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:19 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:20 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:21 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:22 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 10 > > Jan 1 08:08:23 (none) local0.warn osafimmnd[986]: WA > Resending introduce-me - problems with MDS ? 10 > > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: NO > mds_register_callback: dest a040f000003b9 already exist > > Jan 1 08:08:23 (none) local0.err osafamfnd[953]: > saImmOmInitialize FAILED, rc = 6 > > Jan 1 08:08:23 (none) local0.alert osafimmnd[986]: AL AMF > Node Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.alert osaflcknd[997]: AL AMF > Node Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.crit osafamfwd[1018]: > Rebooting OpenSAF NodeId = 0 EE Name = No EE Mapped, Reason: > AMF unexpectedly crashed, OwnNodeId = 656399, SupervisionTime = 60 > > Jan 1 08:08:23 (none) local0.notice osaflcknd[997]: exiting > for shutdown > > Jan 1 08:08:23 (none) local0.alert osafsmfnd[964]: AL AMF > Node Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.alert osafckptnd[1007]: AL AMF > Node Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.notice osafsmfnd[964]: exiting > for shutdown > > Jan 1 08:08:23 (none) local0.alert osafmsgnd[974]: AL AMF > Node Director is down, terminate this process > > Jan 1 08:08:23 (none) local0.notice osafckptnd[1007]: > exiting for shutdown > > Jan 1 08:08:23 (none) local0.notice osafmsgnd[974]: exiting > for shutdown > > Jan 1 08:08:23 (none) local0.notice osafimmnd[986]: exiting > for shutdown > > Jan 1 08:08:23 (none) local0.notice osafimmnd[931]: NO > Implementer locally disconnected. Marking it as doomed 38 <51, > a040f> (MsgQueueService656399) > > Jan 1 08:08:23 (none) local0.err osafimmnd[931]: ER > immnd_evt_proc_discard_node for *this* node 656399 => Cluster > partitioned ("split brain") - exiting > > Jan 1 08:08:23 (none) user.notice opensaf_reboot: Rebooting > local node; timeout=60 > > > > > > ------------------------------------------------------------------------------ > > One dashboard for servers and applications across > Physical-Virtual-Cloud > > Widest out-of-the-box monitoring support with 50+ applications > > Performance metrics, stats and reports that give you > Actionable Insights > > Deep dive visibility with transaction tracing using APM Insight. > > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > <http://ad.doubleclick.net/ddm/clk/290420510%3b117567292%3by> > > _______________________________________________ > > Opensaf-users mailing list > > Opensaf-users@lists.sourceforge.net > <mailto:Opensaf-users@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/opensaf-users > > > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across > Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you > Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > <http://ad.doubleclick.net/ddm/clk/290420510%3b117567292%3by> > _______________________________________________ > Opensaf-users mailing list > Opensaf-users@lists.sourceforge.net > <mailto:Opensaf-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/opensaf-users > > ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Opensaf-users mailing list Opensaf-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-users
