We are seeing osaf stall on startup on a payload card. The ‘ps’ trace below
shows the processes on the payload. It never gets any further than this until
we kill the osafimmnd process running on the controller. As soon as we do that
the payload comes up.
oot 1319 0.0 0.0 0 0 ? ZNs 20:34 0:00 [osaf-immnd]
<defunct>
opensaf 1326 0.3 0.3 173916 11876 ? SNsl 20:34 0:01
/usr/lib64/opensaf/osafimmnd osafimmnd
root 1340 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-noded]
<defunct>
root 1349 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-amfnd]
<defunct>
root 1353 0.0 0.0 159956 1720 ? Ssl 20:34 0:00
/usr/lib64/opensaf/osafamfnd osafamfnd
When we look that the state of the OpenSAF SU we see the following..
scm1:~$ immlist $(amf-find su | grep OpenSAF | grep SCM1 | grep 2N)
Name Type Value(s)
========================================================================
safSu SA_STRING_T safSu=SCM1
saAmfSUType SA_NAME_T
safVersion=4.0.0,safSuType=OpenSafSuTypeServer (46)
saAmfSURestartCount SA_UINT32_T <Empty>
saAmfSUReadinessState SA_UINT32_T <Empty>
saAmfSURank SA_UINT32_T 0 (0x0)
saAmfSUPresenceState SA_UINT32_T <Empty>
saAmfSUPreInstantiable SA_UINT32_T <Empty>
saAmfSUOperState SA_UINT32_T <Empty>
saAmfSUNumCurrStandbySIs SA_UINT32_T <Empty>
saAmfSUNumCurrActiveSIs SA_UINT32_T <Empty>
saAmfSUMaintenanceCampaign SA_NAME_T <Empty>
saAmfSUHostedByNode SA_NAME_T <Empty>
saAmfSUHostNodeOrNodeGroup SA_NAME_T
safAmfNode=SCM1,safAmfCluster=myAmfCluster (42)
saAmfSUFailover SA_UINT32_T 1 (0x1)
saAmfSUAssignedSIs SA_NAME_T <Empty>
saAmfSUAdminState SA_UINT32_T 1 (0x1)
SaImmAttrImplementerName SA_STRING_T safAmfService
SaImmAttrClassName SA_STRING_T SaAmfSU
SaImmAttrAdminOwnerName SA_STRING_T IMMLOADER
This is reproducible all we need to do is remove two payload cards and
re-insert them both.
Does this problem ring any bells?
thanks
—
tony
P.S. full logs are attached.
CMM02A
root 1319 0.0 0.0 0 0 ? ZNs 20:34 0:00 [osaf-immnd]
<defunct>
opensaf 1326 0.3 0.3 173916 11876 ? SNsl 20:34 0:01
/usr/lib64/opensaf/osafimmnd osafimmnd
root 1340 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-noded]
<defunct>
root 1349 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-amfnd]
<defunct>
root 1353 0.0 0.0 159956 1720 ? Ssl 20:34 0:00
/usr/lib64/opensaf/osafamfnd osafamfnd
scm1:~$ immlist $(amf-find su | grep OpenSAF | grep SCM1 | grep 2N)
Name Type Value(s)
========================================================================
safSu SA_STRING_T safSu=SCM1
saAmfSUType SA_NAME_T
safVersion=4.0.0,safSuType=OpenSafSuTypeServer (46)
saAmfSURestartCount SA_UINT32_T <Empty>
saAmfSUReadinessState SA_UINT32_T <Empty>
saAmfSURank SA_UINT32_T 0 (0x0)
saAmfSUPresenceState SA_UINT32_T <Empty>
saAmfSUPreInstantiable SA_UINT32_T <Empty>
saAmfSUOperState SA_UINT32_T <Empty>
saAmfSUNumCurrStandbySIs SA_UINT32_T <Empty>
saAmfSUNumCurrActiveSIs SA_UINT32_T <Empty>
saAmfSUMaintenanceCampaign SA_NAME_T <Empty>
saAmfSUHostedByNode SA_NAME_T <Empty>
saAmfSUHostNodeOrNodeGroup SA_NAME_T
safAmfNode=SCM1,safAmfCluster=myAmfCluster (42)
saAmfSUFailover SA_UINT32_T 1 (0x1)
saAmfSUAssignedSIs SA_NAME_T <Empty>
saAmfSUAdminState SA_UINT32_T 1 (0x1)
SaImmAttrImplementerName SA_STRING_T safAmfService
SaImmAttrClassName SA_STRING_T SaAmfSU
SaImmAttrAdminOwnerName SA_STRING_T IMMLOADER
scm1:~$ amf-state
Service Instances UNLOCKED and UNASSIGNED:
/usr/bin/amf-state: line 37: [: <Empty>: integer expression expected
/usr/bin/amf-state: line 37: [: <Empty>: integer expression expected
/usr/bin/amf-state: line 37: [: <Empty>: integer expression expected
CMM02A
2014-02-25T20:12:25.837820+00:00 cmm02a osafimmnd[1322]: Started
2014-02-25T20:12:25.942329+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
2014-02-25T20:12:26.043212+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING
2014-02-25T20:12:26.143943+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
2014-02-25T20:12:26.144367+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_ISOLATED
2014-02-25T20:12:26.214269+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_W_AVAILABLE
2014-02-25T20:12:26.243582+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
2014-02-25T20:12:30.011710+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 2144
2014-02-25T20:12:30.011760+00:00 cmm02a osafimmnd[1322]: NO RepositoryInitModeT
is SA_IMM_INIT_FROM_FILE
2014-02-25T20:12:30.011781+00:00 cmm02a osafimmnd[1322]: NO Epoch set to 7 in
ImmModel
2014-02-25T20:12:30.055452+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY
2014-02-25T20:12:30.086278+00:00 cmm02a osafclmna[1331]: Started
2014-02-25T20:12:30.189542+00:00 cmm02a osafclmna[1331]: NO
safNode=cmm02a,safCluster=myClmCluster Joined cluster, nodeid=11f0f
2014-02-25T20:12:30.225293+00:00 cmm02a osafamfnd[1340]: Started
2014-02-25T20:12:52.957541+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T20:12:56.911351+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 14277
2014-02-25T20:12:56.911846+00:00 cmm02a osafimmnd[1322]: NO Epoch set to 8 in
ImmModel
2014-02-25T20:24:35.852781+00:00 cmm02a osafimmnd[1322]: NO Implementer
connected: 18 (MsgQueueService73743) <0, 1200f>
SCM1
2014-02-25T19:54:31.863985+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1150f old epoch: 3 new epoch:4
2014-02-25T19:54:31.883677+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1200f old epoch: 0 new epoch:4
2014-02-25T19:54:31.951145+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_SYNC_SERVER --> IMM SERVER READY
2014-02-25T19:54:32.099602+00:00 scm1 osafamfd[2097]: NO Node 'CMM02B' joined
the cluster
2014-02-25T19:54:32.311990+00:00 scm1 osafimmnd[2046]: NO Implementer
connected: 11 (MsgQueueService73743) <0, 1200f>
2014-02-25T19:59:17.123913+00:00 scm1 osafimmd[2036]: NO Node 11f0f request
sync sync-pid:2800 epoch:0
2014-02-25T19:59:17.975676+00:00 scm1 osafimmnd[2046]: NO Announce sync, epoch:5
2014-02-25T19:59:17.976097+00:00 scm1 osafimmd[2036]: NO Successfully announced
sync. New ruling epoch:5
2014-02-25T19:59:17.976339+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
2014-02-25T19:59:17.976374+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T19:59:18.080391+00:00 scm1 osafimmloadd: NO Sync starting
2014-02-25T19:59:21.946804+00:00 scm1 osafimmloadd: IN Synced 5340 objects in
total
2014-02-25T19:59:21.947219+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 13776
2014-02-25T19:59:21.948815+00:00 scm1 osafimmnd[2046]: NO Epoch set to 5 in
ImmModel
2014-02-25T19:59:21.948955+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 10f0f old epoch: 4 new epoch:5
2014-02-25T19:59:21.949540+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1150f old epoch: 4 new epoch:5
2014-02-25T19:59:21.949580+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1200f old epoch: 4 new epoch:5
2014-02-25T19:59:21.953455+00:00 scm1 osafimmloadd: NO Sync ending normally
2014-02-25T19:59:21.970315+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 11f0f old epoch: 0 new epoch:5
2014-02-25T19:59:21.984717+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_SYNC_SERVER --> IMM SERVER READY
2014-02-25T19:59:22.203178+00:00 scm1 osafamfd[2097]: NO Node 'CMM02A' joined
the cluster
2014-02-25T19:59:22.411440+00:00 scm1 osafimmnd[2046]: NO Implementer
connected: 12 (MsgQueueService73487) <0, 11f0f>
2014-02-25T20:03:40.684692+00:00 scm1 osafimmd[2036]: NO New IMMND process is
on STANDBY Controller at 1100f
2014-02-25T20:03:40.886296+00:00 scm1 osafimmd[2036]: WA IMMND on controller
(not currently coord) requests sync
2014-02-25T20:03:40.886660+00:00 scm1 osafimmd[2036]: NO Node 1100f request
sync sync-pid:3394 epoch:0
2014-02-25T20:03:41.021807+00:00 scm1 osafimmnd[2046]: NO Announce sync, epoch:6
2014-02-25T20:03:41.021962+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
2014-02-25T20:03:41.023124+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T20:03:41.023345+00:00 scm1 osafimmd[2036]: NO Successfully announced
sync. New ruling epoch:6
2014-02-25T20:03:41.125684+00:00 scm1 osafimmloadd: NO Sync starting
2014-02-25T20:03:45.135450+00:00 scm1 osafimmloadd: IN Synced 5351 objects in
total
2014-02-25T20:03:45.136278+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 13776
2014-02-25T20:03:45.138558+00:00 scm1 osafimmloadd: NO Sync ending normally
2014-02-25T20:03:45.139063+00:00 scm1 osafimmnd[2046]: NO Epoch set to 6 in
ImmModel
2014-02-25T20:03:45.139333+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 10f0f old epoch: 5 new epoch:6
2014-02-25T20:03:45.140616+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 11f0f old epoch: 5 new epoch:6
2014-02-25T20:03:45.141446+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1200f old epoch: 5 new epoch:6
2014-02-25T20:03:45.142196+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1150f old epoch: 5 new epoch:6
2014-02-25T20:03:45.160955+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1100f old epoch: 0 new epoch:6
2014-02-25T20:03:45.225383+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_SYNC_SERVER --> IMM SERVER READY
2014-02-25T20:03:45.244547+00:00 scm1 osafimmnd[2046]: NO Implementer (applier)
connected: 13 (@safLogService) <0, 1100f>
2014-02-25T20:04:15.908243+00:00 scm1 osafimmnd[2046]: NO Implementer (applier)
connected: 14 (@safAmfService1100f) <0, 1100f>
2014-02-25T20:04:16.002102+00:00 scm1 osafamfd[2097]: NO Node 'SCM2' joined the
cluster
2014-02-25T20:04:16.718230+00:00 scm1 osafimmnd[2046]: NO Implementer
connected: 15 (MsgQueueService69647) <0, 1100f>
2014-02-25T20:09:23.183675+00:00 scm1 osafimmnd[2046]: NO Global discard node
received for nodeId:1150f pid:1668
2014-02-25T20:09:23.183733+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 10 <0, 1150f(down)> (MsgQueueService70927)
2014-02-25T20:09:23.185111+00:00 scm1 osafamfd[2097]: NO Node 'PLD0205' left
the cluster
2014-02-25T20:09:23.189843+00:00 scm1 osafimmnd[2046]: NO Implementer
connected: 16 (MsgQueueService70927) <4740, 10f0f>
2014-02-25T20:09:23.190846+00:00 scm1 osafimmnd[2046]: NO Implementer locally
disconnected. Marking it as doomed 16 <4740, 10f0f> (MsgQueueService70927)
2014-02-25T20:09:23.191862+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 16 <4740, 10f0f> (MsgQueueService70927)
2014-02-25T20:09:33.459006+00:00 scm1 osafimmnd[2046]: WA ERR_BAD_HANDLE:
Handle use is blocked by pending reply on syncronous call
2014-02-25T20:09:33.459089+00:00 scm1 osafimmnd[2046]: NO Implementer locally
disconnected. Marking it as doomed 3 <18, 10f0f> (safAmfService)
2014-02-25T20:09:33.459531+00:00 scm1 osafamfd[2097]: WA saImmOiRtObjectUpdate
of 'safComp=AMFWDOG,safSu=SCM1,safSg=NoRed,safApp=OpenSAF'
saAmfCompRestartCount failed with 9
2014-02-25T20:09:33.459778+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:09:43.465390+00:00 scm1 osafamfd[2097]: WA saImmOiRtObjectUpdate
of 'safComp=AMFWDOG,safSu=SCM1,safSg=NoRed,safApp=OpenSAF'
saAmfCompCurrProxyName failed with 5
2014-02-25T20:09:43.465920+00:00 scm1 osafimmnd[2046]: WA Failed to retrieve
search continuation, client died ?
2014-02-25T20:09:43.465961+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:09:48.884924+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:09:48.886099+00:00 scm1 osafclmd[2078]: ER clms_mds_msg_send
FAILED: 2
2014-02-25T20:09:48.886713+00:00 scm1 osafimmnd[2046]: NO Global discard node
received for nodeId:11f0f pid:2800
2014-02-25T20:09:48.886819+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 12 <0, 11f0f(down)> (MsgQueueService73487)
2014-02-25T20:09:48.886853+00:00 scm1 osafclmd[2078]: ER clms_mds_msg_send
FAILED: 2
2014-02-25T20:09:48.887730+00:00 scm1 osafimmnd[2046]: NO Global discard node
received for nodeId:1200f pid:1801
2014-02-25T20:09:48.887775+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 11 <0, 1200f(down)> (MsgQueueService73743)
2014-02-25T20:09:48.888890+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 3 <18, 10f0f> (safAmfService)
2014-02-25T20:09:48.901147+00:00 scm1 osafimmnd[2046]: NO Implementer
connected: 17 (MsgQueueService73743) <4750, 10f0f>
2014-02-25T20:09:48.901444+00:00 scm1 osafimmnd[2046]: NO Implementer locally
disconnected. Marking it as doomed 17 <4750, 10f0f> (MsgQueueService73743)
2014-02-25T20:09:48.903345+00:00 scm1 osafimmnd[2046]: NO Implementer
disconnected 17 <4750, 10f0f> (MsgQueueService73743)
2014-02-25T20:09:53.475710+00:00 scm1 osafamfd[2097]: WA saImmOiRtObjectUpdate
of 'safComp=AMFWDOG,safSu=SCM1,safSg=NoRed,safApp=OpenSAF'
saAmfCompRestartCount failed with 5
2014-02-25T20:09:53.475861+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:03.486477+00:00 scm1 osafamfd[2097]: WA saImmOiRtObjectUpdate
of 'safComp=AMFWDOG,safSu=SCM1,safSg=NoRed,safApp=OpenSAF'
saAmfCompCurrProxyName failed with 5
2014-02-25T20:10:03.486830+00:00 scm1 osafimmnd[2046]: WA Failed to retrieve
search continuation, client died ?
2014-02-25T20:10:03.487386+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:13.499618+00:00 scm1 osafamfd[2097]: NO Node 'CMM02A' left the
cluster
2014-02-25T20:10:13.545922+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:23.559431+00:00 scm1 osafamfd[2097]: NO Node 'CMM02B' left the
cluster
2014-02-25T20:10:23.615141+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:33.626894+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:43.638079+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:10:53.647198+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:03.747126+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:13.847652+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:23.858677+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:33.869608+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:43.969565+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:11:54.069799+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:04.170289+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:14.271761+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:24.282274+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:24.983510+00:00 scm1 osafimmd[2036]: NO Node 1200f request
sync sync-pid:1348 epoch:0
2014-02-25T20:12:26.214955+00:00 scm1 osafimmd[2036]: NO Node 11f0f request
sync sync-pid:1322 epoch:0
2014-02-25T20:12:26.283499+00:00 scm1 osafimmnd[2046]: NO Announce sync, epoch:7
2014-02-25T20:12:26.283592+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
2014-02-25T20:12:26.284943+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T20:12:26.285086+00:00 scm1 osafimmd[2036]: NO Successfully announced
sync. New ruling epoch:7
2014-02-25T20:12:26.388715+00:00 scm1 osafimmloadd: NO Sync starting
2014-02-25T20:12:30.056105+00:00 scm1 osafimmloadd: IN Synced 5404 objects in
total
2014-02-25T20:12:30.056627+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 13776
2014-02-25T20:12:30.059185+00:00 scm1 osafimmloadd: NO Sync ending normally
2014-02-25T20:12:30.059243+00:00 scm1 osafimmnd[2046]: NO Epoch set to 7 in
ImmModel
2014-02-25T20:12:30.059442+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 10f0f old epoch: 6 new epoch:7
2014-02-25T20:12:30.060081+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1100f old epoch: 6 new epoch:7
2014-02-25T20:12:30.082902+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1200f old epoch: 0 new epoch:7
2014-02-25T20:12:30.083622+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 11f0f old epoch: 0 new epoch:7
2014-02-25T20:12:30.149239+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_SYNC_SERVER --> IMM SERVER READY
2014-02-25T20:12:34.294011+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:44.304800+00:00 scm1 osafamfd[2097]: NO Node 'CMM02B' joined
the cluster
2014-02-25T20:12:44.307947+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:52.939822+00:00 scm1 osafimmd[2036]: NO Node 1150f request
sync sync-pid:1313 epoch:0
2014-02-25T20:12:53.016535+00:00 scm1 osafimmnd[2046]: NO Announce sync, epoch:8
2014-02-25T20:12:53.016621+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_READY --> IMM_SERVER_SYNC_SERVER
2014-02-25T20:12:53.018038+00:00 scm1 osafimmd[2036]: NO Successfully announced
sync. New ruling epoch:8
2014-02-25T20:12:53.018109+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T20:12:53.120597+00:00 scm1 osafimmloadd: NO Sync starting
2014-02-25T20:12:54.319143+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:12:56.966565+00:00 scm1 osafimmloadd: IN Synced 5404 objects in
total
2014-02-25T20:12:56.967648+00:00 scm1 osafimmnd[2046]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 13776
2014-02-25T20:12:56.970646+00:00 scm1 osafimmnd[2046]: NO Epoch set to 8 in
ImmModel
2014-02-25T20:12:56.971366+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1200f old epoch: 7 new epoch:8
2014-02-25T20:12:56.971951+00:00 scm1 osafimmloadd: NO Sync ending normally
2014-02-25T20:12:56.972854+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 11f0f old epoch: 7 new epoch:8
2014-02-25T20:12:56.973894+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 10f0f old epoch: 7 new epoch:8
2014-02-25T20:12:56.974725+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1100f old epoch: 7 new epoch:8
2014-02-25T20:12:56.994253+00:00 scm1 osafimmd[2036]: NO ACT: New Epoch for
IMMND process at node 1150f old epoch: 0 new epoch:8
2014-02-25T20:12:57.057433+00:00 scm1 osafimmnd[2046]: NO SERVER STATE:
IMM_SERVER_SYNC_SERVER --> IMM SERVER READY
2014-02-25T20:13:04.324718+00:00 scm1 osafamfd[2097]: NO Node 'CMM02A' joined
the cluster
2014-02-25T20:13:04.327856+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:13:14.338960+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:13:24.349513+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:13:24.350703+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:13:34.361701+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:13:44.363375+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:13:44.364095+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:13:54.374908+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:04.385736+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:14:04.386443+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:14.396946+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:24.401635+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:14:24.402492+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:34.413346+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:44.669897+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:14:54.680483+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:14:54.681220+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:04.693004+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:14.701700+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:15:14.702532+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:24.712937+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:34.723127+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:15:34.723946+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:44.732995+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:15:54.743908+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:15:54.744961+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:04.756452+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:14.766220+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:16:14.766872+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:24.772248+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:34.782544+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:16:34.783494+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:44.791210+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:16:54.799913+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:16:54.800802+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:04.812554+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:14.822630+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:17:14.823544+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:24.826664+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:34.836760+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:17:34.837601+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:44.849132+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:17:54.859216+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:17:54.860133+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:04.871457+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:14.881223+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:18:14.882775+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:24.892889+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:34.902980+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:18:34.903892+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:44.914855+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:18:54.925583+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:18:54.926433+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:04.938216+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:14.944024+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:19:14.944868+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:24.955935+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:34.963187+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:19:34.963903+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:44.974235+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:19:54.984632+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:19:54.985235+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:04.992851+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:15.003093+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:20:15.003889+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:25.014872+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:35.025303+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:20:35.026440+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:45.036947+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:20:55.047453+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 1200f
2014-02-25T20:20:55.048243+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:05.057988+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:15.068073+00:00 scm1 osafamfd[2097]: WA invalid node state 1
for node 11f0f
2014-02-25T20:21:15.068791+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:25.079620+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:35.090203+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:45.099054+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
2014-02-25T20:21:55.108008+00:00 scm1 osafimmnd[2046]: WA IMMND - Client went
down so no response
root 1319 0.0 0.0 0 0 ? ZNs 20:34 0:00 [osaf-immnd]
<defunct>
opensaf 1326 0.3 0.3 173916 11876 ? SNsl 20:34 0:01
/usr/lib64/opensaf/osafimmnd osafimmnd
root 1340 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-noded]
<defunct>
root 1349 0.0 0.0 0 0 ? Zs 20:34 0:00 [osaf-amfnd]
<defunct>
root 1353 0.0 0.0 159956 1720 ? Ssl 20:34 0:00
/usr/lib64/opensaf/osafamfnd osafamfnd
2014-02-25T20:12:25.837820+00:00 cmm02a osafimmnd[1322]: Started
2014-02-25T20:12:25.942329+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
2014-02-25T20:12:26.043212+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING
2014-02-25T20:12:26.143943+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING
2014-02-25T20:12:26.144367+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_ISOLATED
2014-02-25T20:12:26.214269+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_W_AVAILABLE
2014-02-25T20:12:26.243582+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT
2014-02-25T20:12:30.011710+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 2144
2014-02-25T20:12:30.011760+00:00 cmm02a osafimmnd[1322]: NO RepositoryInitModeT
is SA_IMM_INIT_FROM_FILE
2014-02-25T20:12:30.011781+00:00 cmm02a osafimmnd[1322]: NO Epoch set to 7 in
ImmModel
2014-02-25T20:12:30.055452+00:00 cmm02a osafimmnd[1322]: NO SERVER STATE:
IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY
2014-02-25T20:12:30.086278+00:00 cmm02a osafclmna[1331]: Started
2014-02-25T20:12:30.189542+00:00 cmm02a osafclmna[1331]: NO
safNode=cmm02a,safCluster=myClmCluster Joined cluster, nodeid=11f0f
2014-02-25T20:12:30.225293+00:00 cmm02a osafamfnd[1340]: Started
2014-02-25T20:12:52.957541+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_R_AVAILABLE
2014-02-25T20:12:56.911351+00:00 cmm02a osafimmnd[1322]: NO NODE STATE->
IMM_NODE_FULLY_AVAILABLE 14277
2014-02-25T20:12:56.911846+00:00 cmm02a osafimmnd[1322]: NO Epoch set to 8 in
ImmModel
2014-02-25T20:24:35.852781+00:00 cmm02a osafimmnd[1322]: NO Implementer
connected: 18 (MsgQueueService73743) <0, 1200f>
------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-users
