- **status**: unassigned --> accepted
- **assigned_to**: Anders Bjornerstedt
- **Milestone**: 4.3.3 --> 4.4.1
- **Comment**:
Is the test that provokes this problem a new test (recently introduced)
or is it an old test that has previously not provoked these problems?
2PBE was introduced in OpensF 4.4 so the test could in principle have
been created in relation to 4.4
---
** [tickets:#1057] (2PBE) Slave PBE restarts multiple times**
**Status:** accepted
**Milestone:** 4.4.1
**Created:** Wed Sep 10, 2014 07:00 AM UTC by Sirisha Alla
**Last Updated:** Wed Sep 10, 2014 07:23 AM UTC
**Owner:** Anders Bjornerstedt
The issue is seen on SLES X86. Opensaf is up with changeset 5697 and 2pbe is
loaded with 50k objects.
When CCBs are in progress, prto object creation is attempted. Following is the
syslog of SLOT2 when slave pbe restarted
Sep 9 15:55:37 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Ccb 644 COMMITTED (exowner)
Sep 9 15:55:37 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:37 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:37 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:38 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:39 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:40 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:1000005a5/4294968741 numOps:1
Sep 9 15:55:40 SLES-64BIT-SLOT2 osafimmpbed: NO Prepare
ccb:1000005a5/4294968741 received at Pbe slave when Prior Ccb 644 still
processing
Sep 9 15:55:40 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:40 SLES-64BIT-SLOT2 osafimmnd[6344]: WA Create of PERSISTENT
runtime object 'DistObj3=DistRunTime,DistObj1=DistRunTime' REVERTED. PBE rc:20
Sep 9 15:55:40 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005a5
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Implementer disconnected
579 <0, 2040f> (implementertestMA_verifyObjImplReleaseModCallbackNode_101_133)
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Implementer disconnected
581 <0, 2040f> (@applier1testMA_verifyObjImplReleaseModCallbackNode_101_133)
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: IN Delaying class delete at slave
PBE due to ongoing commit of ccb:284/644
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: NO Slave PBE time-out in waiting
on porepare for PRTO create ccb:1000005a5
dn:DistObj3=DistRunTime,DistObj1=DistRunTime
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: NO 2PBE Error (21) in PRTO create
(ccbId:1000005a5)
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: ER PBE-B got completed callback
for Ccb:284/644 before prepare from PBE-A
Sep 9 15:55:41 SLES-64BIT-SLOT2 osafimmpbed: WA PBE slave exiting in prepare
for ccb 284/644, file should be regenerated.
Sep 9 15:55:42 SLES-64BIT-SLOT2 osafimmnd[6344]: WA SLAVE PBE process has
apparently died at non coord
Sep 9 15:55:42 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Delete of class
testMA_verifyObjImplReleaseModCallbackNode_101_133 is PERSISTENT.
Sep 9 15:55:42 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Implementer disconnected
580 <0, 2030f> (@applier2testMA_verifyObjImplReleaseModCallbackNode_101_133)
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmnd[6344]: NO STARTING SLAVE PBE process.
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmnd[6344]: NO
pbe-db-file-path:/home/sirisha/immsv/immpbe//imm.db.2020f VETERAN:1 B:1
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: logtrace: trace enabled to file
/var/log/opensaf/osafimmnd, mask=0xffffffff
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: IN arg[0] ==
'/usr/lib64/opensaf/osafimmpbed'
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: IN arg[1] == '--recover'
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: IN arg[2] == '--pbe2B'
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: IN arg[3] ==
'/home/sirisha/immsv/immpbe//imm.db.2020f'
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: NO Successfully opened
pre-existing sqlite pbe file /home/sirisha/immsv/immpbe//imm.db.2020f
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: IN saImmRepositoryInit:
SA_IMM_KEEP_REPOSITORY - attaching to repository
Sep 9 15:55:43 SLES-64BIT-SLOT2 osafimmpbed: NO pbeDaemon starting with
obj-count:50477
Sep 9 15:56:31 SLES-64BIT-SLOT2 osafimmnd[6344]: WA Create of PERSISTENT
runtime object 'DistObj3=DistRunTime,DistObj1=DistRunTime' REVERTED. PBE rc:20
Sep 9 15:56:31 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005be
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmpbed: IN PBE slave waiting for prepare
from primary on PRTO create ccb:1000005be
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: NO implementer for class
'testMA_verifyObjApplRejDelCallbackNode_101_133' is released => class extent is
UNSAFE
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmpbed: NO Slave PBE time-out in waiting
on porepare for PRTO create ccb:1000005be
dn:DistObj3=DistRunTime,DistObj1=DistRunTime
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmpbed: NO 2PBE Error (21) in PRTO create
(ccbId:1000005be)
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmpbed: ER PBE-B got completed callback
for Ccb:28a/650 before prepare from PBE-A
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmpbed: WA PBE slave exiting in prepare
for ccb 28a/650, file should be regenerated.
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Implementer locally
disconnected. Marking it as doomed 583 <316, 2020f> (@OpenSafImmPBE)
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Implementer locally
disconnected. Marking it as doomed 584 <317, 2020f> (OsafImmPbeRt_B)
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: NO Precheck of fevs message
of type <0> failed with ERROR:9
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: ER Asyncronous FEVS message
failed verification - dropping message!
Sep 9 15:56:32 SLES-64BIT-SLOT2 osafimmnd[6344]: WA Error code 2 returned for
message type 14 - ignoring
After this slave pbe continuously restarted leading to failures in the
application.
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmnd[6344]: NO
pbe-db-file-path:/home/sirisha/immsv/immpbe//imm.db.2020f VETERAN:1 B:1
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: logtrace: trace enabled to file
/var/log/opensaf/osafimmnd, mask=0xffffffff
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: IN arg[0] ==
'/usr/lib64/opensaf/osafimmpbed'
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: IN arg[1] == '--recover'
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: IN arg[2] == '--pbe2B'
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: IN arg[3] ==
'/home/sirisha/immsv/immpbe//imm.db.2020f'
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: NO Successfully opened
pre-existing sqlite pbe file /home/sirisha/immsv/immpbe//imm.db.2020f
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: IN saImmRepositoryInit:
SA_IMM_KEEP_REPOSITORY - attaching to repository
Sep 9 15:56:33 SLES-64BIT-SLOT2 osafimmpbed: NO pbeDaemon starting with
obj-count:50481
Sep 9 15:56:38 SLES-64BIT-SLOT2 osafimmpbed: ER saImmOiImplementerSet for
@OpenSafImmPBE failed 6
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmnd[6344]: WA SLAVE PBE process has
apparently died at non coord
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmnd[6344]: NO STARTING SLAVE PBE process.
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmnd[6344]: NO
pbe-db-file-path:/home/sirisha/immsv/immpbe//imm.db.2020f VETERAN:1 B:0
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: logtrace: trace enabled to file
/var/log/opensaf/osafimmnd, mask=0xffffffff
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: IN arg[0] ==
'/usr/lib64/opensaf/osafimmpbed'
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: IN arg[1] == '--pbe2B'
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: IN arg[2] ==
'/home/sirisha/immsv/immpbe//imm.db.2020f'
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: IN Generating DB file from
current IMM state. DB file: /home/sirisha/immsv/immpbe//imm.db.2020f
Sep 9 15:56:39 SLES-64BIT-SLOT2 osafimmpbed: NO Successfully opened empty
local sqlite pbe file /tmp/imm.db.4zX3j7
Sep 9 15:56:40 SLES-64BIT-SLOT2 osafimmnd[6344]: WA ERR_NO_RESOURCES: Active
Ccbs still exist in the system
Sep 9 15:56:40 SLES-64BIT-SLOT2 osafimmpbed: ER Failed on
saImmOmSearchInitialize:18 - exiting
Sep 9 15:56:40 SLES-64BIT-SLOT2 osafimmpbed: ER immPbe.cc exiting (line:264)
Slave PBE successfully came up once the application is stopped. Syslog and
immnd traces attached.
---
Sent from sourceforge.net because opensaf-tickets@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-tickets mailing list
Opensaf-tickets@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
