- **status**: review --> fixed
- **Comment**:
changeset: 6060:3b42343c8381
tag: tip
parent: 6057:27c8d8811798
user: Anders Bjornerstedt <[email protected]>
date: Wed Oct 22 14:33:13 2014 +0200
summary: IMM 2PBE: Abort-callback before completed-callback aborts
sqlite-trans [#1063]
changeset: 6059:a07efd76e50f
branch: opensaf-4.5.x
parent: 6054:663296aecefa
user: Anders Bjornerstedt <[email protected]>
date: Wed Oct 22 14:33:13 2014 +0200
summary: IMM 2PBE: Abort-callback before completed-callback aborts
sqlite-trans [#1063]
changeset: 6058:c8e34258a888
branch: opensaf-4.4.x
parent: 6053:00d8d29bd3ed
user: Anders Bjornerstedt <[email protected]>
date: Wed Oct 22 14:33:13 2014 +0200
summary: IMM 2PBE: Abort-callback before completed-callback aborts
sqlite-trans [#1063]
---
** [tickets:#1063] 2PBE: Hung PBE implementer**
**Status:** fixed
**Milestone:** 4.4.2
**Created:** Thu Sep 11, 2014 09:13 AM UTC by Sirisha Alla
**Last Updated:** Wed Oct 22, 2014 12:37 PM UTC
**Owner:** Anders Bjornerstedt
The issue is seenon SLES X86 VMs. Opensaf is running on changeset 5697.
Both the controllers are brought up with 2PBE holding 50k objects. When amf
application components are being created from Standby Controller SC-2,
following error is observed:
Sep 11 12:05:04 SLES-64BIT-SLOT1 osafimmpbed: IN Starting distributed PBE
commit for Ccb:2/2
Sep 11 12:05:10 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:2
Sep 11 12:05:10 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:11 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:2
Sep 11 12:05:11 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:12 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:2
Sep 11 12:05:12 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:13 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:2
Sep 11 12:05:13 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:14 SLES-64BIT-SLOT1 osafimmpbed: WA Start prepare for ccb: 2/2
towards slave PBE returned: '5' from Immsv
Sep 11 12:05:14 SLES-64BIT-SLOT1 osafimmpbed: WA PBE-A failed to prepare
Ccb:2/2 towards PBE-B
Sep 11 12:05:14 SLES-64BIT-SLOT1 osafimmnd[3659]: NO Validation error
(BAD_OPERATION) reported by implementer 'OpenSafImmPBE', Ccb 2 will be aborted
Sep 11 12:05:14 SLES-64BIT-SLOT1 osafimmnd[3659]: NO Ccb 2 ABORTED
(immcfg_SLES-64BIT-SLOT2_4684)
Sep 11 12:05:14 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
2/2
Sep 11 12:05:15 SLES-64BIT-SLOT1 osafimmpbed: IN Starting distributed PBE
commit for Ccb:3/3
Sep 11 12:05:20 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Continuation for AdminOp
was lost, probably due to timeout
Sep 11 12:05:21 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:3
Sep 11 12:05:21 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:22 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:3
Sep 11 12:05:22 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:23 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:3
Sep 11 12:05:23 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:24 SLES-64BIT-SLOT1 osafimmnd[3659]: WA Timeout (6) on transaction
in critical state! ccb:3
Sep 11 12:05:24 SLES-64BIT-SLOT1 osafimmnd[3659]: WA PBE implementer 11 seems
hung!
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmpbed: WA Start prepare for ccb: 3/3
towards slave PBE returned: '5' from Immsv
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmpbed: WA PBE-A failed to prepare
Ccb:3/3 towards PBE-B
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmnd[3659]: NO Validation error
(BAD_OPERATION) reported by implementer 'OpenSafImmPBE', Ccb 3 will be aborted
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmnd[3659]: NO Ccb 3 ABORTED
(immcfg_SLES-64BIT-SLOT2_4691)
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
3/3
Sep 11 12:05:25 SLES-64BIT-SLOT1 osafimmpbed: IN Starting distributed PBE
commit for Ccb:4/4
Syslog of SC-2:
Sep 11 12:05:15 SLES-64BIT-SLOT2 osafimmnd[3538]: NO Validation error
(BAD_OPERATION) reported by implementer 'OpenSafImmPBE', Ccb 2 will be aborted
Sep 11 12:05:15 SLES-64BIT-SLOT2 osafimmnd[3538]: NO Ccb 2 ABORTED
(immcfg_SLES-64BIT-SLOT2_4684)
Sep 11 12:05:20 SLES-64BIT-SLOT2 osafimmpbed: IN ccb-prepare received at PBE
slave ccbId:3/3 numOps:1
Sep 11 12:05:20 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:21 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:21 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:22 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:22 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:23 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:23 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:24 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:24 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:25 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:25 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:26 SLES-64BIT-SLOT2 osafimmnd[3538]: NO ccbResult: CCB 3 in
critical state! Commit/apply in progress
Sep 11 12:05:26 SLES-64BIT-SLOT2 osafimmnd[3538]: NO Validation error
(BAD_OPERATION) reported by implementer 'OpenSafImmPBE', Ccb 3 will be aborted
Sep 11 12:05:26 SLES-64BIT-SLOT2 osafimmnd[3538]: NO Ccb 3 ABORTED
(immcfg_SLES-64BIT-SLOT2_4691)
Sep 11 12:05:26 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:26 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:27 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:27 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:28 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:28 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:29 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:29 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:30 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:30 SLES-64BIT-SLOT2 osafimmpbed: WA Sqlite db locked by other
thread.
Sep 11 12:05:30 SLES-64BIT-SLOT2 osafimmpbed: ER Sqlite db appears blocked on
other transaction
Sep 11 12:05:30 SLES-64BIT-SLOT2 osafimmpbed: WA PBE failed to start sqlite
transaction for CCB 3/3
Sep 11 12:05:30 SLES-64BIT-SLOT2 osafimmpbed: ER Exiting rc=21 (line:1140)
syslog and immnd trace on SC-2 attached. Unfortunately there are no traces
generated on SC-1 around the time of failure.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
