Ticket #1001 illustrates that there is a need for tests that verify *content"
in immsv.
It is not enough to just verify only that the immsv behaves correctly via its
APIs.
In this case it was luck that the corruption was identical for more than one
CCB.
This caused an integrity violation detectable at the basic sqlite level of
primary key uniqueness.
We cant depend on such "luck".
/AndersBj
________________________________
From: Anders Bjornerstedt [mailto:[email protected]]
Sent: den 21 augusti 2014 13:57
To: [opensaf:tickets]
Subject: [opensaf:tickets] #1001 IMMSv is not honoring further CCB commits once
SQL operation failure is reported by PBE
We see this in the traces leading up to the sqlite integrity violation:
CCB 45 creates an object with the empty DN (or possibly non printable DN):
Aug 21 15:39:56.186353 osafimmpbed [13438:immpbe_daemon.cc:1382] TR Begin PBE
transaction for CCB 45 OK
Aug 21 15:39:56.186433 osafimmpbed [13438:immpbe_daemon.cc:0105] TR Create of
object with DN:
Then CCB 46 does the same and gets rejected by sqlite integrity violation
because of duplicate (empty or unprintable) on primary key:
Aug 21 15:39:56.350387 osafimmpbed [13438:immpbe_daemon.cc:1382] TR Begin PBE
transaction for CCB 46 OK
Aug 21 15:39:56.350417 osafimmpbed [13438:immpbe_daemon.cc:0105] TR Create of
object with DN:
Aug 21 15:39:56.350525 osafimmpbed [13438:immpbe_dump.cc:2311] ER SQL
statement('INSERT INTO objects (obj_id, class_id, dn, last_ccb) VALUES (?, ?,
?, ?)') failed with error code: 19
Of course an empty or unprintable DN is not allowed by immsv and in fact
earlier when the create callback for ccb 45 reaches PBE it extrracts a
correct DN:
Aug 21 15:39:55.950013 osafimmpbed [13438:immpbe_daemon.cc:1572] TR Extracted
DN: attrName_Object4(16)
And for ccb 46:
Aug 21 15:39:55.971075 osafimmpbed [13438:immpbe_daemon.cc:1572] TR Extracted
DN: attrName_Object5(16)
So somehow the heap gets corrupted in the PBE....
But this seems "easy" to reproduce so we should be able to find
what causes the corruption soon.
________________________________
[tickets:#1001]<http://sourceforge.net/p/opensaf/tickets/1001> IMMSv is not
honoring further CCB commits once SQL operation failure is reported by PBE
Status: accepted
Milestone: 4.5.0
Created: Thu Aug 21, 2014 05:38 AM UTC by Sirisha Alla
Last Updated: Thu Aug 21, 2014 10:52 AM UTC
Owner: Anders Bjornerstedt
This issue is seen on SLES 64bit 4 node VMs. The changeset of OpenSAF used is
5608 along with patches for #938,#994 and #997.
IMM applications are in progress when the issue is observed. During one of the
CCB commit operations following is observed on the active controller:
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: ER SQL statement('INSERT INTO
objects (obj_id, class_id, dn, last_ccb) VALUES (?, ?, ?, ?)') failed with
error code: 19
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 46 will be aborted
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 46 ABORTED (exowner)
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
2e/46
Once this error is seen further CCB commits(ccbApply()) fail with
FAILED_OPERATION. Cluster becomes unusable for CCB commit operations.
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 47 will be aborted
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 47 ABORTED (adminOwner)
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
2f/47
Aug 21 10:15:52 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class
noDanglingPreconfigurationClass committing with ccbId:100000089
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 92 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 92 ABORTED
(noDanglingPreconfigurationClass)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
5c/92
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Delete of class
noDanglingPreconfigurationClass committing with ccbId:10000008a
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Delete of class
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class
immsv_N0_Dangling_test01 committing with ccbId:10000008b
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class
immsv_N0_Dangling_test01 is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 98 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 98 ABORTED
(immsv_N0_Dangling_test01)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
62/98
This issue is reproducible. Traces are attached. IMMD traces are available and
will be able to share them on request. I am not sure whether this is
consequence after #1000 is observed. I will try to see if this can be
reproduced independantly.
________________________________
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/opensaf/tickets/1001/<https://sourceforge.net/p/opensaf/tickets/1001>
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/<https://sourceforge.net/auth/subscriptions>
---
** [tickets:#1001] IMM: PBE fails in sqlite on commit of CCB object create**
**Status:** fixed
**Milestone:** 4.5.0
**Created:** Thu Aug 21, 2014 05:38 AM UTC by Sirisha Alla
**Last Updated:** Fri Aug 22, 2014 08:36 AM UTC
**Owner:** Zoran Milinkovic
This issue is seen on SLES 64bit 4 node VMs. The changeset of OpenSAF used is
5608 along with patches for #938,#994 and #997.
IMM applications are in progress when the issue is observed. During one of the
CCB commit operations following is observed on the active controller:
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: ER SQL statement('INSERT INTO
objects (obj_id, class_id, dn, last_ccb) VALUES (?, ?, ?, ?)') failed with
error code: 19
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 46 will be aborted
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 46 ABORTED (exowner)
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
2e/46
Once this error is seen further CCB commits(ccbApply()) fail with
FAILED_OPERATION. Cluster becomes unusable for CCB commit operations.
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 47 will be aborted
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 47 ABORTED (adminOwner)
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
2f/47
Aug 21 10:15:52 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class
noDanglingPreconfigurationClass committing with ccbId:100000089
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 92 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 92 ABORTED
(noDanglingPreconfigurationClass)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
5c/92
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Delete of class
noDanglingPreconfigurationClass committing with ccbId:10000008a
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Delete of class
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class
immsv_N0_Dangling_test01 committing with ccbId:10000008b
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class
immsv_N0_Dangling_test01 is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported
implementer 'OpenSafImmPBE', Ccb 98 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 98 ABORTED
(immsv_N0_Dangling_test01)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for
62/98
This issue is reproducible. Traces are attached. IMMD traces are available and
will be able to share them on request. I am not sure whether this is
consequence after #1000 is observed. I will try to see if this can be
reproduced independantly.
---
Sent from sourceforge.net because [email protected] is
subscribed to http://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
http://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds. Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
