- **status**: unassigned --> accepted
- **assigned_to**: A V Mahesh (AVM)
---
** [tickets:#2011] ckptd seg faulted on active controller when trying to create
checkpoint**
**Status:** accepted
**Milestone:** 4.7.2
**Created:** Thu Sep 08, 2016 07:28 AM UTC by Ritu Raj
**Last Updated:** Thu Sep 08, 2016 07:28 AM UTC
**Owner:** A V Mahesh (AVM)
**Attachments:**
-
[ckptd_bt](https://sourceforge.net/p/opensaf/tickets/2011/attachment/ckptd_bt)
(2.6 kB; application/octet-stream)
-
[messages-20160907.bz2](https://sourceforge.net/p/opensaf/tickets/2011/attachment/messages-20160907.bz2)
(380.1 kB; application/x-bzip)
- [syslog2](https://sourceforge.net/p/opensaf/tickets/2011/attachment/syslog2)
(1.4 MB; application/octet-stream)
Environment details
OS : Suse 64bit
Changeset : 7997 ( 5.1.FC)
Setup : 4 nodes ( 2 controllers and 2 payloads with headless feature disabled &
1PBE enabled with 30K objects )
Summary :
ckptd crashed on active controller when trying to create checkpoint during
failover
Steps followed & Observed behaviour
1. Initially ran some CKPT test scenarios, along with failovers. After the end
of the test scenarios, The following IMM objects & replicas are not deleted
sofo-s3:/dev/shm # immfind | grep 101
safCkpt=all_replicas_ckpt_name_101
safCkpt=collocated_ckpt_name_101
safReplica=safNode=PL-3\,safCluster=myClmCluster,safCkpt=all_replicas_ckpt_name_101
safReplica=safNode=PL-3\,safCluster=myClmCluster,safCkpt=collocated_ckpt_name_101
safReplica=safNode=SC-1\,safCluster=myClmCluster,safCkpt=all_replicas_ckpt_name_101
safReplica=safNode=SC-2\,safCluster=myClmCluster,safCkpt=all_replicas_ckpt_name_101
2. When ckpt is created with the earlier name (all_replicas_ckpt_name_101)
observed the following error in syslog. Also CkptOpen failed with ERR_LIBRARY.
>> saImmOiRtObjectCreate_2 failed with error = 14
>>
Sep 7 17:21:11 sofo-s2 osafimmnd[2137]: NO PBE-OI established on this SC.
Dumping incrementally to file imm.db
Sep 7 17:21:12 sofo-s2 osafckptd[2284]: ER create_runtime_ckpt_object -
saImmOiRtObjectCreate_2 failed with error = 14
Sep 7 17:21:12 sofo-s2 osafckptd[2284]: ER create runtime ckpt object failed
with error: 14
Sep 7 17:21:12 sofo-s2 osafckptd[2284]: ER cpd db add ckpt_node failed for
ckpt_id:2
4. After some time cpktd seg faulted on active controller
>>
Sep 7 17:21:43 sofo-s2 osafamfnd[2187]: NO
'safComp=CPD,safSu=SC-2,safSg=2N,safApp=OpenSAF' faulted due to 'avaDown' :
Recovery is 'nodeFailfast'
Sep 7 17:21:43 sofo-s2 osafamfnd[2187]: ER
safComp=CPD,safSu=SC-2,safSg=2N,safApp=OpenSAF Faulted due to:avaDown Recovery
is:nodeFailfast
Sep 7 17:21:43 sofo-s2 osafamfnd[2187]: Rebooting OpenSAF NodeId = 131599 EE
Name = , Reason: Component faulted: recovery is node failfast, OwnNodeId =
131599, SupervisionTime = 60
Sep 7 17:21:43 sofo-s2 opensaf_reboot: Rebooting local node; timeout=60
5. Below is the bt
0- 0x00007fbbd5ffcb20 in memcmp () from /lib64/libc.so.6
1- 0x00007fbbd7a10929 in ncs_patricia_tree_get (pTree=0x67b4c8,
pKey=0x7ffffd22531c "\017\001\002") at patricia.c:435
2- 0x000000000040800d in cpd_cpnd_info_node_get (cpnd_tree=0x67b4c8,
dest=0x67ec60, cpnd_info_node=0x7ffffd225350) at cpd_db.c:706
3- 0x000000000040cd56 in cpd_evt_proc_mds_evt (cb=0x67b340, evt=0x67ec50) at
cpd_evt.c:1378
4- 0x00000000004091cb in cpd_process_evt (evt=0x67ec40) at cpd_evt.c:107
5- 0x000000000041185f in cpd_main_process (cb=0x67b340) at cpd_init.c:661
6 - 0x0000000000411b89 in main (argc=1, argv=0x7ffffd225578) at cpd_main.c:74
Notes:
1. Syslog attached
2. bt attached
3. ckptd traces not enabled
---
Sent from sourceforge.net because opensaf-tickets@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
_______________________________________________
Opensaf-tickets mailing list
Opensaf-tickets@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
