- **status**: review --> needinfo
---
** [tickets:#1510] CKPT: cpnd crashes during checkpoint open timeout with large
sections**
**Status:** needinfo
**Milestone:** future
**Created:** Thu Oct 01, 2015 04:14 PM UTC by Alex Jones
**Last Updated:** Tue Sep 20, 2016 05:51 AM UTC
**Owner:** Alex Jones
When opening a collocated checkpoint replica where the active has large numbers
of sections (~200k), the sync from the active can timeout with errorcode
SA_AIS_ERR_TRY_AGAIN. In this case the code deletes the memory for the node,
but does not delete the node from the db. When the checkpoint access is tried
again, the freed memory for the node is still in the db, and ckptnd crashes.
Valgrind analysis shows the following:
==53610== Thread 1:
==53610== Invalid read of size 4
==53610== at 0x4E4D7C4: ncs_patricia_tree_get (patricia.c:93)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de60 is 0 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7C0: ncs_patricia_tree_get (patricia.c:90)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de70 is 16 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7FB: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de78 is 24 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0B9: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0D0: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de81 is 33 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x4E4D7C4: ncs_patricia_tree_get (patricia.c:93)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de60 is 0 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7C0: ncs_patricia_tree_get (patricia.c:90)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de70 is 16 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7FB: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de78 is 24 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0B9: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0D0: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de81 is 33 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x418613: cpnd_ckpt_sec_get (cpnd_sec.cc:99)
==53610== by 0x418708: cpnd_ckpt_sec_find (cpnd_sec.cc:156)
==53610== by 0x405889: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2609)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687dfd8 is 376 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x418680: cpnd_ckpt_sec_get (cpnd_sec.cc:115)
==53610== by 0x418708: cpnd_ckpt_sec_find (cpnd_sec.cc:156)
==53610== by 0x405889: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2609)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x40589E: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2616)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687e040 is 480 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x411D80: cpnd_ckpt_get_lck_sec_id (cpnd_proc.c:599)
==53610== by 0x4043C7: cpnd_ckpt_sec_add (cpnd_db.c:380)
==53610== by 0x40595E: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2617)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687dfa8 is 328 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
