- **status**: review --> fixed
- **Milestone**: future --> 5.17.10
- **Comment**:
commit ee7cb901b8eb023502e3c21180afb78dd15c328b
Author: Alex Jones <[email protected]>
Date: Tue Oct 24 09:10:15 2017 -0400
---
** [tickets:#1510] CKPT: cpnd crashes during checkpoint open timeout with large
sections**
**Status:** fixed
**Milestone:** 5.17.10
**Created:** Thu Oct 01, 2015 04:14 PM UTC by Alex Jones
**Last Updated:** Tue Oct 17, 2017 02:20 PM UTC
**Owner:** Alex Jones
When opening a collocated checkpoint replica where the active has large numbers
of sections (~200k), the sync from the active can timeout with errorcode
SA_AIS_ERR_TRY_AGAIN. In this case the code deletes the memory for the node,
but does not delete the node from the db. When the checkpoint access is tried
again, the freed memory for the node is still in the db, and ckptnd crashes.
Valgrind analysis shows the following:
==53610== Thread 1:
==53610== Invalid read of size 4
==53610== at 0x4E4D7C4: ncs_patricia_tree_get (patricia.c:93)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de60 is 0 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7C0: ncs_patricia_tree_get (patricia.c:90)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de70 is 16 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7FB: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de78 is 24 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0B9: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0D0: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x40D1A2: cpnd_process_evt (cpnd_evt.c:1957)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de81 is 33 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x4E4D7C4: ncs_patricia_tree_get (patricia.c:93)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de60 is 0 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7C0: ncs_patricia_tree_get (patricia.c:90)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de70 is 16 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x4E4D7FB: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de78 is 24 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0B9: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x4C2D0D0: bcmp (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x4E4D803: ncs_patricia_tree_get (patricia.c:435)
==53610== by 0x40400D: cpnd_ckpt_node_get (cpnd_db.c:42)
==53610== by 0x405872: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2602)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de81 is 33 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x418613: cpnd_ckpt_sec_get (cpnd_sec.cc:99)
==53610== by 0x418708: cpnd_ckpt_sec_find (cpnd_sec.cc:156)
==53610== by 0x405889: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2609)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687dfd8 is 376 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 8
==53610== at 0x418680: cpnd_ckpt_sec_get (cpnd_sec.cc:115)
==53610== by 0x418708: cpnd_ckpt_sec_find (cpnd_sec.cc:156)
==53610== by 0x405889: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2609)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687de80 is 32 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 1
==53610== at 0x40589E: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2616)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687e040 is 480 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
==53610== by 0x40D426: cpnd_process_evt (cpnd_evt.c:202)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610==
==53610== Invalid read of size 4
==53610== at 0x411D80: cpnd_ckpt_get_lck_sec_id (cpnd_proc.c:599)
==53610== by 0x4043C7: cpnd_ckpt_sec_add (cpnd_db.c:380)
==53610== by 0x40595E: cpnd_evt_proc_nd2nd_ckpt_sect_create (cpnd_evt.c:2617)
==53610== by 0x40D2B8: cpnd_process_evt (cpnd_evt.c:335)
==53610== by 0x40E9D6: cpnd_main_process (cpnd_init.c:568)
==53610== by 0x403882: main (cpnd_main.c:72)
==53610== Address 0x687dfa8 is 328 bytes inside a block of size 1,072 free'd
==53610== at 0x4C29D4E: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==53610== by 0x40A827: cpnd_evt_proc_ckpt_open (cpnd_evt.c:983)
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
