Hi Simon, maybe this would be the time to add tracking PC of the code to call alloc() or free() for a memory block? This does not always work that great, as often code centralizes the function where allocations happen. Therefore we would augment this with a another call which allows further ‘stamping’ of the data structure.
I’d make this work is by adding another 32-bit empty space in front of the allocated memory (taking that into account in OS_MEM_TRUE_BLOCK_SIZE()). Then stuff that field with __builtin_return_address(0) in os_memblock_get() and os_memblock_put(). And then the stamper() would overwrite it with current PC value if called (could be a function call so that we could use __builtin_XXX) there as well. This could then later be extended to add guard areas between memory blocks (if memory corruption of adjacent memory blocks ever becomes an issue for somebody). > On Feb 4, 2018, at 12:27 AM, Simon Ratner <[email protected]> wrote: > > Actually, looking at the mbuf situation, the step drops in available mbufs > correlate to the step drops in ble_gattc_proc structs (ts=263753, 359783, > 751324, for example) so I am going to call that strong evidence that both > leaks are caused by the same process: > > 037565 msys_1 88 96 95 80 > 062207 msys_1 88 96 95 80 > 068517 msys_1 88 96 95 80 > 084946 msys_1 88 96 95 80 > 087368 msys_1 88 96 94 80 > 108475 msys_1 88 96 95 80 > 128566 msys_1 88 96 95 80 > 136628 msys_1 88 96 94 80 > 147953 msys_1 88 96 92 80 > 148875 msys_1 88 96 94 80 > 161128 msys_1 88 96 92 80 > 162504 msys_1 88 96 95 80 > 173425 msys_1 88 96 92 80 > 173752 msys_1 88 96 93 80 > 263753 msys_1 88 96 72 60 > 264645 msys_1 88 96 75 60 > 265952 msys_1 88 96 72 60 > 266458 msys_1 88 96 71 60 > 270528 msys_1 88 96 70 60 > 274789 msys_1 88 96 68 60 > 286085 msys_1 88 96 67 59 > 286470 msys_1 88 96 66 59 > 297008 msys_1 88 96 70 58 > 304526 msys_1 88 96 63 53 > 309056 msys_1 88 96 65 53 > 318528 msys_1 88 96 68 53 > 359783 msys_1 88 96 47 33 > 361170 msys_1 88 96 50 33 > 364975 msys_1 88 96 50 33 > 367755 msys_1 88 96 48 33 > 371899 msys_1 88 96 50 33 > 384125 msys_1 88 96 40 33 > 384893 msys_1 88 96 39 33 > 388891 msys_1 88 96 45 33 > 396076 msys_1 88 96 47 33 > 405030 msys_1 88 96 49 33 > 405888 msys_1 88 96 49 33 > 408961 msys_1 88 96 46 33 > 417039 msys_1 88 96 50 33 > 422373 msys_1 88 96 42 33 > 428171 msys_1 88 96 48 33 > 429451 msys_1 88 96 47 33 > 440079 msys_1 88 96 41 33 > 440532 msys_1 88 96 42 33 > 448861 msys_1 88 96 45 31 > 450167 msys_1 88 96 47 31 > 459823 msys_1 88 96 40 31 > 468502 msys_1 88 96 46 31 > 476949 msys_1 88 96 38 31 > 477839 msys_1 88 96 38 31 > 480797 msys_1 88 96 39 29 > 486277 msys_1 88 96 42 29 > 487275 msys_1 88 96 43 29 > 492053 msys_1 88 96 49 29 > 498015 msys_1 88 96 47 29 > 501485 msys_1 88 96 42 29 > 502145 msys_1 88 96 41 29 > 505513 msys_1 88 96 46 29 > 510623 msys_1 88 96 41 29 > 515618 msys_1 88 96 38 25 > 517647 msys_1 88 96 37 25 > 536303 msys_1 88 96 37 25 > 537157 msys_1 88 96 38 25 > 559728 msys_1 88 96 36 23 > 593830 msys_1 88 96 38 23 > 597207 msys_1 88 96 39 23 > 609735 msys_1 88 96 38 23 > 614905 msys_1 88 96 38 23 > 617273 msys_1 88 96 33 23 > 617930 msys_1 88 96 31 23 > 618681 msys_1 88 96 32 23 > 619321 msys_1 88 96 31 23 > 627585 msys_1 88 96 32 23 > 629576 msys_1 88 96 37 21 > 632959 msys_1 88 96 40 21 > 638929 msys_1 88 96 35 21 > 639299 msys_1 88 96 39 21 > 643188 msys_1 88 96 36 21 > 644084 msys_1 88 96 37 21 > 645673 msys_1 88 96 40 21 > 649744 msys_1 88 96 31 21 > 650883 msys_1 88 96 34 21 > 651702 msys_1 88 96 39 21 > 661043 msys_1 88 96 35 21 > 704382 msys_1 88 96 21 12 > 751310 msys_1 88 96 21 2 > 757135 msys_1 88 96 15 2 > 765200 msys_1 88 96 15 2 > 826437 msys_1 88 96 17 2 > 827153 msys_1 88 96 19 2 > 836758 msys_1 88 96 18 2 > 837589 msys_1 88 96 19 2 > 841713 msys_1 88 96 20 2 > 899904 msys_1 88 96 18 2 > 973807 msys_1 88 96 17 2 > 981752 msys_1 88 96 20 2 > 994788 msys_1 88 96 19 2 > 997513 msys_1 88 96 16 2 > 1094420 msys_1 88 96 17 2 > 1112104 msys_1 88 96 20 2 > 1133801 msys_1 88 96 21 2 > 1303205 msys_1 88 96 21 2 > 1361897 msys_1 88 96 16 2 > 1374554 msys_1 88 96 21 2 > 1378502 msys_1 88 96 19 2 > 1392124 msys_1 88 96 17 2 > 1438385 msys_1 88 96 21 2 > 1441331 msys_1 88 96 21 2 > 1488757 msys_1 88 96 15 2 > 1491018 msys_1 88 96 20 2 > 1501561 msys_1 88 96 18 2 > 1735218 msys_1 88 96 13 2 > > > On Sat, Feb 3, 2018 at 2:17 PM, Simon Ratner <[email protected]> wrote: > >> Hey devs, >> >> I am seeing the number of available ble_gattc_proc structs in >> ble_gattc_proc_pool dwindling over time, suspect they are not always being >> freed (see bottom of the email for mpool output). >> >> This seems to be happening under heavy client load, where a significant >> fraction of the clients may be failing or timing out. The radio is >> nrf52832, and the procs in use are: mtu, disc_svc_uuid, disc_chr_uuid, >> disc_all_dscs, read, write, indicate_custom. There may be an mbuf leak >> associated with this, since i see those dwindle too -- but i have not >> completely eliminated application error for that one yet. >> >> I am trying to isolate the culprit, but in the meantime, if someone has an >> idea as to which bit of code is most likely to be responsible, or ran into >> something similar, any information would be appreciated. Scope of change is >> 3742e67..eb3d264 (yes, i know that's a lot). >> >> Thanks! >> >> mpool output over 3 hours: >> >> ticks name blksz cnt free min >> 037579 ble_gattc_proc_pool 56 32 32 27 >> 062221 ble_gattc_proc_pool 56 32 32 27 >> 068531 ble_gattc_proc_pool 56 32 32 27 >> 084960 ble_gattc_proc_pool 56 32 32 27 >> 087382 ble_gattc_proc_pool 56 32 31 27 >> 108489 ble_gattc_proc_pool 56 32 32 27 >> 128580 ble_gattc_proc_pool 56 32 32 27 >> 136642 ble_gattc_proc_pool 56 32 32 27 >> 147967 ble_gattc_proc_pool 56 32 31 27 >> 148889 ble_gattc_proc_pool 56 32 32 27 >> 161142 ble_gattc_proc_pool 56 32 31 27 >> 162518 ble_gattc_proc_pool 56 32 32 27 >> 173439 ble_gattc_proc_pool 56 32 31 27 >> 173766 ble_gattc_proc_pool 56 32 31 27 >> 263767 ble_gattc_proc_pool 56 32 26 22 >> 264659 ble_gattc_proc_pool 56 32 27 22 >> 265966 ble_gattc_proc_pool 56 32 26 22 >> 266472 ble_gattc_proc_pool 56 32 26 22 >> 270542 ble_gattc_proc_pool 56 32 27 22 >> 274803 ble_gattc_proc_pool 56 32 25 22 >> 286099 ble_gattc_proc_pool 56 32 25 22 >> 286484 ble_gattc_proc_pool 56 32 23 22 >> 297022 ble_gattc_proc_pool 56 32 23 22 >> 304540 ble_gattc_proc_pool 56 32 25 20 >> 309070 ble_gattc_proc_pool 56 32 25 20 >> 318542 ble_gattc_proc_pool 56 32 26 20 >> 359797 ble_gattc_proc_pool 56 32 16 11 >> 361184 ble_gattc_proc_pool 56 32 16 11 >> 364989 ble_gattc_proc_pool 56 32 16 11 >> 367769 ble_gattc_proc_pool 56 32 15 11 >> 371913 ble_gattc_proc_pool 56 32 16 11 >> 384139 ble_gattc_proc_pool 56 32 14 11 >> 384907 ble_gattc_proc_pool 56 32 14 11 >> 388905 ble_gattc_proc_pool 56 32 15 11 >> 396090 ble_gattc_proc_pool 56 32 15 11 >> 405044 ble_gattc_proc_pool 56 32 14 11 >> 405902 ble_gattc_proc_pool 56 32 15 11 >> 408975 ble_gattc_proc_pool 56 32 15 11 >> 417053 ble_gattc_proc_pool 56 32 14 11 >> 422387 ble_gattc_proc_pool 56 32 13 11 >> 428185 ble_gattc_proc_pool 56 32 15 11 >> 429465 ble_gattc_proc_pool 56 32 14 11 >> 440093 ble_gattc_proc_pool 56 32 13 11 >> 440546 ble_gattc_proc_pool 56 32 14 11 >> 448875 ble_gattc_proc_pool 56 32 13 10 >> 450181 ble_gattc_proc_pool 56 32 12 10 >> 459837 ble_gattc_proc_pool 56 32 13 10 >> 468516 ble_gattc_proc_pool 56 32 14 10 >> 476963 ble_gattc_proc_pool 56 32 12 10 >> 477853 ble_gattc_proc_pool 56 32 14 10 >> 480811 ble_gattc_proc_pool 56 32 12 10 >> 486291 ble_gattc_proc_pool 56 32 14 10 >> 487289 ble_gattc_proc_pool 56 32 14 10 >> 492067 ble_gattc_proc_pool 56 32 14 10 >> 498029 ble_gattc_proc_pool 56 32 14 10 >> 501499 ble_gattc_proc_pool 56 32 12 10 >> 502159 ble_gattc_proc_pool 56 32 14 10 >> 505527 ble_gattc_proc_pool 56 32 14 10 >> 510637 ble_gattc_proc_pool 56 32 14 10 >> 515632 ble_gattc_proc_pool 56 32 14 8 >> 517661 ble_gattc_proc_pool 56 32 13 8 >> 536317 ble_gattc_proc_pool 56 32 11 8 >> 537171 ble_gattc_proc_pool 56 32 11 8 >> 559742 ble_gattc_proc_pool 56 32 10 6 >> 593844 ble_gattc_proc_pool 56 32 9 5 >> 597221 ble_gattc_proc_pool 56 32 10 5 >> 609749 ble_gattc_proc_pool 56 32 10 5 >> 614919 ble_gattc_proc_pool 56 32 10 5 >> 617287 ble_gattc_proc_pool 56 32 9 5 >> 617944 ble_gattc_proc_pool 56 32 9 5 >> 618695 ble_gattc_proc_pool 56 32 9 5 >> 619334 ble_gattc_proc_pool 56 32 7 5 >> 627599 ble_gattc_proc_pool 56 32 9 5 >> 629590 ble_gattc_proc_pool 56 32 8 5 >> 632973 ble_gattc_proc_pool 56 32 9 5 >> 638943 ble_gattc_proc_pool 56 32 8 5 >> 639313 ble_gattc_proc_pool 56 32 9 5 >> 643202 ble_gattc_proc_pool 56 32 9 5 >> 644098 ble_gattc_proc_pool 56 32 8 5 >> 645687 ble_gattc_proc_pool 56 32 9 5 >> 649758 ble_gattc_proc_pool 56 32 8 5 >> 650897 ble_gattc_proc_pool 56 32 8 4 >> 651717 ble_gattc_proc_pool 56 32 9 4 >> 661057 ble_gattc_proc_pool 56 32 8 4 >> 704396 ble_gattc_proc_pool 56 32 6 3 >> 751324 ble_gattc_proc_pool 56 32 2 0 >> 757149 ble_gattc_proc_pool 56 32 0 0 >> 765214 ble_gattc_proc_pool 56 32 1 0 >> 826451 ble_gattc_proc_pool 56 32 0 0 >> 827167 ble_gattc_proc_pool 56 32 1 0 >> 836772 ble_gattc_proc_pool 56 32 1 0 >> 837603 ble_gattc_proc_pool 56 32 1 0 >> 841727 ble_gattc_proc_pool 56 32 1 0 >> 899918 ble_gattc_proc_pool 56 32 1 0 >> 973821 ble_gattc_proc_pool 56 32 1 0 >> 981766 ble_gattc_proc_pool 56 32 0 0 >> 994802 ble_gattc_proc_pool 56 32 1 0 >> 997527 ble_gattc_proc_pool 56 32 0 0 >> 1094434 ble_gattc_proc_pool 56 32 1 0 >> 1112118 ble_gattc_proc_pool 56 32 1 0 >> 1133815 ble_gattc_proc_pool 56 32 1 0 >> 1303219 ble_gattc_proc_pool 56 32 1 0 >> 1361911 ble_gattc_proc_pool 56 32 0 0 >> 1374568 ble_gattc_proc_pool 56 32 1 0 >> 1378516 ble_gattc_proc_pool 56 32 1 0 >> 1392138 ble_gattc_proc_pool 56 32 0 0 >> 1438400 ble_gattc_proc_pool 56 32 1 0 >> 1441345 ble_gattc_proc_pool 56 32 1 0 >> 1488771 ble_gattc_proc_pool 56 32 0 0 >> 1491032 ble_gattc_proc_pool 56 32 1 0 >> 1501575 ble_gattc_proc_pool 56 32 1 0 >> >>
