Actually, looking at the mbuf situation, the step drops in available mbufs correlate to the step drops in ble_gattc_proc structs (ts=263753, 359783, 751324, for example) so I am going to call that strong evidence that both leaks are caused by the same process:
037565 msys_1 88 96 95 80 062207 msys_1 88 96 95 80 068517 msys_1 88 96 95 80 084946 msys_1 88 96 95 80 087368 msys_1 88 96 94 80 108475 msys_1 88 96 95 80 128566 msys_1 88 96 95 80 136628 msys_1 88 96 94 80 147953 msys_1 88 96 92 80 148875 msys_1 88 96 94 80 161128 msys_1 88 96 92 80 162504 msys_1 88 96 95 80 173425 msys_1 88 96 92 80 173752 msys_1 88 96 93 80 263753 msys_1 88 96 72 60 264645 msys_1 88 96 75 60 265952 msys_1 88 96 72 60 266458 msys_1 88 96 71 60 270528 msys_1 88 96 70 60 274789 msys_1 88 96 68 60 286085 msys_1 88 96 67 59 286470 msys_1 88 96 66 59 297008 msys_1 88 96 70 58 304526 msys_1 88 96 63 53 309056 msys_1 88 96 65 53 318528 msys_1 88 96 68 53 359783 msys_1 88 96 47 33 361170 msys_1 88 96 50 33 364975 msys_1 88 96 50 33 367755 msys_1 88 96 48 33 371899 msys_1 88 96 50 33 384125 msys_1 88 96 40 33 384893 msys_1 88 96 39 33 388891 msys_1 88 96 45 33 396076 msys_1 88 96 47 33 405030 msys_1 88 96 49 33 405888 msys_1 88 96 49 33 408961 msys_1 88 96 46 33 417039 msys_1 88 96 50 33 422373 msys_1 88 96 42 33 428171 msys_1 88 96 48 33 429451 msys_1 88 96 47 33 440079 msys_1 88 96 41 33 440532 msys_1 88 96 42 33 448861 msys_1 88 96 45 31 450167 msys_1 88 96 47 31 459823 msys_1 88 96 40 31 468502 msys_1 88 96 46 31 476949 msys_1 88 96 38 31 477839 msys_1 88 96 38 31 480797 msys_1 88 96 39 29 486277 msys_1 88 96 42 29 487275 msys_1 88 96 43 29 492053 msys_1 88 96 49 29 498015 msys_1 88 96 47 29 501485 msys_1 88 96 42 29 502145 msys_1 88 96 41 29 505513 msys_1 88 96 46 29 510623 msys_1 88 96 41 29 515618 msys_1 88 96 38 25 517647 msys_1 88 96 37 25 536303 msys_1 88 96 37 25 537157 msys_1 88 96 38 25 559728 msys_1 88 96 36 23 593830 msys_1 88 96 38 23 597207 msys_1 88 96 39 23 609735 msys_1 88 96 38 23 614905 msys_1 88 96 38 23 617273 msys_1 88 96 33 23 617930 msys_1 88 96 31 23 618681 msys_1 88 96 32 23 619321 msys_1 88 96 31 23 627585 msys_1 88 96 32 23 629576 msys_1 88 96 37 21 632959 msys_1 88 96 40 21 638929 msys_1 88 96 35 21 639299 msys_1 88 96 39 21 643188 msys_1 88 96 36 21 644084 msys_1 88 96 37 21 645673 msys_1 88 96 40 21 649744 msys_1 88 96 31 21 650883 msys_1 88 96 34 21 651702 msys_1 88 96 39 21 661043 msys_1 88 96 35 21 704382 msys_1 88 96 21 12 751310 msys_1 88 96 21 2 757135 msys_1 88 96 15 2 765200 msys_1 88 96 15 2 826437 msys_1 88 96 17 2 827153 msys_1 88 96 19 2 836758 msys_1 88 96 18 2 837589 msys_1 88 96 19 2 841713 msys_1 88 96 20 2 899904 msys_1 88 96 18 2 973807 msys_1 88 96 17 2 981752 msys_1 88 96 20 2 994788 msys_1 88 96 19 2 997513 msys_1 88 96 16 2 1094420 msys_1 88 96 17 2 1112104 msys_1 88 96 20 2 1133801 msys_1 88 96 21 2 1303205 msys_1 88 96 21 2 1361897 msys_1 88 96 16 2 1374554 msys_1 88 96 21 2 1378502 msys_1 88 96 19 2 1392124 msys_1 88 96 17 2 1438385 msys_1 88 96 21 2 1441331 msys_1 88 96 21 2 1488757 msys_1 88 96 15 2 1491018 msys_1 88 96 20 2 1501561 msys_1 88 96 18 2 1735218 msys_1 88 96 13 2 On Sat, Feb 3, 2018 at 2:17 PM, Simon Ratner <[email protected]> wrote: > Hey devs, > > I am seeing the number of available ble_gattc_proc structs in > ble_gattc_proc_pool dwindling over time, suspect they are not always being > freed (see bottom of the email for mpool output). > > This seems to be happening under heavy client load, where a significant > fraction of the clients may be failing or timing out. The radio is > nrf52832, and the procs in use are: mtu, disc_svc_uuid, disc_chr_uuid, > disc_all_dscs, read, write, indicate_custom. There may be an mbuf leak > associated with this, since i see those dwindle too -- but i have not > completely eliminated application error for that one yet. > > I am trying to isolate the culprit, but in the meantime, if someone has an > idea as to which bit of code is most likely to be responsible, or ran into > something similar, any information would be appreciated. Scope of change is > 3742e67..eb3d264 (yes, i know that's a lot). > > Thanks! > > mpool output over 3 hours: > > ticks name blksz cnt free min > 037579 ble_gattc_proc_pool 56 32 32 27 > 062221 ble_gattc_proc_pool 56 32 32 27 > 068531 ble_gattc_proc_pool 56 32 32 27 > 084960 ble_gattc_proc_pool 56 32 32 27 > 087382 ble_gattc_proc_pool 56 32 31 27 > 108489 ble_gattc_proc_pool 56 32 32 27 > 128580 ble_gattc_proc_pool 56 32 32 27 > 136642 ble_gattc_proc_pool 56 32 32 27 > 147967 ble_gattc_proc_pool 56 32 31 27 > 148889 ble_gattc_proc_pool 56 32 32 27 > 161142 ble_gattc_proc_pool 56 32 31 27 > 162518 ble_gattc_proc_pool 56 32 32 27 > 173439 ble_gattc_proc_pool 56 32 31 27 > 173766 ble_gattc_proc_pool 56 32 31 27 > 263767 ble_gattc_proc_pool 56 32 26 22 > 264659 ble_gattc_proc_pool 56 32 27 22 > 265966 ble_gattc_proc_pool 56 32 26 22 > 266472 ble_gattc_proc_pool 56 32 26 22 > 270542 ble_gattc_proc_pool 56 32 27 22 > 274803 ble_gattc_proc_pool 56 32 25 22 > 286099 ble_gattc_proc_pool 56 32 25 22 > 286484 ble_gattc_proc_pool 56 32 23 22 > 297022 ble_gattc_proc_pool 56 32 23 22 > 304540 ble_gattc_proc_pool 56 32 25 20 > 309070 ble_gattc_proc_pool 56 32 25 20 > 318542 ble_gattc_proc_pool 56 32 26 20 > 359797 ble_gattc_proc_pool 56 32 16 11 > 361184 ble_gattc_proc_pool 56 32 16 11 > 364989 ble_gattc_proc_pool 56 32 16 11 > 367769 ble_gattc_proc_pool 56 32 15 11 > 371913 ble_gattc_proc_pool 56 32 16 11 > 384139 ble_gattc_proc_pool 56 32 14 11 > 384907 ble_gattc_proc_pool 56 32 14 11 > 388905 ble_gattc_proc_pool 56 32 15 11 > 396090 ble_gattc_proc_pool 56 32 15 11 > 405044 ble_gattc_proc_pool 56 32 14 11 > 405902 ble_gattc_proc_pool 56 32 15 11 > 408975 ble_gattc_proc_pool 56 32 15 11 > 417053 ble_gattc_proc_pool 56 32 14 11 > 422387 ble_gattc_proc_pool 56 32 13 11 > 428185 ble_gattc_proc_pool 56 32 15 11 > 429465 ble_gattc_proc_pool 56 32 14 11 > 440093 ble_gattc_proc_pool 56 32 13 11 > 440546 ble_gattc_proc_pool 56 32 14 11 > 448875 ble_gattc_proc_pool 56 32 13 10 > 450181 ble_gattc_proc_pool 56 32 12 10 > 459837 ble_gattc_proc_pool 56 32 13 10 > 468516 ble_gattc_proc_pool 56 32 14 10 > 476963 ble_gattc_proc_pool 56 32 12 10 > 477853 ble_gattc_proc_pool 56 32 14 10 > 480811 ble_gattc_proc_pool 56 32 12 10 > 486291 ble_gattc_proc_pool 56 32 14 10 > 487289 ble_gattc_proc_pool 56 32 14 10 > 492067 ble_gattc_proc_pool 56 32 14 10 > 498029 ble_gattc_proc_pool 56 32 14 10 > 501499 ble_gattc_proc_pool 56 32 12 10 > 502159 ble_gattc_proc_pool 56 32 14 10 > 505527 ble_gattc_proc_pool 56 32 14 10 > 510637 ble_gattc_proc_pool 56 32 14 10 > 515632 ble_gattc_proc_pool 56 32 14 8 > 517661 ble_gattc_proc_pool 56 32 13 8 > 536317 ble_gattc_proc_pool 56 32 11 8 > 537171 ble_gattc_proc_pool 56 32 11 8 > 559742 ble_gattc_proc_pool 56 32 10 6 > 593844 ble_gattc_proc_pool 56 32 9 5 > 597221 ble_gattc_proc_pool 56 32 10 5 > 609749 ble_gattc_proc_pool 56 32 10 5 > 614919 ble_gattc_proc_pool 56 32 10 5 > 617287 ble_gattc_proc_pool 56 32 9 5 > 617944 ble_gattc_proc_pool 56 32 9 5 > 618695 ble_gattc_proc_pool 56 32 9 5 > 619334 ble_gattc_proc_pool 56 32 7 5 > 627599 ble_gattc_proc_pool 56 32 9 5 > 629590 ble_gattc_proc_pool 56 32 8 5 > 632973 ble_gattc_proc_pool 56 32 9 5 > 638943 ble_gattc_proc_pool 56 32 8 5 > 639313 ble_gattc_proc_pool 56 32 9 5 > 643202 ble_gattc_proc_pool 56 32 9 5 > 644098 ble_gattc_proc_pool 56 32 8 5 > 645687 ble_gattc_proc_pool 56 32 9 5 > 649758 ble_gattc_proc_pool 56 32 8 5 > 650897 ble_gattc_proc_pool 56 32 8 4 > 651717 ble_gattc_proc_pool 56 32 9 4 > 661057 ble_gattc_proc_pool 56 32 8 4 > 704396 ble_gattc_proc_pool 56 32 6 3 > 751324 ble_gattc_proc_pool 56 32 2 0 > 757149 ble_gattc_proc_pool 56 32 0 0 > 765214 ble_gattc_proc_pool 56 32 1 0 > 826451 ble_gattc_proc_pool 56 32 0 0 > 827167 ble_gattc_proc_pool 56 32 1 0 > 836772 ble_gattc_proc_pool 56 32 1 0 > 837603 ble_gattc_proc_pool 56 32 1 0 > 841727 ble_gattc_proc_pool 56 32 1 0 > 899918 ble_gattc_proc_pool 56 32 1 0 > 973821 ble_gattc_proc_pool 56 32 1 0 > 981766 ble_gattc_proc_pool 56 32 0 0 > 994802 ble_gattc_proc_pool 56 32 1 0 > 997527 ble_gattc_proc_pool 56 32 0 0 > 1094434 ble_gattc_proc_pool 56 32 1 0 > 1112118 ble_gattc_proc_pool 56 32 1 0 > 1133815 ble_gattc_proc_pool 56 32 1 0 > 1303219 ble_gattc_proc_pool 56 32 1 0 > 1361911 ble_gattc_proc_pool 56 32 0 0 > 1374568 ble_gattc_proc_pool 56 32 1 0 > 1378516 ble_gattc_proc_pool 56 32 1 0 > 1392138 ble_gattc_proc_pool 56 32 0 0 > 1438400 ble_gattc_proc_pool 56 32 1 0 > 1441345 ble_gattc_proc_pool 56 32 1 0 > 1488771 ble_gattc_proc_pool 56 32 0 0 > 1491032 ble_gattc_proc_pool 56 32 1 0 > 1501575 ble_gattc_proc_pool 56 32 1 0 > >
