Hi Simon, I had a look into the code and could not find any obvious mistake. Would it be possible for you to check what is a opcode of the outstanding proc? Also are you running with BLE_HS_DEBUG=1 ? There are some asserts which would help to analyze.
Best Łukasz On 5 February 2018 at 11:36, marko kiiskila <[email protected]> wrote: > Hi Simon, > > maybe this would be the time to add tracking PC of the code to > call alloc() or free() for a memory block? This does not always > work that great, as often code centralizes the function where allocations > happen. Therefore we would augment this with a another call > which allows further ‘stamping’ of the data structure. > > I’d make this work is by adding another 32-bit empty space in front of > the allocated memory (taking that into account in > OS_MEM_TRUE_BLOCK_SIZE()). Then stuff that field with > __builtin_return_address(0) in os_memblock_get() and os_memblock_put(). > And then the stamper() would overwrite it with current PC value if called > (could be a function call so that we could use __builtin_XXX) there as > well. > > This could then later be extended to add guard areas between memory > blocks (if memory corruption of adjacent memory blocks ever becomes > an issue for somebody). > > > On Feb 4, 2018, at 12:27 AM, Simon Ratner <[email protected]> wrote: > > > > Actually, looking at the mbuf situation, the step drops in available > mbufs > > correlate to the step drops in ble_gattc_proc structs (ts=263753, 359783, > > 751324, for example) so I am going to call that strong evidence that both > > leaks are caused by the same process: > > > > 037565 msys_1 88 96 95 80 > > 062207 msys_1 88 96 95 80 > > 068517 msys_1 88 96 95 80 > > 084946 msys_1 88 96 95 80 > > 087368 msys_1 88 96 94 80 > > 108475 msys_1 88 96 95 80 > > 128566 msys_1 88 96 95 80 > > 136628 msys_1 88 96 94 80 > > 147953 msys_1 88 96 92 80 > > 148875 msys_1 88 96 94 80 > > 161128 msys_1 88 96 92 80 > > 162504 msys_1 88 96 95 80 > > 173425 msys_1 88 96 92 80 > > 173752 msys_1 88 96 93 80 > > 263753 msys_1 88 96 72 60 > > 264645 msys_1 88 96 75 60 > > 265952 msys_1 88 96 72 60 > > 266458 msys_1 88 96 71 60 > > 270528 msys_1 88 96 70 60 > > 274789 msys_1 88 96 68 60 > > 286085 msys_1 88 96 67 59 > > 286470 msys_1 88 96 66 59 > > 297008 msys_1 88 96 70 58 > > 304526 msys_1 88 96 63 53 > > 309056 msys_1 88 96 65 53 > > 318528 msys_1 88 96 68 53 > > 359783 msys_1 88 96 47 33 > > 361170 msys_1 88 96 50 33 > > 364975 msys_1 88 96 50 33 > > 367755 msys_1 88 96 48 33 > > 371899 msys_1 88 96 50 33 > > 384125 msys_1 88 96 40 33 > > 384893 msys_1 88 96 39 33 > > 388891 msys_1 88 96 45 33 > > 396076 msys_1 88 96 47 33 > > 405030 msys_1 88 96 49 33 > > 405888 msys_1 88 96 49 33 > > 408961 msys_1 88 96 46 33 > > 417039 msys_1 88 96 50 33 > > 422373 msys_1 88 96 42 33 > > 428171 msys_1 88 96 48 33 > > 429451 msys_1 88 96 47 33 > > 440079 msys_1 88 96 41 33 > > 440532 msys_1 88 96 42 33 > > 448861 msys_1 88 96 45 31 > > 450167 msys_1 88 96 47 31 > > 459823 msys_1 88 96 40 31 > > 468502 msys_1 88 96 46 31 > > 476949 msys_1 88 96 38 31 > > 477839 msys_1 88 96 38 31 > > 480797 msys_1 88 96 39 29 > > 486277 msys_1 88 96 42 29 > > 487275 msys_1 88 96 43 29 > > 492053 msys_1 88 96 49 29 > > 498015 msys_1 88 96 47 29 > > 501485 msys_1 88 96 42 29 > > 502145 msys_1 88 96 41 29 > > 505513 msys_1 88 96 46 29 > > 510623 msys_1 88 96 41 29 > > 515618 msys_1 88 96 38 25 > > 517647 msys_1 88 96 37 25 > > 536303 msys_1 88 96 37 25 > > 537157 msys_1 88 96 38 25 > > 559728 msys_1 88 96 36 23 > > 593830 msys_1 88 96 38 23 > > 597207 msys_1 88 96 39 23 > > 609735 msys_1 88 96 38 23 > > 614905 msys_1 88 96 38 23 > > 617273 msys_1 88 96 33 23 > > 617930 msys_1 88 96 31 23 > > 618681 msys_1 88 96 32 23 > > 619321 msys_1 88 96 31 23 > > 627585 msys_1 88 96 32 23 > > 629576 msys_1 88 96 37 21 > > 632959 msys_1 88 96 40 21 > > 638929 msys_1 88 96 35 21 > > 639299 msys_1 88 96 39 21 > > 643188 msys_1 88 96 36 21 > > 644084 msys_1 88 96 37 21 > > 645673 msys_1 88 96 40 21 > > 649744 msys_1 88 96 31 21 > > 650883 msys_1 88 96 34 21 > > 651702 msys_1 88 96 39 21 > > 661043 msys_1 88 96 35 21 > > 704382 msys_1 88 96 21 12 > > 751310 msys_1 88 96 21 2 > > 757135 msys_1 88 96 15 2 > > 765200 msys_1 88 96 15 2 > > 826437 msys_1 88 96 17 2 > > 827153 msys_1 88 96 19 2 > > 836758 msys_1 88 96 18 2 > > 837589 msys_1 88 96 19 2 > > 841713 msys_1 88 96 20 2 > > 899904 msys_1 88 96 18 2 > > 973807 msys_1 88 96 17 2 > > 981752 msys_1 88 96 20 2 > > 994788 msys_1 88 96 19 2 > > 997513 msys_1 88 96 16 2 > > 1094420 msys_1 88 96 17 2 > > 1112104 msys_1 88 96 20 2 > > 1133801 msys_1 88 96 21 2 > > 1303205 msys_1 88 96 21 2 > > 1361897 msys_1 88 96 16 2 > > 1374554 msys_1 88 96 21 2 > > 1378502 msys_1 88 96 19 2 > > 1392124 msys_1 88 96 17 2 > > 1438385 msys_1 88 96 21 2 > > 1441331 msys_1 88 96 21 2 > > 1488757 msys_1 88 96 15 2 > > 1491018 msys_1 88 96 20 2 > > 1501561 msys_1 88 96 18 2 > > 1735218 msys_1 88 96 13 2 > > > > > > On Sat, Feb 3, 2018 at 2:17 PM, Simon Ratner <[email protected]> wrote: > > > >> Hey devs, > >> > >> I am seeing the number of available ble_gattc_proc structs in > >> ble_gattc_proc_pool dwindling over time, suspect they are not always > being > >> freed (see bottom of the email for mpool output). > >> > >> This seems to be happening under heavy client load, where a significant > >> fraction of the clients may be failing or timing out. The radio is > >> nrf52832, and the procs in use are: mtu, disc_svc_uuid, disc_chr_uuid, > >> disc_all_dscs, read, write, indicate_custom. There may be an mbuf leak > >> associated with this, since i see those dwindle too -- but i have not > >> completely eliminated application error for that one yet. > >> > >> I am trying to isolate the culprit, but in the meantime, if someone has > an > >> idea as to which bit of code is most likely to be responsible, or ran > into > >> something similar, any information would be appreciated. Scope of > change is > >> 3742e67..eb3d264 (yes, i know that's a lot). > >> > >> Thanks! > >> > >> mpool output over 3 hours: > >> > >> ticks name blksz cnt free min > >> 037579 ble_gattc_proc_pool 56 32 32 27 > >> 062221 ble_gattc_proc_pool 56 32 32 27 > >> 068531 ble_gattc_proc_pool 56 32 32 27 > >> 084960 ble_gattc_proc_pool 56 32 32 27 > >> 087382 ble_gattc_proc_pool 56 32 31 27 > >> 108489 ble_gattc_proc_pool 56 32 32 27 > >> 128580 ble_gattc_proc_pool 56 32 32 27 > >> 136642 ble_gattc_proc_pool 56 32 32 27 > >> 147967 ble_gattc_proc_pool 56 32 31 27 > >> 148889 ble_gattc_proc_pool 56 32 32 27 > >> 161142 ble_gattc_proc_pool 56 32 31 27 > >> 162518 ble_gattc_proc_pool 56 32 32 27 > >> 173439 ble_gattc_proc_pool 56 32 31 27 > >> 173766 ble_gattc_proc_pool 56 32 31 27 > >> 263767 ble_gattc_proc_pool 56 32 26 22 > >> 264659 ble_gattc_proc_pool 56 32 27 22 > >> 265966 ble_gattc_proc_pool 56 32 26 22 > >> 266472 ble_gattc_proc_pool 56 32 26 22 > >> 270542 ble_gattc_proc_pool 56 32 27 22 > >> 274803 ble_gattc_proc_pool 56 32 25 22 > >> 286099 ble_gattc_proc_pool 56 32 25 22 > >> 286484 ble_gattc_proc_pool 56 32 23 22 > >> 297022 ble_gattc_proc_pool 56 32 23 22 > >> 304540 ble_gattc_proc_pool 56 32 25 20 > >> 309070 ble_gattc_proc_pool 56 32 25 20 > >> 318542 ble_gattc_proc_pool 56 32 26 20 > >> 359797 ble_gattc_proc_pool 56 32 16 11 > >> 361184 ble_gattc_proc_pool 56 32 16 11 > >> 364989 ble_gattc_proc_pool 56 32 16 11 > >> 367769 ble_gattc_proc_pool 56 32 15 11 > >> 371913 ble_gattc_proc_pool 56 32 16 11 > >> 384139 ble_gattc_proc_pool 56 32 14 11 > >> 384907 ble_gattc_proc_pool 56 32 14 11 > >> 388905 ble_gattc_proc_pool 56 32 15 11 > >> 396090 ble_gattc_proc_pool 56 32 15 11 > >> 405044 ble_gattc_proc_pool 56 32 14 11 > >> 405902 ble_gattc_proc_pool 56 32 15 11 > >> 408975 ble_gattc_proc_pool 56 32 15 11 > >> 417053 ble_gattc_proc_pool 56 32 14 11 > >> 422387 ble_gattc_proc_pool 56 32 13 11 > >> 428185 ble_gattc_proc_pool 56 32 15 11 > >> 429465 ble_gattc_proc_pool 56 32 14 11 > >> 440093 ble_gattc_proc_pool 56 32 13 11 > >> 440546 ble_gattc_proc_pool 56 32 14 11 > >> 448875 ble_gattc_proc_pool 56 32 13 10 > >> 450181 ble_gattc_proc_pool 56 32 12 10 > >> 459837 ble_gattc_proc_pool 56 32 13 10 > >> 468516 ble_gattc_proc_pool 56 32 14 10 > >> 476963 ble_gattc_proc_pool 56 32 12 10 > >> 477853 ble_gattc_proc_pool 56 32 14 10 > >> 480811 ble_gattc_proc_pool 56 32 12 10 > >> 486291 ble_gattc_proc_pool 56 32 14 10 > >> 487289 ble_gattc_proc_pool 56 32 14 10 > >> 492067 ble_gattc_proc_pool 56 32 14 10 > >> 498029 ble_gattc_proc_pool 56 32 14 10 > >> 501499 ble_gattc_proc_pool 56 32 12 10 > >> 502159 ble_gattc_proc_pool 56 32 14 10 > >> 505527 ble_gattc_proc_pool 56 32 14 10 > >> 510637 ble_gattc_proc_pool 56 32 14 10 > >> 515632 ble_gattc_proc_pool 56 32 14 8 > >> 517661 ble_gattc_proc_pool 56 32 13 8 > >> 536317 ble_gattc_proc_pool 56 32 11 8 > >> 537171 ble_gattc_proc_pool 56 32 11 8 > >> 559742 ble_gattc_proc_pool 56 32 10 6 > >> 593844 ble_gattc_proc_pool 56 32 9 5 > >> 597221 ble_gattc_proc_pool 56 32 10 5 > >> 609749 ble_gattc_proc_pool 56 32 10 5 > >> 614919 ble_gattc_proc_pool 56 32 10 5 > >> 617287 ble_gattc_proc_pool 56 32 9 5 > >> 617944 ble_gattc_proc_pool 56 32 9 5 > >> 618695 ble_gattc_proc_pool 56 32 9 5 > >> 619334 ble_gattc_proc_pool 56 32 7 5 > >> 627599 ble_gattc_proc_pool 56 32 9 5 > >> 629590 ble_gattc_proc_pool 56 32 8 5 > >> 632973 ble_gattc_proc_pool 56 32 9 5 > >> 638943 ble_gattc_proc_pool 56 32 8 5 > >> 639313 ble_gattc_proc_pool 56 32 9 5 > >> 643202 ble_gattc_proc_pool 56 32 9 5 > >> 644098 ble_gattc_proc_pool 56 32 8 5 > >> 645687 ble_gattc_proc_pool 56 32 9 5 > >> 649758 ble_gattc_proc_pool 56 32 8 5 > >> 650897 ble_gattc_proc_pool 56 32 8 4 > >> 651717 ble_gattc_proc_pool 56 32 9 4 > >> 661057 ble_gattc_proc_pool 56 32 8 4 > >> 704396 ble_gattc_proc_pool 56 32 6 3 > >> 751324 ble_gattc_proc_pool 56 32 2 0 > >> 757149 ble_gattc_proc_pool 56 32 0 0 > >> 765214 ble_gattc_proc_pool 56 32 1 0 > >> 826451 ble_gattc_proc_pool 56 32 0 0 > >> 827167 ble_gattc_proc_pool 56 32 1 0 > >> 836772 ble_gattc_proc_pool 56 32 1 0 > >> 837603 ble_gattc_proc_pool 56 32 1 0 > >> 841727 ble_gattc_proc_pool 56 32 1 0 > >> 899918 ble_gattc_proc_pool 56 32 1 0 > >> 973821 ble_gattc_proc_pool 56 32 1 0 > >> 981766 ble_gattc_proc_pool 56 32 0 0 > >> 994802 ble_gattc_proc_pool 56 32 1 0 > >> 997527 ble_gattc_proc_pool 56 32 0 0 > >> 1094434 ble_gattc_proc_pool 56 32 1 0 > >> 1112118 ble_gattc_proc_pool 56 32 1 0 > >> 1133815 ble_gattc_proc_pool 56 32 1 0 > >> 1303219 ble_gattc_proc_pool 56 32 1 0 > >> 1361911 ble_gattc_proc_pool 56 32 0 0 > >> 1374568 ble_gattc_proc_pool 56 32 1 0 > >> 1378516 ble_gattc_proc_pool 56 32 1 0 > >> 1392138 ble_gattc_proc_pool 56 32 0 0 > >> 1438400 ble_gattc_proc_pool 56 32 1 0 > >> 1441345 ble_gattc_proc_pool 56 32 1 0 > >> 1488771 ble_gattc_proc_pool 56 32 0 0 > >> 1491032 ble_gattc_proc_pool 56 32 1 0 > >> 1501575 ble_gattc_proc_pool 56 32 1 0 > >> > >> > >
