The easier way would be to get the memory usage summary, it will give a report ordered by the size of used memory based on code location, allowing to assert if it is some startup (allocated once per lifetime of kamailio) or runtime.
Some hints about how to do it: * https://www.kamailio.org/wiki/tutorials/troubleshooting/memory Cheers, Daniel On 03.10.18 08:48, Igor Olhovskiy wrote: > Thanks Daniel > > But how to find out what exact eats all memory? > > pkg.stats > { > entry: 0 > pid: 13 > rank: 0 > used: 3639400 > free: 3323464 > real_used: 5065144 > total_size: 8388608 > total_frags: 38 > } > { > entry: 1 > pid: 14 > rank: 1 > used: 3672088 > free: 3253128 > real_used: 5135480 > total_size: 8388608 > total_frags: 125 > } > { > entry: 2 > pid: 15 > rank: 2 > used: 3675992 > free: 3246312 > real_used: 5142296 > total_size: 8388608 > total_frags: 121 > } > { > entry: 3 > pid: 16 > rank: 3 > used: 3670912 > free: 3254512 > real_used: 5134096 > total_size: 8388608 > total_frags: 129 > } > { > entry: 4 > pid: 17 > rank: 4 > used: 3673712 > free: 3250152 > real_used: 5138456 > total_size: 8388608 > total_frags: 122 > } > { > entry: 5 > pid: 18 > rank: 5 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 6 > pid: 19 > rank: 6 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 7 > pid: 20 > rank: 7 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 8 > pid: 21 > rank: 8 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 9 > pid: 22 > rank: 9 > used: 3643776 > free: 3315032 > real_used: 5073576 > total_size: 8388608 > total_frags: 54 > } > { > entry: 10 > pid: 23 > rank: 10 > used: 3643776 > free: 3315032 > real_used: 5073576 > total_size: 8388608 > total_frags: 54 > } > { > entry: 11 > pid: 24 > rank: 11 > used: 3643776 > free: 3315032 > real_used: 5073576 > total_size: 8388608 > total_frags: 54 > } > { > entry: 12 > pid: 25 > rank: 12 > used: 3643776 > free: 3315032 > real_used: 5073576 > total_size: 8388608 > total_frags: 54 > } > { > entry: 13 > pid: 26 > rank: 13 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 14 > pid: 27 > rank: 14 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 15 > pid: 28 > rank: 15 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 16 > pid: 29 > rank: 16 > used: 3643776 > free: 3318152 > real_used: 5070456 > total_size: 8388608 > total_frags: 24 > } > { > entry: 17 > pid: 30 > rank: -1 > used: 3645488 > free: 3308328 > real_used: 5080280 > total_size: 8388608 > total_frags: 89 > } > { > entry: 18 > pid: 31 > rank: -1 > used: 3645920 > free: 3303528 > real_used: 5085080 > total_size: 8388608 > total_frags: 122 > } > { > entry: 19 > pid: 32 > rank: -1 > used: 3643600 > free: 3318536 > real_used: 5070072 > total_size: 8388608 > total_frags: 24 > } > { > entry: 20 > pid: 33 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 21 > pid: 34 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 22 > pid: 35 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 23 > pid: 36 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 24 > pid: 37 > rank: -128 > used: 0 > free: 0 > real_used: 0 > total_size: 8388608 > total_frags: 0 > } > { > entry: 25 > pid: 38 > rank: -128 > used: 0 > free: 0 > real_used: 0 > total_size: 8388608 > total_frags: 0 > } > { > entry: 26 > pid: 39 > rank: -128 > used: 0 > free: 0 > real_used: 0 > total_size: 8388608 > total_frags: 0 > } > { > entry: 27 > pid: 40 > rank: -128 > used: 0 > free: 0 > real_used: 0 > total_size: 8388608 > total_frags: 0 > } > { > entry: 28 > pid: 41 > rank: -2 > used: 3657016 > free: 3304912 > real_used: 5083696 > total_size: 8388608 > total_frags: 23 > } > { > entry: 29 > pid: 42 > rank: -2 > used: 5810992 > free: 3816 > real_used: 8384792 > total_size: 8388608 > total_frags: 93 > } > { > entry: 30 > pid: 43 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 31 > pid: 44 > rank: -2 > used: 3643744 > free: 3318288 > real_used: 5070320 > total_size: 8388608 > total_frags: 24 > } > { > entry: 32 > pid: 45 > rank: -1 > used: 3643600 > free: 3318536 > real_used: 5070072 > total_size: 8388608 > total_frags: 24 > } > { > entry: 33 > pid: 46 > rank: -1 > used: 3643600 > free: 3318536 > real_used: 5070072 > total_size: 8388608 > total_frags: 24 > } > { > entry: 34 > pid: 47 > rank: 17 > used: 3725080 > free: 3234768 > real_used: 5153840 > total_size: 8388608 > total_frags: 39 > } > { > entry: 35 > pid: 48 > rank: 18 > used: 3725064 > free: 3234992 > real_used: 5153616 > total_size: 8388608 > total_frags: 38 > } > { > entry: 36 > pid: 49 > rank: 19 > used: 3725048 > free: 3235112 > real_used: 5153496 > total_size: 8388608 > total_frags: 37 > } > { > entry: 37 > pid: 50 > rank: 20 > used: 3725048 > free: 3234800 > real_used: 5153808 > total_size: 8388608 > total_frags: 40 > } > { > entry: 38 > pid: 51 > rank: -4 > used: 3723728 > free: 3238824 > real_used: 5149784 > total_size: 8388608 > total_frags: 37 > } > > Regards, Igor > On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla > <[email protected]>, wrote: >> >> Hello, >> >> pkg is out of free size, not shm. See the output for: >> >> kamctl rpc pkg.stats >> >> You may need to increase the pkg allocated pool with -M command line >> parameter. >> >> Cheers, >> Daniel >> >> >> On 02.10.18 20:02, Igor Olhovskiy wrote: >>> Hi! >>> >>> What is the best way to debug messages like... >>> >>> 29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): >>> qm_find_free(0x7fad301cf010, 56); Free fragment not found! >>> 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): >>> qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: >>> dup_lump_list_r(537), module: core; Free fragment not found! >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem >>> 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add >>> branches >>> >>> Cause core.shmmem shows >>> total: 536870912 >>> free: 522859104 >>> used: 12224144 >>> real_used: 14011808 >>> max_used: 14479048 >>> fragments: 10560 >>> >>> >>> Also >>> >>> mod.stats all pkg >>> Module: core >>> { >>> init_io_wait(475): 6960 >>> init_io_wait(524): 6312 >>> new_db_id(272): 512 >>> dupl_string_name(72): 32 >>> db_do_init2(298): 488 >>> mk_switch_cond_table(43): 144 >>> sr_wtimer_add(330): 48 >>> as_asciiz(161): 24 >>> init_modules(1026): 16 >>> rpc_hash_add(146): 2048 >>> db_allocate_columns(150): 32 >>> db_new_result(114): 56 >>> get_hdr_field(116): 1160 >>> db_allocate_columns(160): 16 >>> cnt_hash_add(383): 1024 >>> register_select_table(458): 48 >>> add_callback(59): 320 >>> sr_wtimer_init(308): 136 >>> cfg_new_group(79): 704 >>> cfg_declare(50): 3360 >>> fix_sock_str(420): 480 >>> init_dst_set(83): 32208 >>> grp_hash_add(234): 808 >>> fix_hostname(1301): 152 >>> add_alias(91): 464 >>> add_alias(93): 256 >>> dupl_string(48): 176 >>> parse_headers(325): 432 >>> subst_parser(291): 144 >>> subst_parser(274): 64 >>> exp_optimize_right(411): 24 >>> fix_expr(541): 600 >>> mk_case_stm(3775): 400 >>> fixup_regexp_null(213): 72 >>> mk_elem(90): 560 >>> route_new_list(200): 1344 >>> fix_param(1196): 1336 >>> parse_select(204): 1488 >>> fix_param(1257): 536 >>> fix_match_rve(3028): 320 >>> mk_rval_expr1(2616): 142656 >>> tr_new(1646): 1464 >>> fixup_pvar_all(279): 1624 >>> fix_param(1162): 16144 >>> mk_action(118): 350168 >>> mk_rval_expr2(2674): 489552 >>> mk_rval_expr_v(2537): 1475600 >>> yyparse(2757): 8792 >>> pv_parse_format(1150): 89064 >>> pv_cache_add(347): 31456 >>> parse_params2(585): 1400 >>> set_mod_param_regex(110): 376 >>> set_mod_param_regex(121): 832 >>> tr_table_add(1875): 400 >>> sr_cmd_exports_convert(235): 36288 >>> register_module(280): 6624 >>> load_module(496): 2640 >>> new_sock_info(235): 224 >>> fix_socket_list(1509): 48 >>> new_sock_info(230): 3920 >>> add_alias(93): 56 >>> add_alias(91): 96 >>> yyparse(1774): 64 >>> yyparse(667): 152 >>> yyparse(2306): 128 >>> yyparse(648): 144 >>> sr_push_yy_state(1666): 32 >>> sr_push_yy_state(1656): 104 >>> sr_push_yy_state(1596): 184 >>> addstr(1413): 415752 >>> rpc_hash_add(100): 25656 >>> str_hash_alloc(59): 512 >>> pv_table_add(236): 17864 >>> pv_init_buffer(2055): 327680 >>> pv_init_buffer(2045): 320 >>> init_nonsip_hooks(43): 24 >>> init_rlist(146): 8 >>> mk_rval_expr_v(2548): 1264 >>> route_add(124): 5872 >>> str_hash_alloc(59): 768 >>> rval_get_str(1256): 56632 >>> pp_define(1788): 2248 >>> init_counters(122): 128 >>> cnt_hash_add(332): 20608 >>> str_hash_alloc(59): 1280 >>> Total: 3602152 >>> } >>> >>> Module: rtpengine >>> { >>> child_init(1812): 256 >>> Total: 256 >>> } >>> >>> Module: db_mysql >>> { >>> db_mysql_new_connection(65): 96 >>> db_mysql_new_connection(75): 3040 >>> Total: 3136 >>> } >>> >>> Module: siptrace >>> { >>> mod_init(344): 384 >>> Total: 384 >>> } >>> >>> Module: sl >>> { >>> sl_register_callback(460): 88 >>> Total: 88 >>> } >>> >>> Module: acc >>> { >>> parse_acc_extra(116): 1512 >>> Total: 1512 >>> } >>> >>> Module: dialog >>> { >>> dlg_bridge_init_hdrs(66): 80 >>> Total: 80 >>> } >>> >>> Module: rr >>> { >>> register_rrcb(61): 80 >>> Total: 80 >>> } >>> >>> Module: db_text >>> { >>> dbt_get_columns(67): 64 >>> Total: 64 >>> } >>> >>> Module: permissions >>> { >>> get_pathname(242): 64 >>> Total: 64 >>> } >>> >>> Module: auth >>> { >>> generate_random_secret(239): 32 >>> generate_random_secret(238): 32 >>> Total: 64 >>> } >>> >>> Module: xlog >>> { >>> mod_init(214): 4104 >>> xdbg_fixup_helper(500): 6032 >>> xlog_fixup_helper(535): 26992 >>> Total: 37128 >>> } >>> >>> Module: dmq >>> { >>> make_socket_str_from_uri(149): 24 >>> Total: 24 >>> } >>> >>> Module: sqlops >>> { >>> sqlops_tr_buffer_init(46): 2048 >>> sql_init_con(83): 240 >>> Total: 2288 >>> } >>> >>> Module: textops >>> { >>> tr_txt_parse_re(212): 32 >>> hname_fixup(3217): 768 >>> fixup_method(3287): 1384 >>> Total: 2184 >>> } >>> >>> Module: pv >>> { >>> tr_parse_string(2343): 120 >>> tr_parse_string(2332): 32 >>> tr_parse_string(2378): 96 >>> tr_parse_string(2368): 184 >>> tr_parse_string(2544): 576 >>> add_var(58): 3616 >>> add_var(65): 1112 >>> Total: 5736 >>> } >>> >>> Module: textopsx >>> { >>> fixup_hname_param(598): 352 >>> Total: 352 >>> } >>> >>> Module: avpops >>> { >>> avpops_parse_pvar(57): 416 >>> Total: 416 >>> } >>> >>> Module: userblacklist >>> { >>> check_blacklist_fixup(522): 16 >>> Total: 16 >>> } >>> >>> Module: htable >>> { >>> fixup_ht_key(348): 176 >>> pv_parse_ht_name(158): 520 >>> Total: 696 >>> } >>> >>> Module: dialplan >>> { >>> dp_trans_fixup(396): 296 >>> Total: 296 >>> } >>> >>> Module: dispatcher >>> { >>> Total: 0 >>> } >>> >>> Module: tmx >>> { >>> Total: 0 >>> } >>> >>> Module: tm >>> { >>> Total: 0 >>> } >>> >>> Module: kex >>> { >>> Total: 0 >>> } >>> >>> Module: usrloc >>> { >>> Total: 0 >>> } >>> >>> Module: topos >>> { >>> Total: 0 >>> } >>> >>> Module: utils >>> { >>> Total: 0 >>> } >>> >>> Module: app_python >>> { >>> Total: 0 >>> } >>> >>> Module: cfgutils >>> { >>> Total: 0 >>> } >>> >>> Module: nathelper >>> { >>> Total: 0 >>> } >>> >>> Where is the place to look at? Means what module can cause this? >>> >>> Regards, Igor >>> >>> >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing List >>> [email protected] >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Daniel-Constantin Mierla -- www.asipto.com >> www.twitter.com/miconda -- www.linkedin.com/in/miconda >> Kamailio World Conference -- www.kamailioworld.com >> Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com > > > _______________________________________________ > Kamailio (SER) - Users Mailing List > [email protected] > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference -- www.kamailioworld.com Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
