Re: [pmacct-discussion] nfacctd crash when using pre_tag_map
Hi Felix, Thanks for getting in touch. Can you please get more data about the crash by following this section fo the QUICKSTART (i'd need an output of GDB 'bt'): https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2606-#L2635 You can follow up 1:1 so that we don't disturb everybody with the back/forth that will be needed by the troubleshooting process. We can then summarize resolution on the list. Paolo On Mon, May 27, 2019 at 03:55:04PM +, Felix Stolba wrote: > Hi > > I’m trying to use a pre_tag_map with less than 5000 entries with the purpose > of adding the ingress and egress interface names to the flow records as > labels. When using this map, nfacctd reproducibly crashes, tested using 1.7.1 > and 1.7.3. I would appreciate if someone (Paolo? :) ) could help isolate the > problem. Debug logs can be found attached. I will be happy to provide any > additional info that will be needed. > > When crashing, nfacctd emits this log message: > realloc(): invalid next size > Aborted (core dumped) > > Few tests I've already done: > * Use a smaller map: works - did a PoC using a map of about 200 lines, this > worked great. > * Delete everything below OUTTABLE (see below): works - having only the top > part of the map keeps pmacct running > * Delete some lines below OUTTABLE - produced a different error message: > "corrupted size vs. prev_size" > > The pre_tag_map essentially looks like the ones in the JEQ examples [1]: > > set_label=INTERFACE_NAME ip=ROUTER_IP in=IFINDEX jeq=OUTTABLE > ... 2000 lines of similar mappings ... > set_label=INTERFACE_NAME ip=ROUTER_IP out=IFINDEX label=OUTTABLE > ... 2000 lines of similar mappings ... > > > Best regards > Felix > > [1] https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example > > > > flow01:~/pmacct-to-elasticsearch# nfacctd -f /etc/pmacct/pmacctd.conf -d > DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 'default'/'core'. > DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: > 'elasticsearch_print'/'print'. > DEBUG: [/etc/pmacct/pmacctd.conf] debug_internal_msg:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_time_new:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_account_options:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ip:0.0.0.0 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_port:4739 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_disable_opt_scope_check:true > DEBUG: [/etc/pmacct/pmacctd.conf] > nfacctd_templates_file:/etc/pmacct/nf_templates_cache > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_net:bmp > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_as:bmp > DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_as:false > DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_net:false > WARN: [/etc/pmacct/pmacctd.conf] Invalid network aggregation value 'false' > WARN: [/etc/pmacct/pmacctd.conf:18] Invalid value. Ignored. > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ext_sampling_rate:1024 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_renormalize:true > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon:true > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_ip:0.0.0.0 > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_max_peers:100 > DEBUG: [/etc/pmacct/pmacctd.conf] logfile:/var/log/pmacct/pmacctd.log > DEBUG: [/etc/pmacct/pmacctd.conf] > print_output_file[elasticsearch_print]:/elasticsearch_print.json > DEBUG: [/etc/pmacct/pmacctd.conf] print_output[elasticsearch_print]:json > DEBUG: [/etc/pmacct/pmacctd.conf] > print_trigger_exec[elasticsearch_print]:/etc/pmacct/p2es/triggers/elasticsearch_print > DEBUG: [/etc/pmacct/pmacctd.conf] print_refresh_time[elasticsearch_print]:15 > DEBUG: [/etc/pmacct/pmacctd.conf] aggregate[elasticsearch_print]:src_host, > dst_host, in_iface, out_iface, timestamp_start, timestamp_end, src_port, > dst_port, proto, tos, src_mask, dst_mask, tcpflags, etype, src_host_country, > dst_host_country, vlan, sampling_rate, tag, tag2, label, src_as, dst_as, > as_path, std_comm, ext_comm, lrg_comm, local_pref, med, src_as_path, > src_std_comm, src_ext_comm, src_lrg_comm, src_local_pref, src_med, > mpls_vpn_rd, peer_src_as, peer_dst_as, peer_dst_ip, peer_src_ip, src_roa, > dst_roa, src_net, dst_net > DEBUG: [/etc/pmacct/pmacctd.conf] geoipv2_file:/etc/pmacct/GeoLite2-City.mmdb > DEBUG: [/etc/pmacct/pmacctd.conf] pre_tag_map:/etc/pmacct/ifindex.map > DEBUG: [/etc/pmacct/pmacctd.conf] maps_refresh:true > DEBUG: [/etc/pmacct/pmacctd.conf] maps_entries:64000 > DEBUG: [/etc/pmacct/pmacctd.conf] maps_index:true > DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache:rpki01:8282 > DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache_version:0 > DEBUG: [/etc/pmacct/pmacctd.conf] debug:true > realloc(): invalid next size > Aborted (core dumped) > 2019-05-27T06:59:50Z INFO ( default/core/BMP ): waiting for BMP data on > 0.0.0.0:1790 > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): > plugin_pipe_size=4096000 bytes plugin_buffer_size=1548 bytes > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): ctrl channel: > obtained=
[pmacct-discussion] nfacctd crash when using pre_tag_map
Hi I’m trying to use a pre_tag_map with less than 5000 entries with the purpose of adding the ingress and egress interface names to the flow records as labels. When using this map, nfacctd reproducibly crashes, tested using 1.7.1 and 1.7.3. I would appreciate if someone (Paolo? :) ) could help isolate the problem. Debug logs can be found attached. I will be happy to provide any additional info that will be needed. When crashing, nfacctd emits this log message: realloc(): invalid next size Aborted (core dumped) Few tests I've already done: * Use a smaller map: works - did a PoC using a map of about 200 lines, this worked great. * Delete everything below OUTTABLE (see below): works - having only the top part of the map keeps pmacct running * Delete some lines below OUTTABLE - produced a different error message: "corrupted size vs. prev_size" The pre_tag_map essentially looks like the ones in the JEQ examples [1]: set_label=INTERFACE_NAME ip=ROUTER_IP in=IFINDEX jeq=OUTTABLE ... 2000 lines of similar mappings ... set_label=INTERFACE_NAME ip=ROUTER_IP out=IFINDEX label=OUTTABLE ... 2000 lines of similar mappings ... Best regards Felix [1] https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example flow01:~/pmacct-to-elasticsearch# nfacctd -f /etc/pmacct/pmacctd.conf -d DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 'default'/'core'. DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 'elasticsearch_print'/'print'. DEBUG: [/etc/pmacct/pmacctd.conf] debug_internal_msg:true DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_time_new:true DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_account_options:true DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ip:0.0.0.0 DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_port:4739 DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_disable_opt_scope_check:true DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_templates_file:/etc/pmacct/nf_templates_cache DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_net:bmp DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_as:bmp DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_as:false DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_net:false WARN: [/etc/pmacct/pmacctd.conf] Invalid network aggregation value 'false' WARN: [/etc/pmacct/pmacctd.conf:18] Invalid value. Ignored. DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ext_sampling_rate:1024 DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_renormalize:true DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon:true DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_ip:0.0.0.0 DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_max_peers:100 DEBUG: [/etc/pmacct/pmacctd.conf] logfile:/var/log/pmacct/pmacctd.log DEBUG: [/etc/pmacct/pmacctd.conf] print_output_file[elasticsearch_print]:/elasticsearch_print.json DEBUG: [/etc/pmacct/pmacctd.conf] print_output[elasticsearch_print]:json DEBUG: [/etc/pmacct/pmacctd.conf] print_trigger_exec[elasticsearch_print]:/etc/pmacct/p2es/triggers/elasticsearch_print DEBUG: [/etc/pmacct/pmacctd.conf] print_refresh_time[elasticsearch_print]:15 DEBUG: [/etc/pmacct/pmacctd.conf] aggregate[elasticsearch_print]:src_host, dst_host, in_iface, out_iface, timestamp_start, timestamp_end, src_port, dst_port, proto, tos, src_mask, dst_mask, tcpflags, etype, src_host_country, dst_host_country, vlan, sampling_rate, tag, tag2, label, src_as, dst_as, as_path, std_comm, ext_comm, lrg_comm, local_pref, med, src_as_path, src_std_comm, src_ext_comm, src_lrg_comm, src_local_pref, src_med, mpls_vpn_rd, peer_src_as, peer_dst_as, peer_dst_ip, peer_src_ip, src_roa, dst_roa, src_net, dst_net DEBUG: [/etc/pmacct/pmacctd.conf] geoipv2_file:/etc/pmacct/GeoLite2-City.mmdb DEBUG: [/etc/pmacct/pmacctd.conf] pre_tag_map:/etc/pmacct/ifindex.map DEBUG: [/etc/pmacct/pmacctd.conf] maps_refresh:true DEBUG: [/etc/pmacct/pmacctd.conf] maps_entries:64000 DEBUG: [/etc/pmacct/pmacctd.conf] maps_index:true DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache:rpki01:8282 DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache_version:0 DEBUG: [/etc/pmacct/pmacctd.conf] debug:true realloc(): invalid next size Aborted (core dumped)2019-05-27T06:59:50Z INFO ( default/core/BMP ): waiting for BMP data on 0.0.0.0:1790 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): plugin_pipe_size=4096000 bytes plugin_buffer_size=1548 bytes 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): ctrl channel: obtained=212992 bytes target=21160 bytes 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] (re)loading map. 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): cache entries=16411 base cache memory=66431728 bytes 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): JSON: setting object handlers. 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] maps_index: created index 4001 (2212 entries). 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] maps_index: created index 4000 (1 entries). 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] maps_index: created index 4
