Hi Felix, Thanks for getting in touch. Can you please get more data about the crash by following this section fo the QUICKSTART (i'd need an output of GDB 'bt'):
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2606-#L2635 You can follow up 1:1 so that we don't disturb everybody with the back/forth that will be needed by the troubleshooting process. We can then summarize resolution on the list. Paolo On Mon, May 27, 2019 at 03:55:04PM +0000, Felix Stolba wrote: > Hi > > I’m trying to use a pre_tag_map with less than 5000 entries with the purpose > of adding the ingress and egress interface names to the flow records as > labels. When using this map, nfacctd reproducibly crashes, tested using 1.7.1 > and 1.7.3. I would appreciate if someone (Paolo? :) ) could help isolate the > problem. Debug logs can be found attached. I will be happy to provide any > additional info that will be needed. > > When crashing, nfacctd emits this log message: > realloc(): invalid next size > Aborted (core dumped) > > Few tests I've already done: > * Use a smaller map: works - did a PoC using a map of about 200 lines, this > worked great. > * Delete everything below OUTTABLE (see below): works - having only the top > part of the map keeps pmacct running > * Delete some lines below OUTTABLE - produced a different error message: > "corrupted size vs. prev_size" > > The pre_tag_map essentially looks like the ones in the JEQ examples [1]: > > set_label=INTERFACE_NAME ip=ROUTER_IP in=IFINDEX jeq=OUTTABLE > ... 2000 lines of similar mappings ... > set_label=INTERFACE_NAME ip=ROUTER_IP out=IFINDEX label=OUTTABLE > ... 2000 lines of similar mappings ... > > > Best regards > Felix > > [1] https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example > > > > flow01:~/pmacct-to-elasticsearch# nfacctd -f /etc/pmacct/pmacctd.conf -d > DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 'default'/'core'. > DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: > 'elasticsearch_print'/'print'. > DEBUG: [/etc/pmacct/pmacctd.conf] debug_internal_msg:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_time_new:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_account_options:true > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ip:0.0.0.0 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_port:4739 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_disable_opt_scope_check:true > DEBUG: [/etc/pmacct/pmacctd.conf] > nfacctd_templates_file:/etc/pmacct/nf_templates_cache > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_net:bmp > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_as:bmp > DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_as:false > DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_net:false > WARN: [/etc/pmacct/pmacctd.conf] Invalid network aggregation value 'false' > WARN: [/etc/pmacct/pmacctd.conf:18] Invalid value. Ignored. > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ext_sampling_rate:1024 > DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_renormalize:true > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon:true > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_ip:0.0.0.0 > DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_max_peers:100 > DEBUG: [/etc/pmacct/pmacctd.conf] logfile:/var/log/pmacct/pmacctd.log > DEBUG: [/etc/pmacct/pmacctd.conf] > print_output_file[elasticsearch_print]:/elasticsearch_print.json > DEBUG: [/etc/pmacct/pmacctd.conf] print_output[elasticsearch_print]:json > DEBUG: [/etc/pmacct/pmacctd.conf] > print_trigger_exec[elasticsearch_print]:/etc/pmacct/p2es/triggers/elasticsearch_print > DEBUG: [/etc/pmacct/pmacctd.conf] print_refresh_time[elasticsearch_print]:15 > DEBUG: [/etc/pmacct/pmacctd.conf] aggregate[elasticsearch_print]:src_host, > dst_host, in_iface, out_iface, timestamp_start, timestamp_end, src_port, > dst_port, proto, tos, src_mask, dst_mask, tcpflags, etype, src_host_country, > dst_host_country, vlan, sampling_rate, tag, tag2, label, src_as, dst_as, > as_path, std_comm, ext_comm, lrg_comm, local_pref, med, src_as_path, > src_std_comm, src_ext_comm, src_lrg_comm, src_local_pref, src_med, > mpls_vpn_rd, peer_src_as, peer_dst_as, peer_dst_ip, peer_src_ip, src_roa, > dst_roa, src_net, dst_net > DEBUG: [/etc/pmacct/pmacctd.conf] geoipv2_file:/etc/pmacct/GeoLite2-City.mmdb > DEBUG: [/etc/pmacct/pmacctd.conf] pre_tag_map:/etc/pmacct/ifindex.map > DEBUG: [/etc/pmacct/pmacctd.conf] maps_refresh:true > DEBUG: [/etc/pmacct/pmacctd.conf] maps_entries:64000 > DEBUG: [/etc/pmacct/pmacctd.conf] maps_index:true > DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache:rpki01:8282 > DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache_version:0 > DEBUG: [/etc/pmacct/pmacctd.conf] debug:true > realloc(): invalid next size > Aborted (core dumped) > 2019-05-27T06:59:50Z INFO ( default/core/BMP ): waiting for BMP data on > 0.0.0.0:1790 > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): > plugin_pipe_size=4096000 bytes plugin_buffer_size=1548 bytes > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): ctrl channel: > obtained=212992 bytes target=21160 bytes > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > (re)loading map. > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): cache entries=16411 > base cache memory=66431728 bytes > 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): JSON: setting object > handlers. > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000001 (2212 entries). > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000000 (1 entries). > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000002 (2212 entries). > 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000001 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 > 5:0 6:0 7:0 size: 23626120 > 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000000 buckets: 3 depths: 0:1 1:0 2:0 3:0 4:0 5:0 6:0 7:0 > size: 15960 > 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000002 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 > 5:0 6:0 7:0 size: 23626120 > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] map > successfully (re)loaded. > 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > (re)loading map. > 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000001 (2212 entries). > 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000000 (1 entries). > 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: created index 40000002 (2212 entries). > 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000001 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 > 5:0 6:0 7:0 size: 23626120 > 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000000 buckets: 3 depths: 0:1 1:0 2:0 3:0 4:0 5:0 6:0 7:0 > size: 15960 > 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] > maps_index: index 40000002 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 > 5:0 6:0 7:0 size: 23626120 > 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] map > successfully (re)loaded. > 2019-05-27T06:59:56Z INFO ( default/core ): waiting for NetFlow/IPFIX data on > 0.0.0.0:4739 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1354029506] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=1 num_entries=1 off=0 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1353978550] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=1 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=2 num_entries=1 off=1548 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1354029507] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=2 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=3 num_entries=1 off=3096 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1353978551] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=3 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=542 seq=4 num_entries=1 off=4644 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=542 seq=4 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=5 num_entries=1 off=6192 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1354029508] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=5 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=6 num_entries=1 off=7740 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=6 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=7 num_entries=1 off=9288 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=7 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=530 seq=8 num_entries=1 off=10836 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1354029511] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=530 seq=8 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=9 num_entries=1 off=12384 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=9 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=10 num_entries=1 off=13932 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=10 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1353978553] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=538 seq=11 num_entries=1 off=15480 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=538 seq=11 num_entries=1 > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released > len=544 seq=12 num_entries=1 off=17028 > 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet > from [REDACTED:50141] version [10] seqno [1353978555] > 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received > len=544 seq=12 num_entries=1 > 2019-05-27T07:00:01Z ERROR ( elasticsearch_print/print ): Core process > *seems* gone. Exiting. > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
