Hi Felix,

Thanks for getting in touch. Can you please get more data about the
crash by following this section fo the QUICKSTART (i'd need an output of
GDB 'bt'):

https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2606-#L2635

You can follow up 1:1 so that we don't disturb everybody with the
back/forth that will be needed by the troubleshooting process. We can
then summarize resolution on the list.

Paolo
  
On Mon, May 27, 2019 at 03:55:04PM +0000, Felix Stolba wrote:
>  Hi
> 
> I’m trying to use a pre_tag_map with less than 5000 entries with the purpose 
> of adding the ingress and egress interface names to the flow records as 
> labels. When using this map, nfacctd reproducibly crashes, tested using 1.7.1 
> and 1.7.3. I would appreciate if someone (Paolo? :) ) could help isolate the 
> problem. Debug logs can be found attached. I will be happy to provide any 
> additional info that will be needed.
> 
> When crashing, nfacctd emits this log message:
> realloc(): invalid next size
> Aborted (core dumped)
> 
> Few tests I've already done:
> * Use a smaller map: works - did a PoC using a map of about 200 lines, this 
> worked great.
> * Delete everything below OUTTABLE (see below): works - having only the top 
> part of the map keeps pmacct running
> * Delete some lines below OUTTABLE - produced a different error message: 
> "corrupted size vs. prev_size"
> 
> The pre_tag_map essentially looks like the ones in the JEQ examples [1]:
> 
> set_label=INTERFACE_NAME ip=ROUTER_IP in=IFINDEX jeq=OUTTABLE
> ... 2000 lines of similar mappings ...
> set_label=INTERFACE_NAME ip=ROUTER_IP out=IFINDEX label=OUTTABLE      
> ... 2000 lines of similar mappings ...
> 
> 
> Best regards
> Felix
> 
> [1] https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example
> 
> 
> 

> flow01:~/pmacct-to-elasticsearch# nfacctd -f /etc/pmacct/pmacctd.conf -d
> DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 'default'/'core'.
> DEBUG: [/etc/pmacct/pmacctd.conf] plugin name/type: 
> 'elasticsearch_print'/'print'.
> DEBUG: [/etc/pmacct/pmacctd.conf] debug_internal_msg:true
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_time_new:true
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_account_options:true
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ip:0.0.0.0
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_port:4739
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_disable_opt_scope_check:true
> DEBUG: [/etc/pmacct/pmacctd.conf] 
> nfacctd_templates_file:/etc/pmacct/nf_templates_cache
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_net:bmp
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_as:bmp
> DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_as:false
> DEBUG: [/etc/pmacct/pmacctd.conf] pmacctd_net:false
> WARN: [/etc/pmacct/pmacctd.conf] Invalid network aggregation value 'false'
> WARN: [/etc/pmacct/pmacctd.conf:18] Invalid value. Ignored.
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_ext_sampling_rate:1024
> DEBUG: [/etc/pmacct/pmacctd.conf] nfacctd_renormalize:true
> DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon:true
> DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_ip:0.0.0.0
> DEBUG: [/etc/pmacct/pmacctd.conf] bmp_daemon_max_peers:100
> DEBUG: [/etc/pmacct/pmacctd.conf] logfile:/var/log/pmacct/pmacctd.log
> DEBUG: [/etc/pmacct/pmacctd.conf] 
> print_output_file[elasticsearch_print]:/elasticsearch_print.json
> DEBUG: [/etc/pmacct/pmacctd.conf] print_output[elasticsearch_print]:json
> DEBUG: [/etc/pmacct/pmacctd.conf] 
> print_trigger_exec[elasticsearch_print]:/etc/pmacct/p2es/triggers/elasticsearch_print
> DEBUG: [/etc/pmacct/pmacctd.conf] print_refresh_time[elasticsearch_print]:15
> DEBUG: [/etc/pmacct/pmacctd.conf] aggregate[elasticsearch_print]:src_host, 
> dst_host,  in_iface, out_iface, timestamp_start, timestamp_end, src_port, 
> dst_port, proto, tos, src_mask, dst_mask, tcpflags, etype, src_host_country, 
> dst_host_country, vlan, sampling_rate, tag, tag2, label, src_as, dst_as, 
> as_path, std_comm, ext_comm, lrg_comm, local_pref, med, src_as_path, 
> src_std_comm, src_ext_comm, src_lrg_comm, src_local_pref, src_med, 
> mpls_vpn_rd, peer_src_as, peer_dst_as, peer_dst_ip, peer_src_ip, src_roa, 
> dst_roa, src_net, dst_net
> DEBUG: [/etc/pmacct/pmacctd.conf] geoipv2_file:/etc/pmacct/GeoLite2-City.mmdb
> DEBUG: [/etc/pmacct/pmacctd.conf] pre_tag_map:/etc/pmacct/ifindex.map
> DEBUG: [/etc/pmacct/pmacctd.conf] maps_refresh:true
> DEBUG: [/etc/pmacct/pmacctd.conf] maps_entries:64000
> DEBUG: [/etc/pmacct/pmacctd.conf] maps_index:true
> DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache:rpki01:8282
> DEBUG: [/etc/pmacct/pmacctd.conf] rpki_rtr_cache_version:0
> DEBUG: [/etc/pmacct/pmacctd.conf] debug:true
> realloc(): invalid next size
> Aborted (core dumped)

> 2019-05-27T06:59:50Z INFO ( default/core/BMP ): waiting for BMP data on 
> 0.0.0.0:1790
> 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): 
> plugin_pipe_size=4096000 bytes plugin_buffer_size=1548 bytes
> 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): ctrl channel: 
> obtained=212992 bytes target=21160 bytes
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> (re)loading map.
> 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): cache entries=16411 
> base cache memory=66431728 bytes
> 2019-05-27T06:59:55Z INFO ( elasticsearch_print/print ): JSON: setting object 
> handlers.
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000001 (2212 entries).
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000000 (1 entries).
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000002 (2212 entries).
> 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000001 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 
> 5:0 6:0 7:0 size: 23626120
> 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000000 buckets: 3 depths: 0:1 1:0 2:0 3:0 4:0 5:0 6:0 7:0 
> size: 15960
> 2019-05-27T06:59:55Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000002 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 
> 5:0 6:0 7:0 size: 23626120
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] map 
> successfully (re)loaded.
> 2019-05-27T06:59:55Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> (re)loading map.
> 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000001 (2212 entries).
> 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000000 (1 entries).
> 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: created index 40000002 (2212 entries).
> 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000001 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 
> 5:0 6:0 7:0 size: 23626120
> 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000000 buckets: 3 depths: 0:1 1:0 2:0 3:0 4:0 5:0 6:0 7:0 
> size: 15960
> 2019-05-27T06:59:56Z DEBUG ( default/core ): [/etc/pmacct/ifindex.map] 
> maps_index: index 40000002 buckets: 4441 depths: 0:1991 1:209 2:12 3:0 4:0 
> 5:0 6:0 7:0 size: 23626120
> 2019-05-27T06:59:56Z INFO ( default/core ): [/etc/pmacct/ifindex.map] map 
> successfully (re)loaded.
> 2019-05-27T06:59:56Z INFO ( default/core ): waiting for NetFlow/IPFIX data on 
> 0.0.0.0:4739
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1354029506]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=1 num_entries=1 off=0
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1353978550]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=1 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=2 num_entries=1 off=1548
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1354029507]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=2 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=3 num_entries=1 off=3096
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1353978551]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=3 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=542 seq=4 num_entries=1 off=4644
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=542 seq=4 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=5 num_entries=1 off=6192
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1354029508]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=5 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=6 num_entries=1 off=7740
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=6 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=7 num_entries=1 off=9288
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=7 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=530 seq=8 num_entries=1 off=10836
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1354029511]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=530 seq=8 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=9 num_entries=1 off=12384
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=9 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=10 num_entries=1 off=13932
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=10 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1353978553]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=538 seq=11 num_entries=1 off=15480
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=538 seq=11 num_entries=1
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer released 
> len=544 seq=12 num_entries=1 off=17028
> 2019-05-27T06:59:56Z DEBUG ( default/core ): Received NetFlow/IPFIX packet 
> from [REDACTED:50141] version [10] seqno [1353978555]
> 2019-05-27T06:59:56Z DEBUG ( elasticsearch_print/print ): buffer received 
> len=544 seq=12 num_entries=1
> 2019-05-27T07:00:01Z ERROR ( elasticsearch_print/print ): Core process 
> *seems* gone. Exiting.

> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to