[pmacct-discussion] Using large hex values for "label" in pre_tag_map results in strange SQL
I'm using some large hex values as the set_label in our pre_tag_map and getting some weird behavior.. example map: https://paste.somuch.fail/?bafc96e84fe95322#j6T+54l/gxN90POeMi3yuhBT9XOPMmEqt3IF5cvHOJk= When using this w/ pgsql as the output plugin, I see some errors randomly from postgres (I have dont_try_update+use_copy on): 2020-02-12 23:49:41.148 UTC [11632] postgres@sfacctd ERROR: invalid byte sequence for encoding "UTF8": 0xc0 0x2c 2020-02-12 23:49:41.148 UTC [11632] postgres@sfacctd CONTEXT: COPY acct_v9, line 1 2020-02-12 23:49:41.148 UTC [11632] postgres@sfacctd STATEMENT: COPY acct_v9 (mac_src, mac_dst, vlan, ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, comms, peer_ip_src, port_src, port_dst, tos, ip_proto, sampling_rate, timestamp_arrival, tag, tag2, label, packets, bytes) FROM STDIN DELIMITER ',' I'm also seeing when doing some debugging some strange rows being generated where the label field is longer than any label I have set in the pre_tag_map file itself: DEBUG ( all/pgsql ): ff:ff:fb:f9:b6:c4,ff:ff:ff:a1:8c:2b,0,1.1.1.1,2.2.2.2,0,547,607,30419,,1.1.1.1,23836,2,6,1024,2020-02-12 23:50:21,0,1,c7515ed894354725bc60160ee48775ce0e3b3924fb730,1,307 Any ideas where that could be coming from? -- Tim ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Realistic Scaling of pre_tag_map?
That's good news, since everything I've tested so far the maps_index has worked with. Any worries about reloading the map often/quickly? Also is there a limit to how large the label can be in the pre_tag_map and any characters that aren't supported? Seems as if '-' in any set_label operation means the whole string gets ignored.. The use-case is just mapping ip+ifIndex -> downstream devices with a label, but I've got a lot of interfaces to match there.. -- Tim On Wed, Feb 12, 2020, 12:47 AM Paolo Lucente wrote: > > Hey Tim, > > It really depends whether you can leverage maps_index (*) or not. If yes > then computations are O(1) and hence you can scale it as much as you > like and i can confirm you there is people building maps of the same > magnitude as you have in mind. If not then it's not going to work but > then again i'd be interested in your use-case, how the map would look > like, etc. > > Paolo > > (*) https://github.com/pmacct/pmacct/blob/1.7.4/CONFIG-KEYS#L1878-#L1891 > > On Tue, Feb 11, 2020 at 05:54:27PM -0600, Tim Jackson wrote: > > Just curious, what's the realistic scaling of pre_tag_map? > > > > I'm looking to maybe put 50k+ entries in it and reload it every few > > minutes.. > > > > Any real gotchas w/ that approach? > > > > -- > > Tim > > > ___ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
