Re: [pmacct-discussion] a single aggregate misses almost all traffic

2014-04-11 Thread Paolo Lucente
Hi, For the archives: we found out some traffic was VLAN tagged, hence defeating the aggregate_filter. Johannes to investigate and keep in touch if anything on the pmacctd side of the things is wrong. Cheers, Paolo On Wed, Apr 09, 2014 at 12:01:47AM +0200, Johannes Formann wrote: Hi Paolo,

Re: [pmacct-discussion] Using networks_file for filtering by network prefixes

2014-05-15 Thread Paolo Lucente
Hi Klaas, I believe you are looking for the networks_file_filter: true config directive. Can you confirm it does what you are looking for? Cheers, Paolo On Thu, May 15, 2014 at 09:37:17AM +, Tammling, Klaas wrote: Hi, I'm just trying to set up some traffic accounting with pmacct.

Re: [pmacct-discussion] Using networks_file for filtering by network prefixes

2014-05-16 Thread Paolo Lucente
'? Or would I just have to write a cleanup cronjob which runs every night? Thanks. -Ursprüngliche Nachricht- Von: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] Im Auftrag von Paolo Lucente Gesendet: Donnerstag, 15. Mai 2014 18:15 An: pmacct-discussion@pmacct.net

Re: [pmacct-discussion] Questions about IPFIX and pmacct

2014-06-03 Thread Paolo Lucente
Hi Thomas, Comments in-line: On Tue, Jun 03, 2014 at 02:48:33PM +, Thomas King wrote: We double checked the IPFIX data coming from our router. The sampling rate is contained in the data. It comes via a data record (template id=256) and the relevant fields are named

Re: [pmacct-discussion] Timestamps in RabbitMQ/JSON output

2014-06-05 Thread Paolo Lucente
Hi Chris, On Tue, Jun 03, 2014 at 10:50:21PM +0300, Chris Wilson wrote: So at the moment I am developing this by running pmacctd (not nfacctd) on my own laptop to collect and graph my own traffic. Thanks for the suggestion of using timestamp_start and _end which I didn't know you could

Re: [pmacct-discussion] Length of octetDeltaCount

2014-06-06 Thread Paolo Lucente
and ingressInterface are 2B (should be 4B) 2) ipVersion is 0 in Data Record 2) sourceIPv4Address doesn't match the advertized offset Can you point me to the LOC where the template is built? I couldn't find it by glancing over the source. Best, Jake On Thu, Jun 05, 2014 at 07:08:34AM +, Paolo

Re: [pmacct-discussion] Splitting In and Out traffic, and others questions

2014-06-23 Thread Paolo Lucente
Hi Raphael, Thanks for your kind words about the pmacct project. In-line: On Mon, Jun 23, 2014 at 02:30:35PM +0200, Raphael Mazelier wrote: It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-23 Thread Paolo Lucente
-e USER\|nfacct The [full] IMT is never cleared, and doesn't seem to exhibit this behavior... I'm performing the queries in this instance with a lock now as well. On Sat, Jun 21, 2014 at 10:05 AM, Paolo Lucente pa...@pmacct.net wrote: Hi Tim, Can you please track down memory

Re: [pmacct-discussion] Splitting In and Out traffic, and others questions

2014-06-25 Thread Paolo Lucente
Hi Raphael, Addressing the last open point: On Tue, Jun 24, 2014 at 01:00:37PM +0200, Raphael Mazelier wrote: So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Yes, agree and would recommend so. At least run sql_table_version: 9 and sql_table_type: bgp

Re: [pmacct-discussion] 1.5.0rc3 nfacctd segfaults

2014-06-30 Thread Paolo Lucente
Hi, Just a quick note for the archives, issue solved: http://www.mail-archive.com/pmacct-commits@pmacct.net/msg01138.html Cheers, Paolo On Wed, Jun 25, 2014 at 07:30:54AM +, Paolo Lucente wrote: Hi, I tried your query and a your setup in lab and all seem to behave good to me. Still

Re: [pmacct-discussion] Error inserting mysql

2014-07-01 Thread Paolo Lucente
Hi Raphael, Can you dig what entry is precisely causing this? I've seen that happening especially when enabling IPv6, ie. when pmacct tries to insert strings like ipv6-icmp in a CHAR(4). On the other hand, pmacct is unaware of the length of the field in the schema (nor i'm aware MySQL, or any

Re: [pmacct-discussion] Error inserting mysql

2014-07-02 Thread Paolo Lucente
, Paolo Lucente a écrit : Hi Raphael, Can you dig what entry is precisely causing this? I've seen that happening especially when enabling IPv6, ie. when pmacct tries to insert strings like ipv6-icmp in a CHAR(4). On the other hand, pmacct is unaware of the length of the field in the schema (nor

Re: [pmacct-discussion] pgsql debug messages incomplete

2014-07-23 Thread Paolo Lucente
Hi Pat, You are right: COPY operation, or more formally PQputCopyData() of the PostgreSQL API, returns error only for transmission issues and not for parsing ones. This is documented on a recent pretty heated thread on the pgsql-hackers mailing-list (to me, the following is the key message of

Re: [pmacct-discussion] pgsql debug messages incomplete

2014-07-23 Thread Paolo Lucente
for sql_use_copy would be good enough for me. I tried to create an account on the wiki in preparation to add to the documentation but when I try to create a new account I get the error Unknown action newaccount. Pat On 07/23/2014 03:59 AM, Paolo Lucente wrote: Hi Pat, You are right: COPY

Re: [pmacct-discussion] uacctd on a nat interface

2014-07-27 Thread Paolo Lucente
Hi Richard, I can't really suggest a workaround but it's true i'm no expert of ULOG. You could switch to libpcap and bind pmacctd to the internal interface of your gateway - this will expose you the private IP addresses rather than the public one(s). But it's possible you have several internal

Re: [pmacct-discussion] Mongodb

2014-08-13 Thread Paolo Lucente
Hi Rik, This is not possible yet. I've recently received the same request by other two people. pmacct 1.5.0 is about to be released so it will not include this feature. I'm confident this development can happen (and be released via the CVS repository) in the next month - as it does not appear

Re: [pmacct-discussion] aggregate_filter question -- change in behavior since pmacct 0.12?

2014-08-24 Thread Paolo Lucente
Hi Daniel, Although if you just upgraded a system this should not be the case, is it possible you have somehow got vlans - and hence need to change your filter in: vlan and dst net 10.0.0.0/8 If this is not the case: then i'm puzzled as the behaviour of aggregate_filter, based on underlying

Re: [pmacct-discussion] More precise flow timestamp

2014-09-10 Thread Paolo Lucente
...@pmacct.net] On Behalf Of Paolo Lucente Sent: Wednesday, September 10, 2014 9:14 AM To: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] More precise flow timestamp Hi, Yes, with the timestamp_start primitive you can get a timestamp per sFlow packet. If timestamps_secs

Re: [pmacct-discussion] Aggregation of sflow and netflow

2014-09-10 Thread Paolo Lucente
Hi TC, Inline: On Wed, Sep 10, 2014 at 12:21:08PM +, itria30...@itri.org.tw wrote: Can sfacctd and nfacctd listen on the same port? (in my use case it's port '')?? In another post said it's not allowed. Somehow I think it's worthy to ask for pmacct 1.5 is released. No, you

Re: [pmacct-discussion] mySQL ATON/NTOA and pmacct performance

2014-09-16 Thread Paolo Lucente
Hi Scott, You are looking for sql_num_hosts: true. This is tested to work with IPv4 addresses. I'm reasonably sure it won't work with IPv6: i seem to recall INET6_ATON was not available in 2011, the time of the original implementation of the feature. Should not be a biggie to introduce it.

Re: [pmacct-discussion] mySQL ATON/NTOA and pmacct performance

2014-09-17 Thread Paolo Lucente
Hi Scott, Great. Let me know how your testing goes. Should you hit any bugs or gaps, feel free to contact me privately to solve them. Then we can summarize on the list. Cheers, Paolo On Wed, Sep 17, 2014 at 04:35:05AM +, Scott Pettit - Vorco wrote: In MySQL (any SQL actually), string

Re: [pmacct-discussion] Request for Feedback: Additional Features for pmacct

2014-09-24 Thread Paolo Lucente
Hi Thomas, Great initiative, which i fully support. I'd like to make two points, which is nothing new to the feedback that you already received: 1) what TC and Andreas say in different ways is true: there is not a reliable, featureful, well supported frontend to pmacct. So in essence lots of

Re: [pmacct-discussion] Using pmacct instead nprobe

2014-10-15 Thread Paolo Lucente
Hi Xavier, To your questions: * No, you can't configure the amount of threads. pmacct uses coarse-grained multi-threading meaning specific functions, ie. BGP or IGP daemons, are embedded in a separate thread. Should you want to scale beyond a single core, you can use PF_RING as a

Re: [pmacct-discussion] Example DB schema IP field length

2014-10-20 Thread Paolo Lucente
Hi Thomas, I feel this is documented enough. The QUICKSTART guide, chapter III, invites to read README files in the sql/ dir of the tarball if using RDBMS. README.mysql and equivalents say: * src_host = ip_src (CHAR(15) NOT NULL, see README.IPv6) - or (INT(4) UNSIGNED NOT NULL, if

Re: [pmacct-discussion] trouble compiling 1.5.0

2014-10-20 Thread Paolo Lucente
Hi Chris, Do i then understand you are still unable to compile 1.5.0 on this system, no matter which work around you are trying out? If yes, would it be a possibility to get access to this box for first hand kind of troubleshooting? Thanks, Paolo On Fri, Oct 17, 2014 at 10:29:24AM +1100, Chris

Re: [pmacct-discussion] hsflowd sfacctd - flow generation analysis

2014-11-05 Thread Paolo Lucente
Hi David, Two things to try: 1) simplify your config by printing to stdout or flat-files so to remove the possibility issues are with the schema; 2) make sure no firewall, ie. iptables, is blocking packets: tcpdump socket is served before packet filtering, sfacctd indeed after that. Keep me

Re: [pmacct-discussion] hsflowd sfacctd - flow generation analysis

2014-11-06 Thread Paolo Lucente
schrieb Paolo Lucente: Hi David, Two things to try: 1) simplify your config by printing to stdout or flat-files so to remove the possibility issues are with the schema; 2) make sure no firewall, ie. iptables, is blocking packets: tcpdump socket is served before packet filtering, sfacctd indeed

Re: [pmacct-discussion] Mongodb, mongodb C driver and pmacct

2014-11-10 Thread Paolo Lucente
Hi Hendrik, Inline: On Mon, Nov 10, 2014 at 10:55:36AM +0200, Hendrik Meyburgh wrote: The driver is installed, mongodb is installed, and I tested the connection with the CLI client and the tests on the tutorial but in pmacct I only get the following error message: Connection failed to

Re: [pmacct-discussion] Seconds since IMT had it's statistics cleared?

2014-11-10 Thread Paolo Lucente
of or store outside is always better.. Thanks! -- Tim On Mon, Nov 10, 2014 at 12:28 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Tim, This info is currently not available, you should script something. But it's not a biggie of a work 1) timestamping the event and 2) introducing a knob

Re: [pmacct-discussion] pretag Limitations

2014-11-11 Thread Paolo Lucente
Hi Tim, You want to set maps_entries to ~120k entries from default (256). With such an amount of entries you may want to be sure to fit in a case supported by maps_index - and set it to true. Otherwise it will simply be a no go (read CONFIG-KEYS doc for what is supported, what is not, etc.). If

Re: [pmacct-discussion] Ideal plugin_pipe_size/plugin_buffer_size

2014-11-16 Thread Paolo Lucente
Hi Peter, Elements to calculate the values are documented in docs/INTERNALS chapter V (Communications between core process and plugins). Let me know if that helps, otherwise we can follow up privately (so that new guidelines can be added to the doc for future reference). Cheers, Paolo On Wed,

Re: [pmacct-discussion] IPFIX - NetFlow v9 convert

2014-11-27 Thread Paolo Lucente
Hi Eugene, Translation of protocols and protocol versions is not supported by pmacct. Every now and then somebody asks for it but there has never been enough push to look into it (specifically handle all the corner cases in order to do something proper). Cheers, Paolo On Mon, Nov 24, 2014 at

Re: [pmacct-discussion] nfacctd problems when using pgsql plugin with as_path and sql_use_copy

2014-12-03 Thread Paolo Lucente
Hi Pat, Give a try changing the COPY delimiter using the sql_delimiter directive. You can read more about the directive in CONFIG-KEYS. Keep me posted. Cheers, Paolo On Wed, Dec 03, 2014 at 01:30:20AM -0800, THE MIGHTY VEXORG wrote: Using the pgsql plugin with both sql_use_copy and as_path as

Re: [pmacct-discussion] nfacctd not decoding vlan id?

2014-12-11 Thread Paolo Lucente
Hi Matej, Thanks a lot for your support. Looking at the trace, your switch is sending VLAN ID using NetFlow v9/ IPFIX element #243. This was not supported natively. Now it is and code for it is in the CVS for you to check out. Log from the CVS for this:

Re: [pmacct-discussion] Sampling Rate for NetFlow v9 Records

2015-02-04 Thread Paolo Lucente
Accounting Daemon, nfacctd 1.5.1-cvs (20141119-00) --enable-jansson For suggestions, critics, bugs, contact me: Paolo Lucente pa...@pmacct.net. [user@host ~]# #!--- nfacctd config file ---!# daemonize: true nfacctd_port: 5678 plugins: memory[full] aggregate[full]: tag, tag2, in_iface

Re: [pmacct-discussion] Uacctd doesn't list any flow

2015-01-13 Thread Paolo Lucente
Hi Marco, The catch should be that you are using the NFLOG target rather than the ULOG target (legacy). One limitation of ULOG compared to NFLOG is it does not support IPv6. There are currently no plans to extend the daemon to support the NFLOG socket. With regards to group 5 vs group 10, that

Re: [pmacct-discussion] BGP Peering

2015-01-07 Thread Paolo Lucente
Hi Michael, Starting from the basics (and indeed apologies if i state the obvious): is it possible you are not compiling pmacct with the --enable-jansson switch? Support for JSON via the Jansson library is requirement to log BGP messages (ie. CSV and formatted formats are not supported). Thing

Re: [pmacct-discussion] Compiling on OS X

2015-01-08 Thread Paolo Lucente
Hi Paul, I tried myself precisely that last weekend while travelling and it was a complete mess. I must say i was running 10.7 until before the holidays and all would compile just fine; something got screwed up with the upgrade to 10.9. Going to investigate but to be fair it's low prio task on my

Re: [pmacct-discussion] AMQP Compile issue

2015-03-27 Thread Paolo Lucente
Hi Olaf, Yes, this is a known issue due to a API change on the rabbitmq-c side of the things. The code in the CVS (or daily package) should compile just fine. Cheers, Paolo On Fri, Mar 27, 2015 at 06:06:47PM +1100, Olaf de Bree wrote: Hi Paolo, I am sorry. 1. I miss spelled your name on

Re: [pmacct-discussion] network output

2015-02-27 Thread Paolo Lucente
not a programmer but from my POV it's is fairly easy to add support for network or syslog output? Once again, I am just curious to know nothing else.. :) On Sun, Feb 22, 2015 at 12:09 PM, Paolo Lucente pa...@pmacct.net wrote: Hi Pavel, As you say streaming csv over the network

[pmacct-discussion] pmacct 1.5.1 released !

2015-02-21 Thread Paolo Lucente
VERSION. 1.5.1 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases

Re: [pmacct-discussion] [PATCH] Minor AMQP improvements

2015-04-01 Thread Paolo Lucente
Hi Will, Thanks for both patches. They both make sense and will go in mainstream code. This second one i've already validated and is already applied to the code. The other one i just need some extra minimal time for QA (count it will be committed tomorrow).

Re: [pmacct-discussion] AS 0 problem in nfprobe for IPv6 flows

2015-04-08 Thread Paolo Lucente
Hi Noriyuki-san, peer_src_as is zero because you have to explicitely define the value of the 'bgp_peer_src_as_type' configuration directive. For a quick test you can set bgp_peer_src_as_type to 'bgp' in order to check you can populate with success the peer_src_as primitive. However note that

Re: [pmacct-discussion] peer_{src, dst}_as lost when pmacctd exports to nfacctd collector

2015-04-13 Thread Paolo Lucente
Hi Noriyuki-san, This is expected: consider the following BGP attributes are currently passed from nfprobe to the collector: src_as, dst_as, peer_dst_ip (BGP next-hop). This means AS-PATHs and peer source/destination ASNs are left out. We can discuss in our upcoming meeting whether it makes

Re: [pmacct-discussion] aggregate_filter on DST_AS

2015-04-30 Thread Paolo Lucente
Hi Maxim, aggregate_filter expects a filter in libpcap/tcpdump syntax - and that does not support ASNs. It should be returning an error. You should be using pre_tag_map and pre_tag_filter: a pre_tag_map can contain a line like set_tag=10 ip=0.0.0.0/0 dst_as=0; then you can filter out those with

Re: [pmacct-discussion] print plugin and swap usage

2015-04-30 Thread Paolo Lucente
Hi Pavel, Can we follow-up privately for some further investigation? I'd start with a memory profile, ie. collect every few secs/minute memory usage of every pmacct process, to determine how memory utilization changes over time - and where that leads to. In general i would say: if you keep the

Re: [pmacct-discussion] IPFIX octet delta count with Open-vSwitch

2015-05-14 Thread Paolo Lucente
Hi Jonathan, Can you please send me privately a brief capture of your ipfix packets? Adding support for field ID #352 in alternative to #1 is super easy and the trace will help my QA. You are also right about the aggregate_primitives infrastructure. It allows for aggregation (key) primitives

Re: [pmacct-discussion] pcap_filter with proto[expr:size]

2015-05-13 Thread Paolo Lucente
Hi Linas, As a workaround, can you try if the following works for you? pcap_filter[default]: ip[6:2] 0x1fff = 0 I see it is swallowed fine (apart a minor log that tells you the filter is globalized - which is no harm). Let me know. Cheers, Paolo On Tue, May 12, 2015 at 03:58:03PM +0300,

Re: [pmacct-discussion] sflow aggregate timestamp_start

2015-05-13 Thread Paolo Lucente
Hi Steffen, You are right on the difference between sflow and netflow/ipfix. Only thing i can propose is to drop the timestamp_start primitive in favor of time binning (print_history config directive and a print_output_file with time reference as part of the filename). It will never be as fine

Re: [pmacct-discussion] Issue with Ipfix reporting, VLAN tags and filtering

2015-04-16 Thread Paolo Lucente
Hi Inge, Glad to read back from you. I think you are hitting a classic limit of pcap filters: to match something a packet/flow within a VLAN the filter should be 'vlan and ip' (to say: any IP packet/flow within any vlan). pcap filters write a BPF program: i've not been able to find a way to make

Re: [pmacct-discussion] timestamps in the past

2015-04-03 Thread Paolo Lucente
like it to use that timestamp and not make one up. Would a packet capture and json output file help? Steffen -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Paolo Lucente Sent: Thursday, April 02, 2015 4:56 AM

Re: [pmacct-discussion] nfacctd output dynamic filenames not behaving as expected

2015-05-19 Thread Paolo Lucente
Hi Dariush, You are looking for the print_history companion directives. In order to have files each containing 5 mins worth of data you can add the following to your config: print_history[xxx]: 300 print_history_roundoff[xxx]: m Cheers, Paolo On Thu, May 14, 2015 at 08:29:58PM +0100, Dariush

Re: [pmacct-discussion] MaxmindDB .mmdb format support

2015-07-05 Thread Paolo Lucente
Hi Jerome, Great to read from you! It looks it's about supporting a new (like v2) Maximind API. It does not look the greatest dev effort plus i recall it was asked before, so you find me more than positive on the idea and i'm confident we can converge on this soon. Can you follow up privately if

Re: [pmacct-discussion] Netflow and nfacctd_as_new

2015-05-23 Thread Paolo Lucente
Hi Rob, To confirm that: what you are verifying is not the intended behaviour what you describe as your understanding is the intended behaviour instead. It would help to know if you have prefixes as part of your aggregation; if yes, whether they are also zeroed out if not in the networks_file

Re: [pmacct-discussion] Buffer/pipe/cache/etc sizing

2015-08-22 Thread Paolo Lucente
Hi Ruben, Your email is very timely and i understand such fluctuations between low and high traffic periods can happen in a libpcap deployment. A new feature that has been introduced as part of 1.5.2 (which is currently in the CVS and about to be released) is passing buffers inside pmacct - so

Re: [pmacct-discussion] Cisco ASR 9k CGN Logging with bulk-port-alloc empty ports fields

2015-08-22 Thread Paolo Lucente
Hi Loic, Can you share a brief trace of your IPFIX packets so to frame better the issue? If yes, we may consider following this up privately. Cheers, Paolo On Fri, Aug 21, 2015 at 09:26:41AM +0200, Loïc Rousselot wrote: Hi, as somebody hit this ? nfacctd log flawlessly (src+dst ip,

Re: [pmacct-discussion] Questions regarding tcpflags

2015-07-23 Thread Paolo Lucente
in about 20% of the packets … Or is my mapping of the flags to numbers wrong? greetings Johannes Am 22.07.2015 um 05:20 schrieb Paolo Lucente pa...@pmacct.net: Hi Johannes, Yes, the flags are OR'ed on that field as they come. Don't know which daemon you are using; if nfacctd

Re: [pmacct-discussion] Starving BGP sessions

2015-07-23 Thread Paolo Lucente
Hi Markus, Thanks for the patch; makes sense to me and i see the benefit but i need some test in lab before committing as it has its potential danger ;-) Btw, did you have a look to the config directives bgp_daemon_batch and bgp_daemon_batch_interval? They allow to re-establish the BGP peerings

Re: [pmacct-discussion] GTP inspection with pmacct

2015-07-17 Thread Paolo Lucente
For the list archives: this thread is a duplicate of an ongoing private one. On Tue, Jul 14, 2015 at 07:44:22PM -0400, Kafui Akyea wrote: Hello Paolo, Again i must say great job with this software. I have been looking through the mailing list for how to enable GTP inspection with pmacct

Re: [pmacct-discussion] netflow v9 ifindex always 0 - pmacct version 1.5.1

2015-07-17 Thread Paolo Lucente
Hi Steve, libpcap does not report such info due to no integration with the underlying OS. This is an advantage of using ULOG due to its tight coupling to the OS. Plus, in the QUICKSTART document Quickstart guide to setup a NetFlow agent/probe chapter it is described how pmacct can help setting

Re: [pmacct-discussion] netflow v9 ifindex always 0 - pmacct version 1.5.1

2015-07-19 Thread Paolo Lucente
Hi Steve, Inline: On Fri, Jul 17, 2015 at 07:36:31AM -0400, Steve Clark wrote: Am I not able to simply put something like: interface: p4p1 aggregate: src_host, dst_host, src_port, dst_port, proto, tos, in_iface, out_iface plugins: nfprobe[p4p1] nfprobe_receiver: 10.0.129.71:2055

Re: [pmacct-discussion] multiple nfacctd files being written

2015-10-22 Thread Paolo Lucente
Hi Edward, Mario is right. Plus you can set nfacctd_time_new to true to make nfacctd use the time of arrival at the collector (rather than individual flow start times) for time binning. This approach will be less precise than using flow start times; a few considerations at this propo: 1) if flow

Re: [pmacct-discussion] Virtual Interfaces not appearing in in/out_iface?

2015-11-09 Thread Paolo Lucente
Hi Horst, This is expected because you use pmacctd, the libpcap-based daemon. Libpcap has the beauty of being portable but has the drawback to not have much insight into the underlying OS - hence interfaces are not populated. You may achieve that with uacctd, the ULOG-based daemon. An alternative

Re: [pmacct-discussion] default/mysql: Duplicate entry '0----0-0-0-x.x.x.x-y.y.y.y-54644-4949-tcp-0-2015-11' for key 'PRIMARY'

2015-11-16 Thread Paolo Lucente
Hi Andreas, The issue should not be connected at all to plugin_pipe_size and plugin_buffer_size sizes - did you find a link between the issue and these config directives somewhere in the archives? This may be more connected to sql_cache_entries (although you seem to have it configured already

Re: [pmacct-discussion] Fortigate netflow inaccurate?

2015-11-04 Thread Paolo Lucente
Hi Thomas, Mario, Mario is right with his suggestion. Shall any of you have interest in troubleshooting the root cause why renormalization is not happening 'automagically' out of NetFlow data, feel free to ping me offline; it will require a snapshot of your NetFlow data for inspection and replay

Re: [pmacct-discussion] Build pmacct 1.5.2 failed

2015-10-06 Thread Paolo Lucente
Hi Manfred, That amqp_tcp_socket.h file is part of rabbitmq-c , the RabbitMQ C API/driver. You can find it here: https://github.com/alanxz/rabbitmq-c/ Can you confirm you have it installed? Also: you seem to suggest you are upgrading from an earlier version of pmacct - is this the case? Was that

[pmacct-discussion] pmacct 1.5.2 released !

2015-09-07 Thread Paolo Lucente
VERSION. 1.5.2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases

Re: [pmacct-discussion] Continuous warn src/dst_host_country_geoipv2_handler after geoipv2 activation

2015-09-03 Thread Paolo Lucente
Hi Fabien, Thanks for confirming geoipv2 seems to run perfectly - as 1.5.2 is just about to be released this is an important data point. Wrt the warning message that you mention: that is definitely coming from the Maxmind library: it is returning a code different than MMDB_SUCCESS on some

Re: [pmacct-discussion] Question regarding excluding traffic between internal networks

2015-09-17 Thread Paolo Lucente
Hi Wouter, Great to read from you! I should be correct that the amount of your supernets is manageable to put in a pcap-style filter. Plus the set of supernets should not change much. In such a case you could use a pre_tag_map like: tag=666 filter= Then in your config file: ... ! pre_tag_map:

Re: [pmacct-discussion] Question regarding excluding traffic between internal networks

2015-09-20 Thread Paolo Lucente
e filters, for > readability and managability ? > Or is this performance wise a bad idea ? > > > Does the pre_tag_filter have any CPU load we should care for ? > > > > Thanks ! > > Best regards, > > Wouter > > > > > > > -Original Messag

Re: [pmacct-discussion] FortiGate Netflow woes

2015-12-05 Thread Paolo Lucente
Hi Thomas, I ack the fact pmacct is not handling any post* field types for bytes and packets count. Can we follow-up privately on this; i would need two things: 1) a trace of the NetFlow packets (including templates) so to be able to replay it in lab; 2) a better explanation of what to do with

Re: [pmacct-discussion] Feature request: export flows via ZeroMQ

2015-12-09 Thread Paolo Lucente
Hi Vadim, Thanks for getting in touch. Was wondering the purpose of your feature request. Like, if you just think ZeroMQ would be a nice addition to the current messaging options in pmacct (RabbitMQ and Kafka); or if actually you want to inject data from pmacct into ntopng. In case of the

Re: [pmacct-discussion] 0 timestamp when using print plugin

2015-12-14 Thread Paolo Lucente
Hi Ruben, It should be just matter of adding print_history to your config, ie. 'print_history: 5m' for 5 mins time-bins. Cheers, Paolo On Mon, Dec 14, 2015 at 01:12:27PM +0100, Ruben Laban wrote: > Hi, > > Today I ran into an issue with pmacctd which feels familiar, but I > can't remember how

Re: [pmacct-discussion] files_umask not used for directories

2015-12-17 Thread Paolo Lucente
Hi Ruben, I'm with you. Let me investigate and come back to you on this. Cheers, Paolo On Wed, Dec 16, 2015 at 08:46:22PM +0100, Ruben Laban wrote: > Hi, > > The setting files_umask is only used for files created by pmacctd, > and not for directories created by pmacctd. One can argue that that

Re: [pmacct-discussion] 0 timestamp when using print plugin

2015-12-17 Thread Paolo Lucente
his value instead of print_refresh_time one. > > Seems like this explanation isn't exactly true anymore, as > print_refresh_time was already defined. > > Anyways, I'm glad this is now working as expected again and I can > continue with this (small) implementation. > > Re

Re: [pmacct-discussion] pmacct 1.5.2 released !

2016-01-07 Thread Paolo Lucente
/gravitizer/bin/monitor.pl > print_output_file_append: true > ! > nfacctd_port: 2055 > > Thanks, > > Ed > > On Thu, Jan 7, 2016 at 4:20 PM, Paolo Lucente <pa...@pmacct.net> wrote: > > > Hi Ed, > > > > You mean you kind of just upgraded to 1.5.2

Re: [pmacct-discussion] NetFlow v9 sequence numbers

2015-12-22 Thread Paolo Lucente
, 2015 at 09:43:28PM +0100, Radu Anghel wrote: > Hi Paolo, > > Thank you for your answer. > > For me it is not urgent as I am just starting with this, but it > would be really useful in the future. > > Best wishes, > > Radu > > > On 20.12.2015 15:57, Paol

Re: [pmacct-discussion] Build question

2015-12-23 Thread Paolo Lucente
Hi Javier, Is it possible you are using a MongoDB C driver >= 0.9? Currently, pmacct only supports the legacy C driver (up to release 0.8.1); it can be found here: https://github.com/mongodb/mongo-c-driver-legacy I just realize now that URLs may have changed and hence docs need a refresh.

Re: [pmacct-discussion] No rows written with PostgreSQL and sql_preprocess.

2015-12-23 Thread Paolo Lucente
Hi Harry, Your nfacctd config looks OK; i tried to reproduce in lab (although i have availability of PostgreSQL 9.1 instead of 9.4 i don't think it's making an actual difference) without success. Any chance i can debug this on your box? If yes, we can follow-up privately for the details. In

Re: [pmacct-discussion] NetFlow v9 sequence numbers

2015-12-20 Thread Paolo Lucente
Hi Radu, You are right: sequence number is not a natively supported primitive and, since it's part of the header and not of the flow record, it is not possible to leverage the aggregate_primitives framework either. This said, writing native support for the sequence number is not a super big

Re: [pmacct-discussion] default/mysql: Duplicate entry '0----0-0-0-x.x.x.x-y.y.y.y-54644-4949-tcp-0-2015-11' for key 'PRIMARY'

2015-11-19 Thread Paolo Lucente
t's bridged and the bridge has the ip is ok, too. > > Maybe it's possible to change the severity of the allocate memory > message to ERROR in one of the next releases. > > Nevertheless, this needs to be said: Paolo, you did really great work. > Really cool software and thank you for

Re: [pmacct-discussion] filtering based on IPFIX template ID or..

2015-11-19 Thread Paolo Lucente
Hi Sergey, For template ID you mean flowset ID? If yes, then you can use a pre_tag_map and the flowset_id directive to tag session start/ session end differently. Then a pre_tag_filter can be used to direct different tags to different plugins, ie. because you want to log them in different

Re: [pmacct-discussion] pmacct 1.5.2 released !

2016-01-11 Thread Paolo Lucente
To wrap-up on this. Bug was confirmed, reproduced and fixed. Fix has been also tested working by Ed. Log of the commit is here: https://github.com/pmacct/pmacct/commit/6d518f4a2b0e808ae89e2b896fa3c0ba2c3fc64b Cheers, Paolo On Thu, Jan 07, 2016 at 11:00:43PM +, Paolo Lucente wrote: > Hi

[pmacct-discussion] pmacct 1.6.0 released !

2016-06-07 Thread Paolo Lucente
VERSION. 1.6.0 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data

Re: [pmacct-discussion] Tips on debugging IPFIX/v10 on 1.5.2?

2016-06-04 Thread Paolo Lucente
Hi Inge, Any chance you have some aggregate_filter or any other filtering in place via pre_tag_map? Another option could be the new MX box is exporting less data than the previous one (ie. as a result of a different configured sampling rate) and buffers (plugin_buffer_size mainly) are set too

Re: [pmacct-discussion] MySQL Timezone handling

2016-05-30 Thread Paolo Lucente
Hi Vaggelis, I look forward to any thoughts about data types. Personally, the very first reaction this trigger is: the backend of the accounting system should be set to a timezone that does not change during the year and, even more ideally, to UTC. UTC is ideal because it helps when stuff is

[pmacct-discussion] pmacct 1.5.3 released !

2016-01-14 Thread Paolo Lucente
VERSION. 1.5.3 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases

Re: [pmacct-discussion] Not building with Mongodb

2016-01-17 Thread Paolo Lucente
is appreciated. > > Thanks, > Javier > > On Sat, Jan 16, 2016 at 8:28 AM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Hi Javier, > > > > What version of the MongoDB c driver are you using? It is possible > > you are using som

Re: [pmacct-discussion] Sflow counters

2016-01-17 Thread Paolo Lucente
he amqp plugin. > > Best regards > > On 05/11/2015 04:59, Paolo Lucente wrote: > > Hi Robin, > > > > Thanks for your kind words. > > > > About sFlow counters: you are right, currenty only streamed output to > > files is supported - no AMQP or Kafka

Re: [pmacct-discussion] pmacct 1.5.3 released !

2016-01-18 Thread Paolo Lucente
Hi Will, Absolutely reat to hear; as Kafka support is in its infancy in pmacct, please keep me posted for any issues (or requests). Yes, since December the code is now on GitHub and it's not anymore a mirror of the CVS repository; Job Snijders helped massively to make this happen. It was right

Re: [pmacct-discussion] IPFIX -> NetFlow v9/v5 convert

2016-02-06 Thread Paolo Lucente
Hi, > > Some time ago I asked about converting IPFIX to NetFlow v5/v9. Is it > possible now? > > 2014-11-27 15:17 GMT+03:00 Paolo Lucente <pa...@pmacct.net>: > > Hi Eugene, > > > > Translation of protocols and protocol versions is not supported by &g

Re: [pmacct-discussion] Feature Request - nDPI Classification

2016-02-06 Thread Paolo Lucente
timelines. Cheers, Paolo On Mon, Jan 18, 2016 at 05:22:21AM +, Paolo Lucente wrote: > Hi Thomas, > > Thanks for bringing this up. This is on my todo list for some time > due to the aging status of L7-Filter; please anybody using pmacctd/ > uacctd add your voice to this

Re: [pmacct-discussion] src_as/dst_as populating as 0

2016-02-06 Thread Paolo Lucente
Hi Steve, Is it possible nfacctd is not configured as RR client on the routers and hance it is getting only partial routes? Cheers, Paolo On Fri, Feb 05, 2016 at 01:36:49PM -0700, Steve Dodd wrote: > I?m having an issue where a large number of flows aren?t populating with > src_as/dst_as

Re: [pmacct-discussion] Question about teeing and sampling

2016-02-10 Thread Paolo Lucente
Hi Pau, On the sampling part: this is not supported but for a good reason, i would say. Sampling is, yes, about sending less data over but also about being able to renormalize data using some math; sampling packets passing via an interface makes sense; dropping some well-formed NetFlow packets

Re: [pmacct-discussion] Question about teeing and sampling

2016-02-10 Thread Paolo Lucente
Hi Mario, Wrt the balancing algorithm & templates. Definitely the round-robin balancing algorithm is suitable only for - pass me the term - non- contextual protocols/protocol versions (ie. sFlow and NetFlow v5); NetFlow v9/IPFIX, which are template-based, require the 'hash-agent' one where the IP

Re: [pmacct-discussion] Multiple pmacct processes listening at similar interface

2016-02-24 Thread Paolo Lucente
Hi Franz, Yes, it's no problem if, in general, two processes running libpcap are binding to the same interface. You can in fact not only have any two pmacctd binding there, but also a pmacctd and a tcpdump, etc. Cheers, Paolo On Tue, Feb 23, 2016 at 01:23:29PM +0100, fboehm wrote: > Hi, > > I

Re: [pmacct-discussion] hsflowd & sfacctd - flow generation & analysis

2016-02-24 Thread Paolo Lucente
Hi Nicolas, Support for sFlow counters was introduced in 1.5.2 and made more robust in 1.5.3. However consider this is interface counter stats; the host sFlow structs is currently not supported - we can think about it if there is interest around it. Same applies to the agent side of the things,

Re: [pmacct-discussion] ?^??: ?^??: Multiple pmacct processes listening at similar interface

2016-02-29 Thread Paolo Lucente
Hi TC, I would simply not recommend to run both sFlow and NetFlow on the same port; the only way possible is the one you mention in your last email: use a replicator to feed the actual daemons; but it seems too involved to me if you do not have strong reasons for it (technical limitations or

Re: [pmacct-discussion] pcap/ulog performance on Linux

2016-02-29 Thread Paolo Lucente
Hi Vincent, You are right with your assumption. Support of NFLOG has been requested, ie. to support IPv6, but is still pending and i don't have it currently on my radar (ie. 1.6.0 / 1.6.1). Cheers, Paolo On Mon, Feb 29, 2016 at 05:55:51PM +0100, Vincent Bernat wrote: > ??? 26 f??vrier 2016

Re: [pmacct-discussion] Multiple pmacct processes listening at similar interface

2016-02-25 Thread Paolo Lucente
Hi TC, Consider nfacctd and sfacctd do not use libpcap in order to read the incoming NetFlow/IPFIX and sFlow packets respectively; only pmacctd uses libpcap. This is why you can't let both nfacctd and sfacctd bind to the same port and IP address. Cheers, Paolo On Thu, Feb 25, 2016 at

<    1   2   3   4   5   6   7   8   9   10   >