Re: [pmacct-discussion] Using pf_ring together with pmacct?

2014-02-04 Thread Paolo Lucente
> >PID TID PSR %CPU > 15139 15139 2 54.6 > > > > > > 2014-02-04 Paolo Lucente : > > > Hi Joan, > > > > Perfect, thanks for having summarized your solution. > > > > Cheers, > > Paolo > > > > On Tue, Feb 04, 20

Re: [pmacct-discussion] TCP connection start timestamp and duration

2014-02-04 Thread Paolo Lucente
Hi Stathis, Inline: On Tue, Feb 04, 2014 at 02:09:05PM +0200, Stathis Gkotsis wrote: > I am mainly interested in TCP. I would like the final export to contain one > line per src_host,dst_host,src_port,dst_port,proto combination, along with > the start timestamp of the corresponding TCP connect

Re: [pmacct-discussion] Using pf_ring together with pmacct?

2014-02-03 Thread Paolo Lucente
-s /usr/local/lib/libpcap.so.1 libpcap.so.1 > > Once that was done, pmacct could start, and the performance is really much > better, I'd say about a 30% less load in the system. > I'll report any other findings I do, > > Thanks for your work again, > > Joan > >

Re: [pmacct-discussion] TCP connection start timestamp and duration

2014-02-03 Thread Paolo Lucente
Hi Stathis, You do not outline what is the capturing method you intend to use, ie. libpcap, NetFlow/IPFIX, sFlow, etc. If using NetFlow/ IPFIX you are sorted already, as you just add timestamp_start and timestamp_end to your aggregation method to the quintuple. If using libpcap, well, a NetFlow p

Re: [pmacct-discussion] Using pf_ring together with pmacct?

2014-02-03 Thread Paolo Lucente
Hi Joan, I did not look at PF_RING recently but the steps you outline are those that make sense to me. Can you say what is precisely not working linking pmacct against PF_RING libpcap? Feel free to send me some output here or (maybe preferrably) privately, then we summarize outcome here. Cheers,

Re: [pmacct-discussion] Changing main process title?

2014-02-01 Thread Paolo Lucente
. As far as I can see, the logging still uses the > hardcoded "default" everywhere. > > Regards, > Ruben > > On 2013-07-06 14:45, Paolo Lucente wrote: > >Hi Brian, Ruben, > > > >This is now available in the CVS via a new set of configuration &g

Re: [pmacct-discussion] Minor cosmetic patch

2014-01-31 Thread Paolo Lucente
Hi Ruben, Thanks for that. Patch accepted and will commit to CVS later today. Cheers, Paolo On Fri, Jan 31, 2014 at 10:23:44AM +0100, Ruben Laban wrote: > Hi Paolo, > > I've been meaning to send you this patch for ages (unless I did and > forgot about it); > > Index: src/signals.c > ==

Re: [pmacct-discussion] raising MAX_BGP_STD_COMMS/MAX_BGP_EXT_COMMS/MAX_BGP_ASPATH

2014-01-31 Thread Paolo Lucente
Hi Adam, Sure, send me a backtrace. A low-touch option is to leave everything as is in the main-stream code, make you work with some value of 512 bytes, and document it back on the wiki so that it can be useful resource for future. Actually, I'm curious about your use-case for this. I ask because

Re: [pmacct-discussion] sFlow related questions

2014-01-24 Thread Paolo Lucente
special". Indeed, that is a "special" indicating that there is no input/output interface (depending which field the 0x3FFF is found). This is typically the case if you ping the switch itself, for example. Cheers, Paolo > On 2014-01-23 14:50, Paolo Lucente wrote: > >Hi R

Re: [pmacct-discussion] sFlow related questions

2014-01-23 Thread Paolo Lucente
Hi Ruben, Those are input and ouput interfaces of the switch, expressed as SNMP ifIndexes. If you see later in the CSV you have SRC_PORT and DST_PORT fields which are zero - making sense since the packets IP protocol is ICMP. In general, if you see anything strange with sFlow and want to debug

Re: [pmacct-discussion] Classification error in pre-tag-mapping with filter

2014-01-13 Thread Paolo Lucente
Hi Martin, On Mon, Jan 13, 2014 at 02:45:25PM +0100, Martin Topholm wrote: > On Fri, 10 Jan 2014, Paolo Lucente wrote: > > [ .. ] > > > Any chance the traffic is VLAN-tagged and/or MPLS-labelled and > > VLAN tag and/or MPLS labels are exposed to pmacct via IPFIX? In &g

Re: [pmacct-discussion] Classification error in pre-tag-mapping with filter

2014-01-10 Thread Paolo Lucente
Hi Martin, To clarify: no traffic at all, both originated from and delivered to your address blocks listed, gets tagged with 612/613/712/713. Correct? Or some is and some is not? Any chance the traffic is VLAN-tagged and/or MPLS-labelled and VLAN tag and/or MPLS labels are exposed to pmacct via I

Re: [pmacct-discussion] pmacct 1.5.0rc2 released !

2014-01-06 Thread Paolo Lucente
you could offer remote-access to get this specific issue looked into. I propose to continue this off-list. Cheers, Paolo On Tue, Jan 07, 2014 at 01:06:26AM +0200, Viacheslav Dubrovskyi wrote: > 07.01.2014 00:20, Paolo Lucente пишет: > >Hi Slava, > Hi Paolo, > > Get another error:

Re: [pmacct-discussion] pmacct 1.5.0rc2 released !

2014-01-06 Thread Paolo Lucente
Dubrovskyi wrote: > 06.01.2014 22:56, Paolo Lucente пишет: > >Hi Slava, > > > >Good to have compiler peakiness turned on. Fixed the issue, log > >from the CVS below: > > > >http://www.mail-archive.com/pmacct-commits@pmacct.net/msg00946.html > > > >If you

Re: [pmacct-discussion] pmacct 1.5.0rc2 released !

2014-01-06 Thread Paolo Lucente
definition with u_int64_t. Just let me know should you stumble into further compiling issues. Cheers, Paolo On Tue, Dec 31, 2013 at 01:53:32PM +0200, Viacheslav Dubrovskyi wrote: > 25.12.2013 23:47, Paolo Lucente пишет: > >VERSION. > >1.5.0rc2 > Hi Paolo, > > Get err

[pmacct-discussion] pmacct 1.5.0rc2 released !

2013-12-25 Thread Paolo Lucente
VERSION. 1.5.0rc2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (Mon

Re: [pmacct-discussion] pmacct-1.5.0rc1

2013-12-19 Thread Paolo Lucente
Hi Steve, You are entirely correct, i've just fixed the doc which will be shortly available through the CVS repository. Thanks very much for reporting the issue. Cheers, Paolo On Wed, Dec 18, 2013 at 11:43:53AM -0500, Steve Clark wrote: > Hi, > > Reading section X. of the QUICKSTART GUIDE I am

Re: [pmacct-discussion] nfacctd plugin crash

2013-12-12 Thread Paolo Lucente
Hi Adam, Can you please share (here or privately) your configuration? >From what i see i would at least expect you have 'class' as part of your aggregation method. Cheers, Paolo On Wed, Dec 11, 2013 at 03:47:45PM -0500, Adam Jacob Muller wrote: > Hi, > I have this somewhat unusual crash that hap

Re: [pmacct-discussion] Netflow proxy

2013-12-11 Thread Paolo Lucente
Hi Rafael, The scenario is not supported. While it could be implemented for a specific case, ie. NetFlow v5, this would not actually work very well for more current protocol versions, ie. NetFlow v9, IPIFX and sFlow, due to their variable record structure. An example is: a NetFlow datagram arrives

Re: [pmacct-discussion] Documentation for aggregation primitives

2013-12-09 Thread Paolo Lucente
Irmatov wrote: > Hi, Paolo! > > On Sat, Dec 7, 2013 at 11:28 PM, Paolo Lucente wrote: > > In 1.5.0rc1 and later you have the -a CLI switch. It kind of expands > > description of primitives to what you can see below: > > > > shell> nfacctd -a > > L3 > >

Re: [pmacct-discussion] # ? !! comments

2013-12-08 Thread Paolo Lucente
Hi Joel, Your proposal makes sense. However, as you know, devil is in these innocent one liners. I'd not go ahead with this change approx one month from upcoming release, 1.5.0rc2. I'd also encourage you to patch your code and confirm you do not run into any issues. I'm adding to my todo list to p

Re: [pmacct-discussion] Documentation for aggregation primitives

2013-12-07 Thread Paolo Lucente
Hi Joel, In 1.5.0rc1 and later you have the -a CLI switch. It kind of expands description of primitives to what you can see below: shell> nfacctd -a NetFlow Accounting Daemon, nfacctd 1.5.0rc2-cvs (20131206-00) L2 cos : Ethernet CoS, 802.1P

Re: [pmacct-discussion] reload/refresh config?

2013-12-07 Thread Paolo Lucente
Hi Joel, pmacct processes are not reloadable, you have to stop/start upon, say, changing the configuration. But (most) maps are reloadable at runtime unless specified otherwise. To flag pmacct you want to reload a map you can send a SIGUSR2, ie. "killall -USR2 nfacctd". Cheers, Paolo On Sat, Dec

Re: [pmacct-discussion] Byte Count Scaling with IPFIX?

2013-12-07 Thread Paolo Lucente
Hi Joel, You are correct. It appears sampling information is not passed over by the NetFlow exporter. I will let anybody else on this list using Juniper & IPFIX recently reply more broadly but: my experience on this up to 2011 is captured here: https://puck.nether.net/pipermail/juniper-nsp/2011-J

Re: [pmacct-discussion] nfacctd Networks Problem

2013-12-04 Thread Paolo Lucente
Hi Terry, What version of pmacct are you running? If a recent one, ie. >= 0.14.3, you should have 'networks_file_filter: true' in your config in order to explicitely enable filtering (as is documented in the CONFIG-KEYS file). Cheers, Paolo On Thu, Dec 05, 2013 at 12:37:58AM +, Terry Duchch

Re: [pmacct-discussion] buffer overflow / backtrace on 1.5rc1

2013-12-04 Thread Paolo Lucente
Hi Joel, Wow, interesting. What OS are you running? What BGP capabilities are enabled and which address families are you sending over? It would help if you can run the daemon under gdb and collect 'bt' information (send it directly to me). Post in the same email also your config. We can take it fr

Re: [pmacct-discussion] Off by one warning?

2013-12-04 Thread Paolo Lucente
0:6001agent=X:0 > WARN: expecting flow '234613' but received '25593510' > collector=0.0.0.0:6001agent=X:0 > WARN: expecting flow '25593572' but received '234613' > collector=0.0.0.0:6001agent=X:0 > WARN: expecting flow '234614' but rece

Re: [pmacct-discussion] nfacctd, mongodb, nfacctd_time_new and mongo_tables variables

2013-12-03 Thread Paolo Lucente
B Plugin [all] > > After another minute, this grows very, very quickly until the box will OOM: > fresh-netflow pmacct-1.5.0rc1 # ps aux|grep -c Mongo > 72 > > Is this possibly just because MongoDB can't keep up with the insert > rate I'm asking it for here? > &

Re: [pmacct-discussion] nfacctd, mongodb, nfacctd_time_new and mongo_tables variables

2013-12-01 Thread Paolo Lucente
Hi Adam, You should set mongo_history to something, ie.: mongo_history[all]: 5m mongo_history_roundoff[all]: m This enables historical accounting and time-bins and you should then see collection names being populated with the correct date (and time). Let me know if this works for you. When you

Re: [pmacct-discussion] nfacctd, ipfix and tee transparent mode

2013-11-22 Thread Paolo Lucente
adcast behaviour? > > > -Adam > > On 11/21/2013 07:04 AM, Paolo Lucente wrote: > >Hi Adam, > > > >You are right, there is a bug lying around 1.5.0rc1 when not setting > >an explicit value for plugin_pipe_size and/or plugin_buffer_size. The > >issue was alre

Re: [pmacct-discussion] nfacctd, ipfix and tee transparent mode

2013-11-21 Thread Paolo Lucente
Hi Adam, You are right, there is a bug lying around 1.5.0rc1 when not setting an explicit value for plugin_pipe_size and/or plugin_buffer_size. The issue was already fixed in the CVS code: http://www.mail-archive.com/pmacct-commits@pmacct.net/msg00896.html Cheers, Paolo On Thu, Nov 21, 2013 at

Re: [pmacct-discussion] sfprobe|nfprobe dying with IMT

2013-11-21 Thread Paolo Lucente
Hi Stig, I've also looked at the Vyatta tiket, I seem unable to reproduce it. Since it appears it's the core proess failing, would you mind running it under gdb and send me a backtrace once it crashes? Let's follow-up privately as debugger info might not be of very general interest. Cheers, Paolo

Re: [pmacct-discussion] plugin_pipe_size = 0 ???

2013-11-20 Thread Paolo Lucente
Hi Stig, Great to hear from you, long time no speak. You just anticipated me :) That was precisely the issue and is already patched in CVS. Cheers, Paolo On Wed, Nov 20, 2013 at 02:40:57PM -0800, Stig Thormodsrud wrote: > Ok, false alarm. I did some more debugging and noticed one difference > b

Re: [pmacct-discussion] pmacct and more than 1 AS

2013-11-18 Thread Paolo Lucente
Hi Adam, The scenario is supported by pmacct, there are two pieces to it: * pmacct BGP daemon acts as a passive BGP neighbor and replies to an incoming BGP OPEN message with the same AS number contained in the OPEN. This means a single collector can peer with different ASNs no problem. If y

Re: [pmacct-discussion] Off by one warning?

2013-11-11 Thread Paolo Lucente
Hi Joel, Could also be packets are received out of order, which can be harmless depending on the use-cases. Anyway if annoying these messages can be disabled by setting nfacctd_disable_checks to true. I propose this idea because i don't seem to have seen such warnings on a regular basis on other I

Re: [pmacct-discussion] IPFIX Decodes

2013-11-11 Thread Paolo Lucente
Hi Joel, Thanks for your feedback, much appreciated. Actually IE #152 and #153 are understood in the code (timestamp_start, timestamp_end primitives and sql_history). The issue is purely visual, in the context of the debug message. IE #136 is not natively supported instead. But pmacct release 1

Re: [pmacct-discussion] pmaactd bgp as-path missing on sql inserts

2013-10-30 Thread Paolo Lucente
Hi Derrick, Perfect. About your questions: 1. src_host, dst_host and src_net, dst_net are mutual exclusive in the sense they are multiplexed on the same field. If you remove src_net and dst_net from the 'aggregate' you will get individual hosts. If you also need IP prefixes readily avail

Re: [pmacct-discussion] pmaactd bgp as-path missing on sql inserts

2013-10-29 Thread Paolo Lucente
Hi Derrick, Excellent capturing of information of yours. >From the SQL inserts you posted it's only evident that the src_as_path is not working properly - ie. does not say anything about as_path: you sure you did verify the same problem with that primitive? If yes, can you post something about it

Re: [pmacct-discussion] sfprobe documentation error?

2013-10-25 Thread Paolo Lucente
Hi Brian, You are correct: there is a mistake in the docs (just fixed, will be committed to CVS soon). Configuration directive to use is sampling_rate rather than sfprobe_sampling_rate. Cheers, Paolo On Thu, Oct 24, 2013 at 06:00:15PM -0400, Brian Rak wrote: > I can't seem to configure the sampl

Re: [pmacct-discussion] BGP+VPNv4 Address Family & Netflow v5

2013-10-14 Thread Paolo Lucente
t useful for me). Is there anything > to do? > > > regards, > Olivier > > > > Le 11 oct. 2013 à 21:02, Paolo Lucente a écrit : > > > Hi Olivier, > > > > The scenario is supported. You using 1.5.0rc1? If so, you need to add > > a flow_to_rd_map m

Re: [pmacct-discussion] BGP+VPNv4 Address Family & Netflow v5

2013-10-11 Thread Paolo Lucente
Hi Olivier, The scenario is supported. You using 1.5.0rc1? If so, you need to add a flow_to_rd_map map. More info in: "examples/flow_to_rd.map.example" in the pmacct distribution tarball. In principle you should need just a line like: id= ip=0.0.0.0/0 Let me know how it goes. Cheers, Paolo On

Re: [pmacct-discussion] sfacctd counts much less traffic than SNMP port stat

2013-10-10 Thread Paolo Lucente
Hi Stanislaw, I would suggest a few tests to drill this issue down. You are aggregating src_as, dst_as but to better compare against SNMP i would suggest an aggregation peer_src_ip, in_iface - this way you know for sure you are comparing apples with apples without relying on the knowledge of one o

Re: [pmacct-discussion] amqp_plugin and the stamp_inserted timestamp

2013-10-09 Thread Paolo Lucente
gt; "2013-10-09 02:05:01", "stamp_inserted": "2013-10-09 01:55:00", "packets": 0, > "bytes": 300} > > Complete config is as follows: > debug: true > > daemonize: false > nfacctd_port: 9996 > > nfacctd_disable_checks: true &

Re: [pmacct-discussion] amqp_plugin and the stamp_inserted timestamp

2013-10-08 Thread Paolo Lucente
Hi Nathan, I did try to reproduce your issue but with no luck. Can you say what capturing method are you using (libpcap, NetFlow, sFlow, or ..?). Is it that all data is mistakenly placed into the previous timeslot or it is possible you might be effectively receiving only bits of data belonging to

Re: [pmacct-discussion] Compiling 1.5.0rc1 on Debian 7.1 with RabbitMQ

2013-10-01 Thread Paolo Lucente
Hi Nathan, Thanks for having reported the issue. I'm already following it up with RabbitMQ people. Should have also already be given a possible fix for it - time to implement and test. Keep you posted as soon as this is done and code is committed to CVS. Cheers, Paolo On Tue, Oct 01, 2013 at 12:

Re: [pmacct-discussion] BGP multipath

2013-09-30 Thread Paolo Lucente
Hi, On Mon, Sep 30, 2013 at 03:48:15PM +0200, Ana Marija Banovac wrote: > I have a few questions - newbie ones. First one regarding BGP multipath. I > have it implemented in my network. I saw the official examples and it is > stated there that is not implemented. So, can I presume that is the rea

Re: [pmacct-discussion] Unable to account MPLS labeled packets

2013-09-30 Thread Paolo Lucente
Hi Marco, On Mon, Sep 30, 2013 at 03:30:09PM +0200, Marco Marzetti wrote: > If i replace the above pcap_filter with "mpls ( dst net 192.0.2.0/24 > or dst net 198.51.100.0/24 or dst net 203.0.113.0/24 )" i see a lot > of non-sense entries in the pipe: > > [ .. ] Solution to the former problem, i

Re: [pmacct-discussion] It says that i have not enabled NetFlow data export

2013-09-09 Thread Paolo Lucente
Hi Rey, Config is basic but good. 1) you can verify on the probe that NetFlow traffic is being generated to x.y.z.t:2055 (i'd say with Wireshark since it will also validate the packets if you decode them with the 'cflow' dissector); 2) you can verify on the collector box that NetFlow from the prob

Re: [pmacct-discussion] Trouble with pre_tag_filter and pre_tag_maps

2013-09-09 Thread Paolo Lucente
Hi Jonathan, This is actually great troubleshooting, thanks very much as it's going to help a lot solving the issue - much appreciated. Can you please send me privately a trace of a few NetFlow packets containing offending flows (also the NetFlow v9 template must be part of the trace)? I will use

Re: [pmacct-discussion] sfacctd byte accounting not accurate

2013-09-09 Thread Paolo Lucente
27; with ulog to provide flow samples ( > http://blog.sflow.com/2010/12/ulog.html ). > i suppose that pmacctd is more accurate because i understand and correct me > that it uses a promiscuous mode. tests are runing on a local machine. > thanks for replying so quickly. > > &

Re: [pmacct-discussion] sfacctd byte accounting not accurate

2013-09-08 Thread Paolo Lucente
Hi Oussama, Can you elaborate on your target deployment for billing (ie. sFlow or promiscuous mode)? Can you also elaborate on the setup for your test? What tool is generating sFlow data being sent to sfacctd? If you have control over the host 192.168.42.226: which of the two counters looks the mo

Re: [pmacct-discussion] Trouble with pre_tag_filter and pre_tag_maps

2013-09-06 Thread Paolo Lucente
Hi Jonathan, Very good test. That should rule out the possibility i had in mind this could be data between xx:xx:xx:ff:25:10 and xx:xx:xx:ff:25:11. You can do another test, to progress this one step further, to see whether the issue is on the nfprobe plugin or with nfacctd somehow mis-classifying

Re: [pmacct-discussion] Trouble with pre_tag_filter and pre_tag_maps

2013-09-05 Thread Paolo Lucente
Hi Jonathan, Since the four tags in pre_tag_map are mutual exclusive and it anyway works first-match-wins fashion, remove the 'return=true' part from the equation. Let me know if this solves anything. If not, it would be beneficial to know whether it's the probe which is mistaken tagging (which is

Re: [pmacct-discussion] CGN NEL nfacct shows nothing

2013-09-05 Thread Paolo Lucente
Hi Gregoire, The template looks allright, it contains NAT event among the other things. From the template it would look like you are exporting from ASR1K, ASA or similar - it would be great if you could confirm (as i don't see TCP/UDP ports mentioned). Can you please try with a different aggregati

Re: [pmacct-discussion] Error: 'sql_multi_values' is too small (100). Try with a larger value

2013-09-03 Thread Paolo Lucente
Hi Chris, Got you, the issue is now fixed: http://www.mail-archive.com/pmacct-commits@pmacct.net/msg00824.html Cheers, Paolo On Tue, Sep 03, 2013 at 01:28:54PM +0100, Chris Wilson wrote: > Hi Paolo, > > On Tue, 3 Sep 2013, Paolo Lucente wrote: > > >Maybe a bug in documentat

Re: [pmacct-discussion] Error: 'sql_multi_values' is too small (100). Try with a larger value

2013-09-03 Thread Paolo Lucente
Hi Chris, Maybe a bug in documentation in the release you are using? CONFIG-KEYS says: "The value of the directive is intended to be the size (in bytes) of the multi-values buffer.". So 100 bytes is on the low side, and by default MySQL comes with a 1MB buffer - after that you should tweak MySQL c

Re: [pmacct-discussion] Pro-rating netflow accounting data across time intervals

2013-09-01 Thread Paolo Lucente
Hi Edward, First, thanks for this exaustive email, very interesting. My first question to scope it better is whether you are using any sampling rate, and if yes how much. I ask because i'd intuitively say if a flow is created from a single sampled packet (which gets typical on most traffic, not al

[pmacct-discussion] pmacct 1.5.0rc1 released !

2013-08-29 Thread Paolo Lucente
VERSION. 1.5.0rc1 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (Mon

Re: [pmacct-discussion] WARN: Failed during write: No buffer space available

2013-08-14 Thread Paolo Lucente
Hi Krzysztof, I'm not expert of FreeBSD: can make sure the SO allows to set up a pipe of plugin_pipe_size? On Linux you would do it by touching the files /proc/sys/net/core/[rw]mem_max. Also, if you could check out the pmacct release currently in the CVS it behaves differently from the past where:

Re: [pmacct-discussion] sfacctd receiving sflow from 2 switches

2013-08-07 Thread Paolo Lucente
Hi Andrew, You making use of the peer_src_ip aggregation primitive to confirm which switch is sending which flow? If yes then i would have no idea: maybe you can post your config. If that gives no clue aswell, you can send me privately a few sFlow packets so to see what's going on. Cheers, Paolo

Re: [pmacct-discussion] Auto-reconnect to DB

2013-07-29 Thread Paolo Lucente
l_history of 1s, but dunno if it's related. > > > > On Fri, Jun 21, 2013 at 2:39 PM, George-Cristian Bîrzan > wrote: > > > I'll try to, but I'm not so sure it'll be trivial to reproduce. > > > > > > On Thu, Jun 20, 2013 at 8:09 PM,

Re: [pmacct-discussion] Some IPs not logging outbound traffic

2013-07-29 Thread Paolo Lucente
h_time: 300 > sql_history: 5m > sql_history_roundoff: m > sql_dont_try_update: true > sql_preprocess[in]: minp=10 > plugin_pipe_size: 4096000 > plugin_buffer_size: 10240 > > Just removed some IPs. Other than that it's the config we use. > > Best, > Martin >

Re: [pmacct-discussion] Some IPs not logging outbound traffic

2013-07-27 Thread Paolo Lucente
Hi Martin, Can you specify which plugin and pmacct version are you using? Was wondering if it could be a memory plugin which table fills up of entries, but you would definitely see signs of this in the logs. Maybe worth also posting your configuration. Cheers, Paolo On Fri, Jul 26, 2013 at 12:14

Re: [pmacct-discussion] Pmacct not adding the as number

2013-07-09 Thread Paolo Lucente
rue > > See the attached document for the dump of the flows that I am doing in the > flow collector. > > And an sample entry in the networks.lst file for one of the matches in the > file: > 29073,80.82.64.0/24 > > > > 2013/7/5 Paolo Lucente > > > xHi

Re: [pmacct-discussion] Changing main process title?

2013-07-06 Thread Paolo Lucente
, 2013 at 11:38:15PM +, Paolo Lucente wrote: > Hi Brian, Ruben, > > It's a good proposal, it makes sense. Will introduce a config > directive to allow to customize the core process name default > being, for backward compatibility, "default". If you have any > d

Re: [pmacct-discussion] Crash in pmacct

2013-07-05 Thread Paolo Lucente
Hi Joan, I can verify the backtrace you provided does not apply to the current (and 0.14.3 release to that matter) code. Also, the issue is related to querying the content of a networks_file - which is a part of the code that got some changes meanwhile. I propose you download/compile 0.14.3 releas

Re: [pmacct-discussion] Pmacct not adding the as number

2013-07-05 Thread Paolo Lucente
xHi Joan, Thanks for explaining the background, it makes sense. To get ASNs info populated you should add src_as and dst_as primitives to your aggregate directive. Same as any further info you wish to see populated. Let me know how that goes. I see you dropped a separate email about a crash, alo

Re: [pmacct-discussion] slow RDBMS? (mysql) how to aggregate best?

2013-07-04 Thread Paolo Lucente
gt; > But it makes sense if it would be possible :) (I have never tried to be > honest). > > Thanks, > Andras > > > > On Thu, Jul 4, 2013 at 1:47 AM, Paolo Lucente wrote: > > > Hi Andras, > > > > Inline: > > > > On Wed, Jul 03, 2013 a

Re: [pmacct-discussion] slow RDBMS? (mysql) how to aggregate best?

2013-07-03 Thread Paolo Lucente
Hi Andras, Inline: On Wed, Jul 03, 2013 at 01:29:38PM +0200, Andras Horvai wrote: > So in mongodb I have the historical data of connection and in mysql I have > the data what I can use for accounting (how many bytes a host receives and > sends). I would like to use one database backands but I d

Re: [pmacct-discussion] Changing main process title?

2013-07-03 Thread Paolo Lucente
Hi Brian, Ruben, It's a good proposal, it makes sense. Will introduce a config directive to allow to customize the core process name default being, for backward compatibility, "default". If you have any different ideas please let me know. Otherwise will give you a shout here once this is available

Re: [pmacct-discussion] Build fails to find libmysqlclient on 64-bit CentOS

2013-06-25 Thread Paolo Lucente
Hi Chris, Sure, thanks for the tip: makes sense, will do. Cheers, Paolo On Tue, Jun 25, 2013 at 03:13:58PM +0100, Chris Wilson wrote: > Hi Paolo, > > Configure fails to find /usr/lib64/mysql/libmysqlclient.so on 64-bit > CentOS. You might want to add that to the list of search directories > in

Re: [pmacct-discussion] adding NFv9 template fields

2013-06-25 Thread Paolo Lucente
Hi Osama, On Mon, Jun 24, 2013 at 01:29:07PM +0200, Osama Abuelsorour wrote: > We already export #234 and #235 capturing the ingress and egress VRF ID > respectively (Cisco). You are right, they are 32 bit integers. I did link the > results with SNMP (Cisco) OID 1.3.6.1.4.1.9.9.711.1.1.1.1.2 to

Re: [pmacct-discussion] adding NFv9 template fields

2013-06-20 Thread Paolo Lucente
ght just add them myself (unless you are > already working on that). > > Thanks again for the great work! > > On Jun 9, 2013, at 12:49 AM, Paolo Lucente wrote: > > > Briefly to follow-up on this, to say: > > > > * support has now been introduced for MPLS_TOP_

Re: [pmacct-discussion] Traffic cut off at 50GByte/min

2013-06-20 Thread Paolo Lucente
Hi Reto, First thing that comes to mind is: you are running pre 0.14.3 version without having enabled 64bit counters at configure time (from 0.14.3 this is enabled by default). Can this be the case? Otherwise it does not ring a bell to me and it does not smell you need to increase any caches. Anot

Re: [pmacct-discussion] Auto-reconnect to DB

2013-06-20 Thread Paolo Lucente
Hi George-Cristian, One or more plugins that bail out and consequently core process that closes up after all plugins are gone (essentially, the message you posted) could be symptom of plugins crashing for some reason. It can help if you run the daemon under gdb with follow-fork-mode set to child a

Re: [pmacct-discussion] distingusih by netflow send

2013-06-17 Thread Paolo Lucente
Hi Andras, Yes. First let me remind you can distinguish which NetFlow sender is generating the flows by enabling the peer_src_ip primitive - just in case this is what you are trying to accomplish. A config that does what you want follows, consider a MySQL plugin can write to a single MySQL DB/tabl

Re: [pmacct-discussion] help: nfacctd bind() to ip=(null) port=5678/udp failed (errno: 48).

2013-06-14 Thread Paolo Lucente
Hi Ronald, It would help if you could look up what errno #48 is on your FreeBSD. >From some online docs it looks like it could be EADDRINUSE - so, for example, can it be the case multiple nfacctd instances are trying to bind the same IP address, port? Or maybe you implemented something to automati

Re: [pmacct-discussion] Received malformed BGP packet (marker check failed).

2013-06-10 Thread Paolo Lucente
Hi Felix, Great to know this is solved and thanks for offering to investigate this further but since it's both an old release and a pre-compiled binary there is not much benefit in the exercise. Don't hesitate to get in touch should you run into any further issues. Cheers, Paolo On Mon, Jun 10,

Re: [pmacct-discussion] Received malformed BGP packet (marker check failed).

2013-06-10 Thread Paolo Lucente
Hi Felix, Can't see anything wrong in your configs & this should just work for you. Is what you are seeing deterministic, would you be able to tell me how to reproduce? Are you using a recent version of pmacct (and Quagga)? I have seen that error only during development, never once the BGP deamon

Re: [pmacct-discussion] adding NFv9 template fields

2013-06-08 Thread Paolo Lucente
Hi Osama, On Thu, May 09, 2013 at 11:13:56AM +0200, Osama Abuelsorour wrote: > What we are trying to do is to gain insights about traffic from the PE router > towards our core network by applying NetFlow on the aggregate interface > towards the core. Ideally, we want to use the MPLS tags to be

Re: [pmacct-discussion] Tips on dealing with overflowing 32-bit fields?

2013-05-25 Thread Paolo Lucente
Hi Ed, Although you might be running into other typical C7600 issues with NetFlow, ie. inaccuracy introduced by NetFlow TCAM space exhaustion (especially if your RSP720 is not XL series), I concur with Brent suggestion to first of all try enabling sampling. Also i'd definitely recommend to switch

Re: [pmacct-discussion] Table for outbound traffic stays empty

2013-05-24 Thread Paolo Lucente
Hi Bjorn, Is it possible part of the traffic, the outbound one, is VLAN tagged? It's the only thing that comes to mind; if this is the case you can solve it by rewriting the current filter as: "vlan and src net 95.211.55.128/26" Let me know. If it does not ring a bell and/or solve, can you plea

Re: [pmacct-discussion] Few questions regarding MySQL, IPFIX and sFlow

2013-05-20 Thread Paolo Lucente
m the F5 gear. > > Cheers, > Seamus > > -Original Message- > From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On > Behalf Of Paolo Lucente > Sent: Friday, May 17, 2013 7:02 PM > To: pmacct-discussion@pmacct.net > Subject: Re: [pmacct-dis

Re: [pmacct-discussion] Few questions regarding MySQL, IPFIX and sFlow

2013-05-17 Thread Paolo Lucente
Hi Seamus, About your two issues: * 115446 records in a single table are not critical at all, would not expect poor performances from MySQL even on old/downsized hardware. Maybe you need to be more specific, ie. where do you verify poor performance - on insert, on query? Your config is bas

Re: [pmacct-discussion] some noob questions

2013-05-17 Thread Paolo Lucente
Hi Vito, On Wed, May 15, 2013 at 05:00:29PM +0200, j...@live.com wrote: > At the moment I'm using the nfprobe_plugin + nfacctd to collect the > netflow data (btw: to fullfill the vlan field I need Netflow v9, right? > ) and save them into the db, Yes, in order to have the VLAN field filled you h

Re: [pmacct-discussion] some noob questions

2013-05-15 Thread Paolo Lucente
Hi Vito, On Tue, May 14, 2013 at 04:06:14PM +0200, j...@live.com wrote: > It is possible to account flow data (I mean the start and the end of a > connection flow) without running the couple nfacctd + pmacctd? sry for > this silly question ... Are you using pmacctd with nfprobe plugin (to create

Re: [pmacct-discussion] pmacct-contribs now on GitHub

2013-05-14 Thread Paolo Lucente
Hi Vincent, George-Cristian, Have no plans currently but let's see if it makes sense to do something like that. For pmacct-contribs the resoning is intuitive since i'm really just a maintainer for it - 100% so far is contributed and what is grouped under that umbrella is maybe 5%, if not less, of

[pmacct-discussion] pmacct-contribs now on GitHub

2013-05-11 Thread Paolo Lucente
Dears, A brief announcement to say pmacct-contribs, the effort to put together 3rd party contributions to the pmacct project (scripts, tools, frontends, etc.), is now published on GitHub. This is in order to facilitate and encourage sharing of new contributions as well as to try reducing scatterin

Re: [pmacct-discussion] some noob questions

2013-05-10 Thread Paolo Lucente
Hi Vito, On Fri, May 10, 2013 at 06:13:46PM +0200, j...@live.com wrote: > Interesting, I missed this feature out as I'm actually using the debian > packet (version 0.14.0.1) and if I'm right it's was introduces in the > last version of your software, specifically 0.14.3. Correct. > How would th

Re: [pmacct-discussion] some noob questions

2013-05-09 Thread Paolo Lucente
Hi Vito, On Thu, May 09, 2013 at 11:36:21AM +0200, j...@live.com wrote: > So now, my concerns are about multiple connection with the same key that > I've reduced to > PRIMARY KEY (vlan, ip_src, ip_dst, src_port, dst_port, ip_proto) > what happens if two connections with the same key set are opene

Re: [pmacct-discussion] adding NFv9 template fields

2013-05-09 Thread Paolo Lucente
Hi Osama, On Thu, May 09, 2013 at 11:13:56AM +0200, Osama Abuelsorour wrote: > What we are trying to do is to gain insights about traffic from the PE router > towards our core network by applying NetFlow on the aggregate interface > towards the core. Ideally, we want to use the MPLS tags to be

Re: [pmacct-discussion] Patch for mktime() call to cover DST roll over

2013-05-08 Thread Paolo Lucente
Hi Matthew, What you suggest makes sense. Let me review and get back to you - for now thanks very much for contributing your patch. Cheers, Paolo On Tue, May 07, 2013 at 09:40:34AM +1200, Matthew Grant wrote: > Hi! > > Adding some more details > > On 06/05/13 16:01, Matthew Grant wrote: > > H

Re: [pmacct-discussion] some noob questions

2013-05-08 Thread Paolo Lucente
Hi Vito, Inline: On Wed, May 08, 2013 at 06:50:43PM +0200, j...@live.com wrote: > [ .. ] > > 1) is possible to have a roundoff value as milliseconds? Can you elaborate on this? sql_history (temporal aggregation feature) stops to seconds resolution. You looking for sub-second temporal aggregatio

Re: [pmacct-discussion] adding NFv9 template fields

2013-05-05 Thread Paolo Lucente
Hi Osama, Very good to hear from you. About the UDP ports: field types #180 and #181 are supported but the fact the template contains both these two and their traditional counterparts, field types #7 and #11, kills the heuristics currently in place to decide which one to pick. What is the reason b

Re: [pmacct-discussion] pmacct and SNMP interface ID in as-stats

2013-05-04 Thread Paolo Lucente
2=100 label=input > id2=200 label=output > > > By the way, when I configure (like in "QUICKSTART chapter X") the > nfprobe_direction and nfprobe_ifindex without [inbound/outbound] > name there is an error occured in running pmacctd: > [...] > nfprobe_direct

Re: [pmacct-discussion] pmacct and SNMP interface ID in as-stats

2013-05-03 Thread Paolo Lucente
Hi Marek, It seems you want nfprobe_ifindex and/or nfprobe_direction features; you can read brief description in CONFIG-KEYS, some more explanation about them in QUICKSTART chapter X - where you can also find a couple of examples. Depending on the specific scenario you might want to keep it simpl

[pmacct-discussion] pmacct 0.14.3 released !

2013-05-03 Thread Paolo Lucente
VERSION. 0.14.3 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected data into memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (Mongo

Re: [pmacct-discussion] pmacct hangs after a while

2013-05-02 Thread Paolo Lucente
Hi Timur, On Thu, May 02, 2013 at 05:06:27PM +0500, Timur Irmatov wrote: > No, links are stable. There is no link down/up events in kernel log. > Both processes just hang. I tried attaching to them via strace, one of > them was stuck in futex call, another one shows up as in > restart_syscall. O

Re: [pmacct-discussion] pmacct hangs after a while

2013-05-02 Thread Paolo Lucente
Hi Timur, If both processes drop to zero CPU utilization then it looks like the issue might be in what is feeding pmacct. Although pmacctd protects from interface flaps (ie. if the interface drops, it tries to re-bind) can you check your system logs to spot if there has been any link down-ups? Wha

Re: [pmacct-discussion] pmacct and quagga

2013-05-02 Thread Paolo Lucente
Hi Marek, I've just tested against Quagga 0.99.22 and it works no problem peering with a pmacct instance over 127.0.0.1. I'd suggest upgrading quagga but it's strange: i did test this back in 2009-2010; 0.99.20.1 is from last year; 0.99.22 is from 2013. Don't believe something did break temporarly

<    2   3   4   5   6   7   8   9   10   11   >