Hi Ruben, Let me follow up privately for some troubleshooting on this.
Cheers, Paolo On Fri, Aug 28, 2015 at 02:13:24PM +0200, Ruben Laban wrote: > Hi Paolo, > > Either I'm doing something wrong or there's a bug in this part of > pmacct. Current config: > > ! > debug: false > ! > daemonize: true > ! > core_proc_name: bond0 > ! > aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host, > src_port, dst_port > aggregate[bond0-aggr]: none > ! > plugins: print[bond0-all], print[bond0-aggr] > ! > !plugin_buffer_size: 1024 > !plugin_buffer_size: 4096 > !plugin_pipe_size: 10240000 > !plugin_pipe_size: 40960000 > plugin_pipe_amqp: true > ! > !plugin_pipe_amqp_routing_key[bond0-all]: bond0-all > !plugin_pipe_amqp_routing_key[bond0-aggr]: bond0-aggr > ! > files_umask: 027 > !files_uid: 0 > files_gid: 112 > ! > interface: bond0.1234 > ! > promisc: false > ! > syslog: daemon > ! > pidfile: /var/run/pmacctd-bond0.pid > ! > print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv > print_output_file[bond0-aggr]: /tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv > ! > print_refresh_time: 60 > ! > print_time_roundoff: m > ! > print_history: 1m > ! > print_cache_entries: 323333 > !print_cache_entries: 100003 > ! > print_output_file_append: true > ! > print_markers: true > ! > !print_output: json > print_output: csv > > > I think there's something wrong with the > plugin_pipe_amqp_routing_key stuff. If I set it to some value, the > data vanishes. If I leave it to default, all data ends up in a > single pile (data for both plugins get written to > /tmp/pmacct-bond0-all-*.csv). > > These are my first steps in AMQP/RabbitMQ land, so it's quite > possible I'm doing something wrong. This is just a default > installation of rabbitmq-server on Ubuntu 14.04 (localhost, guest, > etc). > > Regards, > Ruben Laban > > On 28-8-2015 10:42, Ruben Laban wrote: > >Hi Paolo, > > > >Strange, the bmp/ dir does show up in a fresh checkout (and nightly > >tarball, and git). For some reason my previous checkout refused to pull > >in that dir. I'll continue testing the rabbitmq stuff now. > > > >Regards, > >Ruben Laban > > > >On 27-8-2015 12:03, Paolo Lucente wrote: > >>Hi Ruben, > >> > >>Can you give it another try? I don't manage to reproduce this issue > >>with the CVS - all appears to work good for me. As a workaround, should > >>you still have problems, you can download the latest daily package from > >>the webapage: http://www.pmacct.net/pmacct-daily.tar.gz . As yet another > >>alternative you can check out latest code on GitHub. Please let me know. > >> > >>Cheers, > >>Paolo > >> > >>On Wed, Aug 26, 2015 at 09:41:29AM +0200, Ruben Laban wrote: > >>>Hi Paolo, > >>> > >>>I just did a fresh CVS checkout and tried to build it. Alas, it > >>>failed. It complains about not finding ../bmp/bmp.h as referenced in > >>>bgp/bgp_logdump.c. And my checkout doesn't even have that directory > >>>(bmp). > >>> > >>>Regards, > >>>Ruben > >>> > >>>On 26-8-2015 01:02, Paolo Lucente wrote: > >>>>Hi Ruben, > >>>> > >>>>Yes, there is some config. You can read chapter "Internal buffering and > >>>>queueing" of the QUICKSTART document: > >>>> > >>>>https://github.com/paololucente/pmacct/blob/master/pmacct/QUICKSTART > >>>> > >>>>Essentially plugin_pipe_size becomes something not to care anymore and > >>>>plugin_pipe_amqp goes to 'true'. Then any extra directive is to config > >>>>aspects to connect to the RabbitMQ exchange (user, password, etc.). > >>>> > >>>>Cheers, > >>>>Paolo > >>>> > >>>>On Sun, Aug 23, 2015 at 11:44:31PM +0200, Ruben Laban wrote: > >>>>>Hi Paolo, > >>>>> > >>>>>This sure sounds interesting. I'll see if I can get around to > >>>>>deploying latest CVS to my test setup asap. Does the change of the > >>>>>queuing mechanism introduce a change in configuration as well? Like > >>>>>new/changed options, different sizing recommendations, etc. > >>>>> > >>>>>Regards, > >>>>>Ruben Laban > >>>>> > >>>>>On 22-8-2015 19:41, Paolo Lucente wrote: > >>>>>>Hi Ruben, > >>>>>> > >>>>>>Your email is very timely and i understand such fluctuations > >>>>>>between low > >>>>>>and high traffic periods can happen in a libpcap deployment. A new > >>>>>>feature > >>>>>>that has been introduced as part of 1.5.2 (which is currently in > >>>>>>the CVS > >>>>>>and about to be released) is passing buffers inside pmacct - so > >>>>>>from the > >>>>>>core process to the plugins - using a RabbitMQ broker, as an > >>>>>>alternative > >>>>>>to the homegrown queueing infrastructure. This should help you > >>>>>>working > >>>>>>with small buffers (so that data does not get "delayed" in case of > >>>>>>low > >>>>>>traffic times) while keeping the queue size dynamic. > >>>>>> > >>>>>>Let me know if this looks of interest to you and, if yes, whether > >>>>>>maybe > >>>>>>you have room to give it a try before 1.5.2 release, say > >>>>>>throughout next > >>>>>>week. > >>>>>> > >>>>>>Cheers, > >>>>>>Paolo > >>>>>> > >>>>>>On Fri, Aug 21, 2015 at 08:04:21AM +0200, Ruben Laban wrote: > >>>>>>>Hi, > >>>>>>> > >>>>>>>Lately I've been experimenting a lot with plugin_buffer_size, > >>>>>>>plugin_pipe_size and print_cache_size. My current conclusion is that > >>>>>>>it is in no way possible to tune pmacct in such a way that it will > >>>>>>>deal with low and (very) high traffic levels accurately: > >>>>>>> > >>>>>>>* Tune for low traffic: loss of data at high traffic > >>>>>>>* Tune for high traffic: data gets "delayed" at low traffic > >>>>>>> > >>>>>>>Am I missing something here is this something I'll have to live with > >>>>>>>(or otherwise work around). > >>>>>>> > >>>>>>>What I was hoping for was that by using large buffers (to > >>>>>>>accommodate high traffic levels) I'd get to deal properly with low > >>>>>>>and high traffic levels. I kinda expected buffers to flushed > >>>>>>>whenever data is being flushed to disk (using print plugin right > >>>>>>>now). This does not seem to be the case (completely). > >>>>>>>The fact that tuning for high traffic levels requires (a lot) more > >>>>>>>memory is easier to live with, memory's cheap after all. > >>>>>>> > >>>>>>>Some background info: > >>>>>>>Ubuntu 14.01 > >>>>>>>Pmacct 1.5.1 > >>>>>>>libpcap, without PF_RING or anything > >>>>>>> > >>>>>>>Current config: > >>>>>>> > >>>>>>>! > >>>>>>>debug: false > >>>>>>>! > >>>>>>>daemonize: true > >>>>>>>! > >>>>>>>core_proc_name: bond0 > >>>>>>>! > >>>>>>>aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host, > >>>>>>>src_port, dst_port > >>>>>>>aggregate[bond0-aggr]: none > >>>>>>>! > >>>>>>>plugins: print[bond0-all], print[bond0-aggr] > >>>>>>>! > >>>>>>>!plugin_buffer_size: 1024 > >>>>>>>!plugin_buffer_size: 4096 > >>>>>>>!plugin_pipe_size: 10240000 > >>>>>>>plugin_pipe_size: 40960000 > >>>>>>>! > >>>>>>>files_umask: 027 > >>>>>>>!files_uid: 0 > >>>>>>>files_gid: 112 > >>>>>>>! > >>>>>>>interface: bond0.1234 > >>>>>>>! > >>>>>>>promisc: false > >>>>>>>! > >>>>>>>syslog: daemon > >>>>>>>! > >>>>>>>pidfile: /var/run/pmacctd-bond0.pid > >>>>>>>! > >>>>>>>print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv > >>>>>>>print_output_file[bond0-aggr]: > >>>>>>>/tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv > >>>>>>>! > >>>>>>>print_refresh_time: 60 > >>>>>>>! > >>>>>>>print_time_roundoff: m > >>>>>>>! > >>>>>>>print_history: 1m > >>>>>>>! > >>>>>>>print_cache_entries: 323333 > >>>>>>>!print_cache_entries: 100003 > >>>>>>>! > >>>>>>>print_output_file_append: true > >>>>>>>! > >>>>>>>print_markers: true > >>>>>>>! > >>>>>>>!print_output: json > >>>>>>>print_output: csv > >>>>>>> > >>>>>>>Regards, > >>>>>>>Ruben Laban > >>>>>>> > >>>>>>>_______________________________________________ > >>>>>>>pmacct-discussion mailing list > >>>>>>>http://www.pmacct.net/#mailinglists > >>>>>> > >>>>>>_______________________________________________ > >>>>>>pmacct-discussion mailing list > >>>>>>http://www.pmacct.net/#mailinglists > >>>>>> > >>>>> > >>> > > > > > >_______________________________________________ > >pmacct-discussion mailing list > >http://www.pmacct.net/#mailinglists > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
