Re: [pmacct-discussion] Getting nfacctd to NOT aggregate ?

[Dariush Marsh-Mossadeghi]

A… aggregating on something which can’t be aggregated. Nice hack :-)
Thanks Robert

> On 19 Apr 2016, at 15:26, Robert Juric <robert.ju...@gmail.com> wrote:
> 
> I found you have to build the tables with timestamps and then when you 
> aggregate with timestamp_start and timestamp_end you can get the individual 
> flow records as opposed to aggregating the records.
> 
> Robert Juric
> 
> 
> 
> On Tue, Apr 19, 2016 at 9:00 AM, Dariush Marsh-Mossadeghi 
> <dari...@gravitas.co.uk <mailto:dari...@gravitas.co.uk>> wrote:
> Hi List,
> 
> Is there a way to get pmacctd/nfacctd to NOT do any aggregation of flow 
> records ?
> 
> Specifically,  I’ve got IPFIX coming off a router being handed by nfacctd and 
> it would be useful to temporarily have visibility of every flow record. 
> Tcpdump is not a suitable as part of what I’m trying to ascertain is whether 
> the IPFIX data is accurate.
> 
> Any suggestions/recipes/config snippets/pointers to RTFM would be gratefully 
> received.
> 
> Thanks
> Dariush



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] collecting large number of netflows

[Dariush Marsh-Mossadeghi]

For that sort of volume I’d investigate using and ELK stack on the backend and 
de-coupling pmacct from storage and analysis with a message queue like Rabbit. 
I’ve used this approach (not to that scale though) and it works well.

ELK == ElasticSearch Logstash Kibana

> 
> 
> On 08/04/2016 11:48 AM, David McKen wrote:
>> For that type of scale maybe a SQL like NoSQL db like cassandra may work 
>> better for you.
>> 
>> On Thu, Aug 4, 2016 at 11:01 AM, Stephen Clark > > wrote:
>> Hi List,
>> 
>> I am looking to collect a large number of netflow records, on the order of a 
>> 100 million a day,
>> and store them in a postgres DB. Has anyone done this or something similar 
>> using pmacct?
>> 
>> Thanks,
>> Steve
>> 
>> 
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists 
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

[Dariush Marsh-Mossadeghi]

> On 25 Jun 2018, at 11:54, Alex K  wrote:
> 
> Hi all,
> 
> I have a setup with uacctd monitoring traffic of several interfaces through 
> NFLOG.
> With uacctd stopped I see that the server (a relatively small device with 4 
> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
> 
> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
> 
> Thanx,
> Alex
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


tldr; post the output of the free command when uacctd is running and I’ll do my 
best to interpret it for you :-)

Is it _really_ using it, or is it just not completely unused?

The linux kernel generally does pretty good job of keeping stuff that might be 
useful in memory, but getting rid of it very very quickly if the space is 
needed for something else.
The challenge you face is that most of the userspace tools come with a long 
list of caveats about what they appear to report vs what’s really happening, 
this is mainly due to the way the kernel shares memory between processes, and 
its use of as much memory as possible for various caches.

Some threads worth reading if you’re not familiar with the in’s and out’s of 
linux kernel memory management...

https://www.linuxatemyram.com/ 
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage
 

https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
 


HTH
Dariush


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

[Dariush Marsh-Mossadeghi]

OK, so we’re moving from bottom-posting to top-posting… that’ll make it 
interesting for other readers ;-)

The output of free doesn’t look desperate, but it is starting to look a bit 
tight.
You’ve got about a gig of buffer/cache, which the kernel will evict if it needs 
it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in use, that may or 
may not be a problem depending on what else is running on the box and how it’s 
memory use varies over time.

I’m not intimatley familiar with the internals of uacctd’s mysql plugins, but 
my advice would be:

- Monitor the memory usage. If it doesn’t vary much over time you could go on 
for years and be just fine. Keep a close watch on swap usage, if that varies a 
lot or grows over time you’ll want to do something about it.

- Read up on OOM Killer, what it does and how it behaves. If you see OOM Killer 
entries in your logs, it’s time to think again

- Can you put some more memory in the box? 8GB of RAM will cost you about £60. 
How much is your time worth ?

HTH
Dariush


> On 25 Jun 2018, at 12:32, Alex K  wrote:
> 
> Thanx for the reply.
> 
> The output of free is the following:
> 
> free
> totalusedfreeshared buff/cache   
> available
> Mem:4046572 2832576  152012  784240 1061984  
> 204248
> Swap:   3906556 1086080 2820476
> 
> While is as below when stopped:
> 
> free
>   totalusedfree  shared  buff/cache   
> available
> Mem:4046572  520352 3223884   14916  302336 
> 3286952
> Swap:   3906556  485040 3421516
> 
> Seems that mysql plugins are reserving quite some memory as they list first 
> in htop when sorted with memory.
> 
> Thanx,
> Alex
> 
> 
> On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi 
> mailto:dari...@gravitas.co.uk>> wrote:
> 
>> On 25 Jun 2018, at 11:54, Alex K > <mailto:rightkickt...@gmail.com>> wrote:
>> 
>> Hi all,
>> 
>> I have a setup with uacctd monitoring traffic of several interfaces through 
>> NFLOG.
>> With uacctd stopped I see that the server (a relatively small device with 4 
>> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
>> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>> 
>> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
>> 
>> Thanx,
>> Alex
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
> 
> tldr; post the output of the free command when uacctd is running and I’ll do 
> my best to interpret it for you :-)
> 
> Is it _really_ using it, or is it just not completely unused?
> 
> The linux kernel generally does pretty good job of keeping stuff that might 
> be useful in memory, but getting rid of it very very quickly if the space is 
> needed for something else.
> The challenge you face is that most of the userspace tools come with a long 
> list of caveats about what they appear to report vs what’s really happening, 
> this is mainly due to the way the kernel shares memory between processes, and 
> its use of as much memory as possible for various caches.
> 
> Some threads worth reading if you’re not familiar with the in’s and out’s of 
> linux kernel memory management...
> 
> https://www.linuxatemyram.com/ <https://www.linuxatemyram.com/>
> https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage
>  
> <https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage>
> https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
>  
> <https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>
> 
> HTH
> Dariush
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists>
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

[Dariush Marsh-Mossadeghi]

OK, so it’s effectively an embedded system scenario, with a fixed config 
hardware, or similar
No silver bullets or one-liner fixes here :-\

You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about kernel 
tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented distros, rather 
than OOTB Debian. Although Debian is skinnier than most, there’s a lot that can 
be done to minimise footprint. The fact that you’ve got a 64bit Debian9 running 
on it makes me fairly optimistic that you could run pretty much any Debian 
derivative.
- Profiling your userspace software stack and see if there are any dependencies 
you don’t need, strip them out. Whether you can do this will very much depend 
on the modularity of the components.

All of the above lead you down the path of maintaining forks and compiling your 
own packages, to a greater or lesser extent. That’s a maintenance overhead you 
want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of units, the 
economics of software maintenance may stack up for you.

HTH
Dariush


> On 25 Jun 2018, at 14:32, Alex K  wrote:
> 
> Let me change the posting style... :)
> 
> On Mon, Jun 25, 2018 at 3:51 PM, Dariush Marsh-Mossadeghi 
> mailto:dari...@gravitas.co.uk>> wrote:
> OK, so we’re moving from bottom-posting to top-posting… that’ll make it 
> interesting for other readers ;-)
> 
> The output of free doesn’t look desperate, but it is starting to look a bit 
> tight.
> You’ve got about a gig of buffer/cache, which the kernel will evict if it 
> needs it.
> You’ve got 200M of genuinely free memory.
> What is potentially a little concerning is the gig of swap in use, that may 
> or may not be a problem depending on what else is running on the box and how 
> it’s memory use varies over time.
> 
> I’m not intimatley familiar with the internals of uacctd’s mysql plugins, but 
> my advice would be:
> 
> - Monitor the memory usage. If it doesn’t vary much over time you could go on 
> for years and be just fine. Keep a close watch on swap usage, if that varies 
> a lot or grows over time you’ll want to do something about it.
> Monitoring of last 2 weeks is showing that free mem is fluctuating form 50M 
> to 220 MB and swaps remains steady. Thats why I am trying to find a solution.
> 
> 
> - Read up on OOM Killer, what it does and how it behaves. If you see OOM 
> Killer entries in your logs, it’s time to think again
> I am aware of that. thanx
> 
> - Can you put some more memory in the box? 8GB of RAM will cost you about 
> £60. How much is your time worth ?
> This is not an option. What I have is 4 GB. This is not just a personal 
> project. There are going to be thousands of such installations
> 
> HTH
> Dariush
> 
> 
>> On 25 Jun 2018, at 12:32, Alex K > <mailto:rightkickt...@gmail.com>> wrote:
>> 
>> Thanx for the reply.
>> 
>> The output of free is the following:
>> 
>> free
>> totalusedfreeshared buff/cache   
>> available
>> Mem:4046572 2832576  152012  784240 1061984  
>> 204248
>> Swap:   3906556 1086080 2820476
>> 
>> While is as below when stopped:
>> 
>> free
>>   totalusedfree  shared  buff/cache   
>> available
>> Mem:4046572  520352 3223884   14916  302336     
>> 3286952
>> Swap:   3906556  485040 3421516
>> 
>> Seems that mysql plugins are reserving quite some memory as they list first 
>> in htop when sorted with memory.
>> 
>> Thanx,
>> Alex
>> 
>> 
>> On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi 
>> mailto:dari...@gravitas.co.uk>> wrote:
>> 
>>> On 25 Jun 2018, at 11:54, Alex K >> <mailto:rightkickt...@gmail.com>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I have a setup with uacctd monitoring traffic of several interfaces through 
>>> NFLOG.
>>> With uacctd stopped I see that the server (a relatively small device with 4 
>>> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
>>> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>>> 
>>> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
>>> 
>>> Thanx,
>>> Alex
>>> ___
>>> pmacct-discussion mailing list
>>> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists&

Re: [pmacct-discussion] uacctd memory usage

[Dariush Marsh-Mossadeghi]

Rasto's comments got me thinking…

Not being privy to your application and it’s architecture this may not work for 
you at all.

I’ve had success in the past using rabbitmq to offload flow logs.
We used it to deliver netflow to ELK stacks running on AWS, and it worked 
really well.
It has the major advantage of operational resilience.

Something like this:

netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path 
across public internet -> rabbitmq -> logstash -> elasticsearch

as ever YMMV :-)
Dariush


> On 25 Jun 2018, at 16:53, Alex K  wrote:
> 
> 
> 
> On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt  <mailto:pho...@axfr.org>> wrote:
> Hello,
> 
> If there is no more RAM available, i would test sqlite3 plugin, as
> sqlite3 is more suited for limited resources usage. Youl will possibly
> need to change your workflow to export sqlite3 files and load it
> somewhere else, but it should be a lot cheaper memory wise.
> 
> Thanx. Seems that I have to stick with mysql at least at this phase.
> 
> This one is swapping pretty heavily, i would suggest to set to around 10
> for 4GB RAM.
> 
> sysctl vm.swappiness=10 so it will start using swap when available RAM
> is around 400MB.
> This is already done.
> 
> 
> r.
> 
> On 06/25/2018 05:35 PM, Alex K wrote:
> > Thanks Dariush. Appreciate your feedback.
> >
> > I was testing several stripped down kernels by compiling and removing
> > most of unused modules. The gain I had was in the range of few MB.
> > Seems that I have to find a more elegant approach at uacctd
> > configuration since with current setup I am loading several mysql
> > plugins so as to be able to filter traffic with directions, networks and
> > ports.
> > I was testing an edge case where these plugins may rise up to 210
> > instances thus leading to this memory usage. Normally this will not be
> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
> > Thus in normal cases seems that I am ok.
> >
> > Cheers
> >
> >
> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
> > mailto:dari...@gravitas.co.uk> 
> > <mailto:dari...@gravitas.co.uk <mailto:dari...@gravitas.co.uk>>> wrote:
> >
> > OK, so it’s effectively an embedded system scenario, with a fixed
> > config hardware, or similar
> > No silver bullets or one-liner fixes here :-\
> >
> > You’ve a number of options to slim down your memory footprint
> > - Strip Debian of all the packages you don’t need, learn a lot about
> > kernel tuning, and tune the kernel to your needs.
> > - Start looking at one of the more embedded systems oriented
> > distros, rather than OOTB Debian. Although Debian is skinnier than
> > most, there’s a lot that can be done to minimise footprint. The fact
> > that you’ve got a 64bit Debian9 running on it makes me fairly
> > optimistic that you could run pretty much any Debian derivative.
> > - Profiling your userspace software stack and see if there are any
> > dependencies you don’t need, strip them out. Whether you can do this
> > will very much depend on the modularity of the components.
> >
> > All of the above lead you down the path of maintaining forks and
> > compiling your own packages, to a greater or lesser extent. That’s a
> > maintenance overhead you want to avoid unless you have to.
> > Having said all that, if you’re looking at deploying thousands of
> > units, the economics of software maintenance may stack up for you.
> >
> > HTH
> > Dariush
> >
> >
> >> On 25 Jun 2018, at 14:32, Alex K  >> <mailto:rightkickt...@gmail.com>
> >> <mailto:rightkickt...@gmail.com <mailto:rightkickt...@gmail.com>>> 
> >> wrote:
> >>
> >> Let me change the posting style... :)
> >>
> >> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
> >> Marsh-Mossadeghi  >> <mailto:dari...@gravitas.co.uk>
> >> <mailto:dari...@gravitas.co.uk <mailto:dari...@gravitas.co.uk>>> wrote:
> >>
> >> OK, so we’re moving from bottom-posting to top-posting…
> >> that’ll make it interesting for other readers ;-)
> >>
> >> The output of free doesn’t look desperate, but it is starting
> >> to look a bit tight.
> >> You’ve got about a gig of buffer/cache, which the kernel will
> >> evict if it needs it.
> >> You’ve got 200M of genuinely free memory.
> >>