Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Dariush Marsh-Mossadeghi

> On 25 Jun 2018, at 11:54, Alex K  wrote:
> 
> Hi all,
> 
> I have a setup with uacctd monitoring traffic of several interfaces through 
> NFLOG.
> With uacctd stopped I see that the server (a relatively small device with 4 
> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
> 
> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
> 
> Thanx,
> Alex
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


tldr; post the output of the free command when uacctd is running and I’ll do my 
best to interpret it for you :-)

Is it _really_ using it, or is it just not completely unused?

The linux kernel generally does pretty good job of keeping stuff that might be 
useful in memory, but getting rid of it very very quickly if the space is 
needed for something else.
The challenge you face is that most of the userspace tools come with a long 
list of caveats about what they appear to report vs what’s really happening, 
this is mainly due to the way the kernel shares memory between processes, and 
its use of as much memory as possible for various caches.

Some threads worth reading if you’re not familiar with the in’s and out’s of 
linux kernel memory management...

https://www.linuxatemyram.com/ 
https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage
 

https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
 


HTH
Dariush


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Alex K
Let me change the posting style... :)

On Mon, Jun 25, 2018 at 3:51 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

> OK, so we’re moving from bottom-posting to top-posting… that’ll make it
> interesting for other readers ;-)
>
> The output of free doesn’t look desperate, but it is starting to look a
> bit tight.
> You’ve got about a gig of buffer/cache, which the kernel will evict if it
> needs it.
> You’ve got 200M of genuinely free memory.
> What is potentially a little concerning is the gig of swap in use, that
> may or may not be a problem depending on what else is running on the box
> and how it’s memory use varies over time.
>
> I’m not intimatley familiar with the internals of uacctd’s mysql plugins,
> but my advice would be:
>
> - Monitor the memory usage. If it doesn’t vary much over time you could go
> on for years and be just fine. Keep a close watch on swap usage, if that
> varies a lot or grows over time you’ll want to do something about it.
>
Monitoring of last 2 weeks is showing that free mem is fluctuating form 50M
to 220 MB and swaps remains steady. Thats why I am trying to find a
solution.


> - Read up on OOM Killer, what it does and how it behaves. If you see OOM
> Killer entries in your logs, it’s time to think again
>
I am aware of that. thanx

>
> - Can you put some more memory in the box? 8GB of RAM will cost you about
> £60. How much is your time worth ?
>
This is not an option. What I have is 4 GB. This is not just a personal
project. There are going to be thousands of such installations...

HTH
Dariush


On 25 Jun 2018, at 12:32, Alex K  wrote:

Thanx for the reply.

The output of free is the following:

free
totalusedfreeshared buff/cache
available
Mem:4046572 2832576  152012  784240 1061984
204248
Swap:   3906556 1086080 2820476

While is as below when stopped:

free
  totalusedfree  shared  buff/cache
available
Mem:4046572  520352 3223884   14916  302336
3286952
Swap:   3906556  485040 3421516

Seems that mysql plugins are reserving quite some memory as they list first
in htop when sorted with memory.

Thanx,
Alex


On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

>
> On 25 Jun 2018, at 11:54, Alex K  wrote:
>
> Hi all,
>
> I have a setup with uacctd monitoring traffic of several interfaces
> through NFLOG.
> With uacctd stopped I see that the server (a relatively small device with
> 4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
> up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>
> Is there any tweaks I can use to put a limit on the memory usage of
> uacctd.
>
> Thanx,
> Alex
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> tldr; post the output of the free command when uacctd is running and I’ll
> do my best to interpret it for you :-)
>
> Is it _really_ using it, or is it just not completely unused?
>
> The linux kernel generally does pretty good job of keeping stuff that
> might be useful in memory, but getting rid of it very very quickly if the
> space is needed for something else.
> The challenge you face is that most of the userspace tools come with a
> long list of caveats about what they appear to report vs what’s really
> happening, this is mainly due to the way the kernel shares memory between
> processes, and its use of as much memory as possible for various caches.
>
> Some threads worth reading if you’re not familiar with the in’s and out’s
> of linux kernel memory management...
>
> https://www.linuxatemyram.com/
> https://stackoverflow.com/questions/4802481/how-to-see-top-
> processes-sorted-by-actual-memory-usage
> https://stackoverflow.com/questions/3784974/want-to-know-
> whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
>
> HTH
> Dariush
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Alex K
Thanx for the reply.

The output of free is the following:

free
totalusedfreeshared buff/cache
available
Mem:4046572 2832576  152012  784240 1061984
204248
Swap:   3906556 1086080 2820476

While is as below when stopped:

free
  totalusedfree  shared  buff/cache
available
Mem:4046572  520352 3223884   14916  302336
3286952
Swap:   3906556  485040 3421516

Seems that mysql plugins are reserving quite some memory as they list first
in htop when sorted with memory.

Thanx,
Alex


On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

>
> On 25 Jun 2018, at 11:54, Alex K  wrote:
>
> Hi all,
>
> I have a setup with uacctd monitoring traffic of several interfaces
> through NFLOG.
> With uacctd stopped I see that the server (a relatively small device with
> 4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
> up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>
> Is there any tweaks I can use to put a limit on the memory usage of
> uacctd.
>
> Thanx,
> Alex
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> tldr; post the output of the free command when uacctd is running and I’ll
> do my best to interpret it for you :-)
>
> Is it _really_ using it, or is it just not completely unused?
>
> The linux kernel generally does pretty good job of keeping stuff that
> might be useful in memory, but getting rid of it very very quickly if the
> space is needed for something else.
> The challenge you face is that most of the userspace tools come with a
> long list of caveats about what they appear to report vs what’s really
> happening, this is mainly due to the way the kernel shares memory between
> processes, and its use of as much memory as possible for various caches.
>
> Some threads worth reading if you’re not familiar with the in’s and out’s
> of linux kernel memory management...
>
> https://www.linuxatemyram.com/
> https://stackoverflow.com/questions/4802481/how-to-see-
> top-processes-sorted-by-actual-memory-usage
> https://stackoverflow.com/questions/3784974/want-to-
> know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
>
> HTH
> Dariush
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Dariush Marsh-Mossadeghi
OK, so we’re moving from bottom-posting to top-posting… that’ll make it 
interesting for other readers ;-)

The output of free doesn’t look desperate, but it is starting to look a bit 
tight.
You’ve got about a gig of buffer/cache, which the kernel will evict if it needs 
it.
You’ve got 200M of genuinely free memory.
What is potentially a little concerning is the gig of swap in use, that may or 
may not be a problem depending on what else is running on the box and how it’s 
memory use varies over time.

I’m not intimatley familiar with the internals of uacctd’s mysql plugins, but 
my advice would be:

- Monitor the memory usage. If it doesn’t vary much over time you could go on 
for years and be just fine. Keep a close watch on swap usage, if that varies a 
lot or grows over time you’ll want to do something about it.

- Read up on OOM Killer, what it does and how it behaves. If you see OOM Killer 
entries in your logs, it’s time to think again

- Can you put some more memory in the box? 8GB of RAM will cost you about £60. 
How much is your time worth ?

HTH
Dariush


> On 25 Jun 2018, at 12:32, Alex K  wrote:
> 
> Thanx for the reply.
> 
> The output of free is the following:
> 
> free
> totalusedfreeshared buff/cache   
> available
> Mem:4046572 2832576  152012  784240 1061984  
> 204248
> Swap:   3906556 1086080 2820476
> 
> While is as below when stopped:
> 
> free
>   totalusedfree  shared  buff/cache   
> available
> Mem:4046572  520352 3223884   14916  302336 
> 3286952
> Swap:   3906556  485040 3421516
> 
> Seems that mysql plugins are reserving quite some memory as they list first 
> in htop when sorted with memory.
> 
> Thanx,
> Alex
> 
> 
> On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi 
> mailto:dari...@gravitas.co.uk>> wrote:
> 
>> On 25 Jun 2018, at 11:54, Alex K > > wrote:
>> 
>> Hi all,
>> 
>> I have a setup with uacctd monitoring traffic of several interfaces through 
>> NFLOG.
>> With uacctd stopped I see that the server (a relatively small device with 4 
>> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
>> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>> 
>> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
>> 
>> Thanx,
>> Alex
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists 
> 
> tldr; post the output of the free command when uacctd is running and I’ll do 
> my best to interpret it for you :-)
> 
> Is it _really_ using it, or is it just not completely unused?
> 
> The linux kernel generally does pretty good job of keeping stuff that might 
> be useful in memory, but getting rid of it very very quickly if the space is 
> needed for something else.
> The challenge you face is that most of the userspace tools come with a long 
> list of caveats about what they appear to report vs what’s really happening, 
> this is mainly due to the way the kernel shares memory between processes, and 
> its use of as much memory as possible for various caches.
> 
> Some threads worth reading if you’re not familiar with the in’s and out’s of 
> linux kernel memory management...
> 
> https://www.linuxatemyram.com/ 
> https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage
>  
> 
> https://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
>  
> 
> 
> HTH
> Dariush
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists 
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Dariush Marsh-Mossadeghi
OK, so it’s effectively an embedded system scenario, with a fixed config 
hardware, or similar
No silver bullets or one-liner fixes here :-\

You’ve a number of options to slim down your memory footprint
- Strip Debian of all the packages you don’t need, learn a lot about kernel 
tuning, and tune the kernel to your needs.
- Start looking at one of the more embedded systems oriented distros, rather 
than OOTB Debian. Although Debian is skinnier than most, there’s a lot that can 
be done to minimise footprint. The fact that you’ve got a 64bit Debian9 running 
on it makes me fairly optimistic that you could run pretty much any Debian 
derivative.
- Profiling your userspace software stack and see if there are any dependencies 
you don’t need, strip them out. Whether you can do this will very much depend 
on the modularity of the components.

All of the above lead you down the path of maintaining forks and compiling your 
own packages, to a greater or lesser extent. That’s a maintenance overhead you 
want to avoid unless you have to.
Having said all that, if you’re looking at deploying thousands of units, the 
economics of software maintenance may stack up for you.

HTH
Dariush


> On 25 Jun 2018, at 14:32, Alex K  wrote:
> 
> Let me change the posting style... :)
> 
> On Mon, Jun 25, 2018 at 3:51 PM, Dariush Marsh-Mossadeghi 
> mailto:dari...@gravitas.co.uk>> wrote:
> OK, so we’re moving from bottom-posting to top-posting… that’ll make it 
> interesting for other readers ;-)
> 
> The output of free doesn’t look desperate, but it is starting to look a bit 
> tight.
> You’ve got about a gig of buffer/cache, which the kernel will evict if it 
> needs it.
> You’ve got 200M of genuinely free memory.
> What is potentially a little concerning is the gig of swap in use, that may 
> or may not be a problem depending on what else is running on the box and how 
> it’s memory use varies over time.
> 
> I’m not intimatley familiar with the internals of uacctd’s mysql plugins, but 
> my advice would be:
> 
> - Monitor the memory usage. If it doesn’t vary much over time you could go on 
> for years and be just fine. Keep a close watch on swap usage, if that varies 
> a lot or grows over time you’ll want to do something about it.
> Monitoring of last 2 weeks is showing that free mem is fluctuating form 50M 
> to 220 MB and swaps remains steady. Thats why I am trying to find a solution.
> 
> 
> - Read up on OOM Killer, what it does and how it behaves. If you see OOM 
> Killer entries in your logs, it’s time to think again
> I am aware of that. thanx
> 
> - Can you put some more memory in the box? 8GB of RAM will cost you about 
> £60. How much is your time worth ?
> This is not an option. What I have is 4 GB. This is not just a personal 
> project. There are going to be thousands of such installations
> 
> HTH
> Dariush
> 
> 
>> On 25 Jun 2018, at 12:32, Alex K > > wrote:
>> 
>> Thanx for the reply.
>> 
>> The output of free is the following:
>> 
>> free
>> totalusedfreeshared buff/cache   
>> available
>> Mem:4046572 2832576  152012  784240 1061984  
>> 204248
>> Swap:   3906556 1086080 2820476
>> 
>> While is as below when stopped:
>> 
>> free
>>   totalusedfree  shared  buff/cache   
>> available
>> Mem:4046572  520352 3223884   14916  302336 
>> 3286952
>> Swap:   3906556  485040 3421516
>> 
>> Seems that mysql plugins are reserving quite some memory as they list first 
>> in htop when sorted with memory.
>> 
>> Thanx,
>> Alex
>> 
>> 
>> On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi 
>> mailto:dari...@gravitas.co.uk>> wrote:
>> 
>>> On 25 Jun 2018, at 11:54, Alex K >> > wrote:
>>> 
>>> Hi all,
>>> 
>>> I have a setup with uacctd monitoring traffic of several interfaces through 
>>> NFLOG.
>>> With uacctd stopped I see that the server (a relatively small device with 4 
>>> GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes up 
>>> to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>>> 
>>> Is there any tweaks I can use to put a limit on the memory usage of uacctd.
>>> 
>>> Thanx,
>>> Alex
>>> ___
>>> pmacct-discussion mailing list
>>> http://www.pmacct.net/#mailinglists 
>> 
>> tldr; post the output of the free command when uacctd is running and I’ll do 
>> my best to interpret it for you :-)
>> 
>> Is it _really_ using it, or is it just not completely unused?
>> 
>> The linux kernel generally does pretty good job of keeping stuff that might 
>> be useful in memory, but getting rid of it very very quickly if the space is 
>> needed for something else.
>> The challenge you face is that most of the userspace tools come with a long 
>> list of caveats about what they appear to 

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Rasto Rickardt
Hello,

If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.

This one is swapping pretty heavily, i would suggest to set to around 10
for 4GB RAM.

sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.

r.

On 06/25/2018 05:35 PM, Alex K wrote:
> Thanks Dariush. Appreciate your feedback.
> 
> I was testing several stripped down kernels by compiling and removing
> most of unused modules. The gain I had was in the range of few MB. 
> Seems that I have to find a more elegant approach at uacctd
> configuration since with current setup I am loading several mysql
> plugins so as to be able to filter traffic with directions, networks and
> ports.
> I was testing an edge case where these plugins may rise up to 210
> instances thus leading to this memory usage. Normally this will not be
> done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
> Thus in normal cases seems that I am ok.
> 
> Cheers
> 
> 
> On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
> mailto:dari...@gravitas.co.uk>> wrote:
> 
> OK, so it’s effectively an embedded system scenario, with a fixed
> config hardware, or similar
> No silver bullets or one-liner fixes here :-\
> 
> You’ve a number of options to slim down your memory footprint
> - Strip Debian of all the packages you don’t need, learn a lot about
> kernel tuning, and tune the kernel to your needs.
> - Start looking at one of the more embedded systems oriented
> distros, rather than OOTB Debian. Although Debian is skinnier than
> most, there’s a lot that can be done to minimise footprint. The fact
> that you’ve got a 64bit Debian9 running on it makes me fairly
> optimistic that you could run pretty much any Debian derivative.
> - Profiling your userspace software stack and see if there are any
> dependencies you don’t need, strip them out. Whether you can do this
> will very much depend on the modularity of the components.
> 
> All of the above lead you down the path of maintaining forks and
> compiling your own packages, to a greater or lesser extent. That’s a
> maintenance overhead you want to avoid unless you have to.
> Having said all that, if you’re looking at deploying thousands of
> units, the economics of software maintenance may stack up for you.
> 
> HTH
> Dariush 
> 
> 
>> On 25 Jun 2018, at 14:32, Alex K > > wrote:
>>
>> Let me change the posting style... :)
>>
>> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
>> Marsh-Mossadeghi > > wrote:
>>
>> OK, so we’re moving from bottom-posting to top-posting…
>> that’ll make it interesting for other readers ;-)
>>
>> The output of free doesn’t look desperate, but it is starting
>> to look a bit tight.
>> You’ve got about a gig of buffer/cache, which the kernel will
>> evict if it needs it.
>> You’ve got 200M of genuinely free memory.
>> What is potentially a little concerning is the gig of swap in
>> use, that may or may not be a problem depending on what else
>> is running on the box and how it’s memory use varies over time.
>>
>> I’m not intimatley familiar with the internals of uacctd’s
>> mysql plugins, but my advice would be:
>>
>> - Monitor the memory usage. If it doesn’t vary much over time
>> you could go on for years and be just fine. Keep a close watch
>> on swap usage, if that varies a lot or grows over time you’ll
>> want to do something about it.
>>
>> Monitoring of last 2 weeks is showing that free mem is fluctuating
>> form 50M to 220 MB and swaps remains steady. Thats why I am trying
>> to find a solution.
>>
>>
>> - Read up on OOM Killer, what it does and how it behaves. If
>> you see OOM Killer entries in your logs, it’s time to think again 
>>
>> I am aware of that. thanx
>>
>>
>> - Can you put some more memory in the box? 8GB of RAM will
>> cost you about £60. How much is your time worth ?
>>
>> This is not an option. What I have is 4 GB. This is not just a
>> personal project. There are going to be thousands of such
>> installations
>>
>> HTH
>> Dariush
>>
>>
>>> On 25 Jun 2018, at 12:32, Alex K >> > wrote:
>>>
>>> Thanx for the reply. 
>>>
>>> The output of free is the following: 
>>>
>>> free
>>> total    used    free    shared
>>> buff/cache   available
>>> Mem:    4046572 2832576  152012  784240
>>> 1061984  204248
>>> Swap:   3906556 1086080 2820476
>>>

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Alex K
Thanks Dariush. Appreciate your feedback.

I was testing several stripped down kernels by compiling and removing most
of unused modules. The gain I had was in the range of few MB.
Seems that I have to find a more elegant approach at uacctd configuration
since with current setup I am loading several mysql plugins so as to be
able to filter traffic with directions, networks and ports.
I was testing an edge case where these plugins may rise up to 210 instances
thus leading to this memory usage. Normally this will not be done so the
normal case will be using 2.5 GB of RAM instead of 3.5 GB.
Thus in normal cases seems that I am ok.

Cheers


On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

> OK, so it’s effectively an embedded system scenario, with a fixed config
> hardware, or similar
> No silver bullets or one-liner fixes here :-\
>
> You’ve a number of options to slim down your memory footprint
> - Strip Debian of all the packages you don’t need, learn a lot about
> kernel tuning, and tune the kernel to your needs.
> - Start looking at one of the more embedded systems oriented distros,
> rather than OOTB Debian. Although Debian is skinnier than most, there’s a
> lot that can be done to minimise footprint. The fact that you’ve got a
> 64bit Debian9 running on it makes me fairly optimistic that you could run
> pretty much any Debian derivative.
> - Profiling your userspace software stack and see if there are any
> dependencies you don’t need, strip them out. Whether you can do this will
> very much depend on the modularity of the components.
>
> All of the above lead you down the path of maintaining forks and compiling
> your own packages, to a greater or lesser extent. That’s a maintenance
> overhead you want to avoid unless you have to.
> Having said all that, if you’re looking at deploying thousands of units,
> the economics of software maintenance may stack up for you.
>
> HTH
> Dariush
>
>
> On 25 Jun 2018, at 14:32, Alex K  wrote:
>
> Let me change the posting style... :)
>
> On Mon, Jun 25, 2018 at 3:51 PM, Dariush Marsh-Mossadeghi  gravitas.co.uk> wrote:
>
>> OK, so we’re moving from bottom-posting to top-posting… that’ll make it
>> interesting for other readers ;-)
>>
>> The output of free doesn’t look desperate, but it is starting to look a
>> bit tight.
>> You’ve got about a gig of buffer/cache, which the kernel will evict if it
>> needs it.
>> You’ve got 200M of genuinely free memory.
>> What is potentially a little concerning is the gig of swap in use, that
>> may or may not be a problem depending on what else is running on the box
>> and how it’s memory use varies over time.
>>
>> I’m not intimatley familiar with the internals of uacctd’s mysql plugins,
>> but my advice would be:
>>
>> - Monitor the memory usage. If it doesn’t vary much over time you could
>> go on for years and be just fine. Keep a close watch on swap usage, if that
>> varies a lot or grows over time you’ll want to do something about it.
>>
> Monitoring of last 2 weeks is showing that free mem is fluctuating form
> 50M to 220 MB and swaps remains steady. Thats why I am trying to find a
> solution.
>
>
>> - Read up on OOM Killer, what it does and how it behaves. If you see OOM
>> Killer entries in your logs, it’s time to think again
>>
> I am aware of that. thanx
>
>>
>> - Can you put some more memory in the box? 8GB of RAM will cost you about
>> £60. How much is your time worth ?
>>
> This is not an option. What I have is 4 GB. This is not just a personal
> project. There are going to be thousands of such installations
>
>
> HTH
> Dariush
>
>
> On 25 Jun 2018, at 12:32, Alex K  wrote:
>
> Thanx for the reply.
>
> The output of free is the following:
>
> free
> totalusedfreeshared buff/cache
> available
> Mem:4046572 2832576  152012  784240 1061984
> 204248
> Swap:   3906556 1086080 2820476
>
> While is as below when stopped:
>
> free
>   totalusedfree  shared  buff/cache
> available
> Mem:4046572  520352 3223884   14916  302336
> 3286952
> Swap:   3906556  485040 3421516
>
> Seems that mysql plugins are reserving quite some memory as they list
> first in htop when sorted with memory.
>
> Thanx,
> Alex
>
>
> On Mon, Jun 25, 2018 at 2:22 PM, Dariush Marsh-Mossadeghi  gravitas.co.uk> wrote:
>
>>
>> On 25 Jun 2018, at 11:54, Alex K  wrote:
>>
>> Hi all,
>>
>> I have a setup with uacctd monitoring traffic of several interfaces
>> through NFLOG.
>> With uacctd stopped I see that the server (a relatively small device with
>> 4 GB of RAM) consumes 450MB of RAM. Once I start uacctd the mem usage goes
>> up to 3.5 GB. I am using mysql plugin and this is running on Debian9 64 bit.
>>
>> Is there any tweaks I can use to put a limit on the memory usage of
>> uacctd.
>>
>> Thanx,
>> Alex
>> ___
>> pmacct-discussion 

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Alex K
On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt  wrote:

> Hello,
>
> If there is no more RAM available, i would test sqlite3 plugin, as
> sqlite3 is more suited for limited resources usage. Youl will possibly
> need to change your workflow to export sqlite3 files and load it
> somewhere else, but it should be a lot cheaper memory wise.
>
> Thanx. Seems that I have to stick with mysql at least at this phase.

This one is swapping pretty heavily, i would suggest to set to around 10
> for 4GB RAM.
>
> sysctl vm.swappiness=10 so it will start using swap when available RAM
> is around 400MB.
>
This is already done.


> r.
>
> On 06/25/2018 05:35 PM, Alex K wrote:
> > Thanks Dariush. Appreciate your feedback.
> >
> > I was testing several stripped down kernels by compiling and removing
> > most of unused modules. The gain I had was in the range of few MB.
> > Seems that I have to find a more elegant approach at uacctd
> > configuration since with current setup I am loading several mysql
> > plugins so as to be able to filter traffic with directions, networks and
> > ports.
> > I was testing an edge case where these plugins may rise up to 210
> > instances thus leading to this memory usage. Normally this will not be
> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
> > Thus in normal cases seems that I am ok.
> >
> > Cheers
> >
> >
> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
> > mailto:dari...@gravitas.co.uk>> wrote:
> >
> > OK, so it’s effectively an embedded system scenario, with a fixed
> > config hardware, or similar
> > No silver bullets or one-liner fixes here :-\
> >
> > You’ve a number of options to slim down your memory footprint
> > - Strip Debian of all the packages you don’t need, learn a lot about
> > kernel tuning, and tune the kernel to your needs.
> > - Start looking at one of the more embedded systems oriented
> > distros, rather than OOTB Debian. Although Debian is skinnier than
> > most, there’s a lot that can be done to minimise footprint. The fact
> > that you’ve got a 64bit Debian9 running on it makes me fairly
> > optimistic that you could run pretty much any Debian derivative.
> > - Profiling your userspace software stack and see if there are any
> > dependencies you don’t need, strip them out. Whether you can do this
> > will very much depend on the modularity of the components.
> >
> > All of the above lead you down the path of maintaining forks and
> > compiling your own packages, to a greater or lesser extent. That’s a
> > maintenance overhead you want to avoid unless you have to.
> > Having said all that, if you’re looking at deploying thousands of
> > units, the economics of software maintenance may stack up for you.
> >
> > HTH
> > Dariush
> >
> >
> >> On 25 Jun 2018, at 14:32, Alex K  >> > wrote:
> >>
> >> Let me change the posting style... :)
> >>
> >> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
> >> Marsh-Mossadeghi  >> > wrote:
> >>
> >> OK, so we’re moving from bottom-posting to top-posting…
> >> that’ll make it interesting for other readers ;-)
> >>
> >> The output of free doesn’t look desperate, but it is starting
> >> to look a bit tight.
> >> You’ve got about a gig of buffer/cache, which the kernel will
> >> evict if it needs it.
> >> You’ve got 200M of genuinely free memory.
> >> What is potentially a little concerning is the gig of swap in
> >> use, that may or may not be a problem depending on what else
> >> is running on the box and how it’s memory use varies over time.
> >>
> >> I’m not intimatley familiar with the internals of uacctd’s
> >> mysql plugins, but my advice would be:
> >>
> >> - Monitor the memory usage. If it doesn’t vary much over time
> >> you could go on for years and be just fine. Keep a close watch
> >> on swap usage, if that varies a lot or grows over time you’ll
> >> want to do something about it.
> >>
> >> Monitoring of last 2 weeks is showing that free mem is fluctuating
> >> form 50M to 220 MB and swaps remains steady. Thats why I am trying
> >> to find a solution.
> >>
> >>
> >> - Read up on OOM Killer, what it does and how it behaves. If
> >> you see OOM Killer entries in your logs, it’s time to think
> again
> >>
> >> I am aware of that. thanx
> >>
> >>
> >> - Can you put some more memory in the box? 8GB of RAM will
> >> cost you about £60. How much is your time worth ?
> >>
> >> This is not an option. What I have is 4 GB. This is not just a
> >> personal project. There are going to be thousands of such
> >> installations
> >>
> >> HTH
> >> Dariush
> >>
> >>
> >>> On 25 Jun 2018, at 12:32, Alex K  >>> 

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Dariush Marsh-Mossadeghi
Rasto's comments got me thinking…

Not being privy to your application and it’s architecture this may not work for 
you at all.

I’ve had success in the past using rabbitmq to offload flow logs.
We used it to deliver netflow to ELK stacks running on AWS, and it worked 
really well.
It has the major advantage of operational resilience.

Something like this:

netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path 
across public internet -> rabbitmq -> logstash -> elasticsearch

as ever YMMV :-)
Dariush


> On 25 Jun 2018, at 16:53, Alex K  wrote:
> 
> 
> 
> On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt  > wrote:
> Hello,
> 
> If there is no more RAM available, i would test sqlite3 plugin, as
> sqlite3 is more suited for limited resources usage. Youl will possibly
> need to change your workflow to export sqlite3 files and load it
> somewhere else, but it should be a lot cheaper memory wise.
> 
> Thanx. Seems that I have to stick with mysql at least at this phase.
> 
> This one is swapping pretty heavily, i would suggest to set to around 10
> for 4GB RAM.
> 
> sysctl vm.swappiness=10 so it will start using swap when available RAM
> is around 400MB.
> This is already done.
> 
> 
> r.
> 
> On 06/25/2018 05:35 PM, Alex K wrote:
> > Thanks Dariush. Appreciate your feedback.
> >
> > I was testing several stripped down kernels by compiling and removing
> > most of unused modules. The gain I had was in the range of few MB.
> > Seems that I have to find a more elegant approach at uacctd
> > configuration since with current setup I am loading several mysql
> > plugins so as to be able to filter traffic with directions, networks and
> > ports.
> > I was testing an edge case where these plugins may rise up to 210
> > instances thus leading to this memory usage. Normally this will not be
> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
> > Thus in normal cases seems that I am ok.
> >
> > Cheers
> >
> >
> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
> > mailto:dari...@gravitas.co.uk> 
> > >> wrote:
> >
> > OK, so it’s effectively an embedded system scenario, with a fixed
> > config hardware, or similar
> > No silver bullets or one-liner fixes here :-\
> >
> > You’ve a number of options to slim down your memory footprint
> > - Strip Debian of all the packages you don’t need, learn a lot about
> > kernel tuning, and tune the kernel to your needs.
> > - Start looking at one of the more embedded systems oriented
> > distros, rather than OOTB Debian. Although Debian is skinnier than
> > most, there’s a lot that can be done to minimise footprint. The fact
> > that you’ve got a 64bit Debian9 running on it makes me fairly
> > optimistic that you could run pretty much any Debian derivative.
> > - Profiling your userspace software stack and see if there are any
> > dependencies you don’t need, strip them out. Whether you can do this
> > will very much depend on the modularity of the components.
> >
> > All of the above lead you down the path of maintaining forks and
> > compiling your own packages, to a greater or lesser extent. That’s a
> > maintenance overhead you want to avoid unless you have to.
> > Having said all that, if you’re looking at deploying thousands of
> > units, the economics of software maintenance may stack up for you.
> >
> > HTH
> > Dariush
> >
> >
> >> On 25 Jun 2018, at 14:32, Alex K  >> 
> >> >> 
> >> wrote:
> >>
> >> Let me change the posting style... :)
> >>
> >> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
> >> Marsh-Mossadeghi  >> 
> >> >> wrote:
> >>
> >> OK, so we’re moving from bottom-posting to top-posting…
> >> that’ll make it interesting for other readers ;-)
> >>
> >> The output of free doesn’t look desperate, but it is starting
> >> to look a bit tight.
> >> You’ve got about a gig of buffer/cache, which the kernel will
> >> evict if it needs it.
> >> You’ve got 200M of genuinely free memory.
> >> What is potentially a little concerning is the gig of swap in
> >> use, that may or may not be a problem depending on what else
> >> is running on the box and how it’s memory use varies over time.
> >>
> >> I’m not intimatley familiar with the internals of uacctd’s
> >> mysql plugins, but my advice would be:
> >>
> >> - Monitor the memory usage. If it doesn’t vary much over time
> >> you could go on for years and be just fine. Keep a close watch
> >> on swap usage, if that varies a lot or grows over time you’ll
> >> want to do something about 

Re: [pmacct-discussion] uacctd memory usage

2018-06-25 Thread Alex K
I have not experience with RabbitMQ but I've seen this guy around on high
volume data transfer scenarios.
A message queuing approach sounds more resilient to me but this will need
consideration at a later point. Thanx though for the thoughts.
What I currently do is: iptables -> uacctd -> mysql -> processing of mysql
and further aggregation -> json file -> push to central hub (rsync) ->
import to DB -> visualize... (I've been playing like this for at least 3
years without issues)
It is interesting to hear how you tackle such issues.

Alex


On Mon, Jun 25, 2018 at 7:51 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

> Rasto's comments got me thinking…
>
> Not being privy to your application and it’s architecture this may not
> work for you at all.
>
> I’ve had success in the past using rabbitmq to offload flow logs.
> We used it to deliver netflow to ELK stacks running on AWS, and it worked
> really well.
> It has the major advantage of operational resilience.
>
> Something like this:
>
> netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path
> across public internet -> rabbitmq -> logstash -> elasticsearch
>
> as ever YMMV :-)
> Dariush
>
>
> On 25 Jun 2018, at 16:53, Alex K  wrote:
>
>
>
> On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt  wrote:
>
>> Hello,
>>
>> If there is no more RAM available, i would test sqlite3 plugin, as
>> sqlite3 is more suited for limited resources usage. Youl will possibly
>> need to change your workflow to export sqlite3 files and load it
>> somewhere else, but it should be a lot cheaper memory wise.
>>
>> Thanx. Seems that I have to stick with mysql at least at this phase.
>
> This one is swapping pretty heavily, i would suggest to set to around 10
>> for 4GB RAM.
>>
>> sysctl vm.swappiness=10 so it will start using swap when available RAM
>> is around 400MB.
>>
> This is already done.
>
>
>> r.
>>
>> On 06/25/2018 05:35 PM, Alex K wrote:
>> > Thanks Dariush. Appreciate your feedback.
>> >
>> > I was testing several stripped down kernels by compiling and removing
>> > most of unused modules. The gain I had was in the range of few MB.
>> > Seems that I have to find a more elegant approach at uacctd
>> > configuration since with current setup I am loading several mysql
>> > plugins so as to be able to filter traffic with directions, networks and
>> > ports.
>> > I was testing an edge case where these plugins may rise up to 210
>> > instances thus leading to this memory usage. Normally this will not be
>> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
>> > Thus in normal cases seems that I am ok.
>> >
>> > Cheers
>> >
>> >
>> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
>> > mailto:dari...@gravitas.co.uk>> wrote:
>> >
>> > OK, so it’s effectively an embedded system scenario, with a fixed
>> > config hardware, or similar
>> > No silver bullets or one-liner fixes here :-\
>> >
>> > You’ve a number of options to slim down your memory footprint
>> > - Strip Debian of all the packages you don’t need, learn a lot about
>> > kernel tuning, and tune the kernel to your needs.
>> > - Start looking at one of the more embedded systems oriented
>> > distros, rather than OOTB Debian. Although Debian is skinnier than
>> > most, there’s a lot that can be done to minimise footprint. The fact
>> > that you’ve got a 64bit Debian9 running on it makes me fairly
>> > optimistic that you could run pretty much any Debian derivative.
>> > - Profiling your userspace software stack and see if there are any
>> > dependencies you don’t need, strip them out. Whether you can do this
>> > will very much depend on the modularity of the components.
>> >
>> > All of the above lead you down the path of maintaining forks and
>> > compiling your own packages, to a greater or lesser extent. That’s a
>> > maintenance overhead you want to avoid unless you have to.
>> > Having said all that, if you’re looking at deploying thousands of
>> > units, the economics of software maintenance may stack up for you.
>> >
>> > HTH
>> > Dariush
>> >
>> >
>> >> On 25 Jun 2018, at 14:32, Alex K > >> > wrote:
>> >>
>> >> Let me change the posting style... :)
>> >>
>> >> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
>> >> Marsh-Mossadeghi > >> > wrote:
>> >>
>> >> OK, so we’re moving from bottom-posting to top-posting…
>> >> that’ll make it interesting for other readers ;-)
>> >>
>> >> The output of free doesn’t look desperate, but it is starting
>> >> to look a bit tight.
>> >> You’ve got about a gig of buffer/cache, which the kernel will
>> >> evict if it needs it.
>> >> You’ve got 200M of genuinely free memory.
>> >> What is potentially a little concerning is the gig of swap in
>> >> use, that may or may not be a problem