I have not experience with RabbitMQ but I've seen this guy around on high
volume data transfer scenarios.
A message queuing approach sounds more resilient to me but this will need
consideration at a later point. Thanx though for the thoughts.
What I currently do is: iptables -> uacctd -> mysql -> processing of mysql
and further aggregation -> json file -> push to central hub (rsync) ->
import to DB -> visualize... (I've been playing like this for at least 3
years without issues)
It is interesting to hear how you tackle such issues.

Alex


On Mon, Jun 25, 2018 at 7:51 PM, Dariush Marsh-Mossadeghi <
dari...@gravitas.co.uk> wrote:

> Rasto's comments got me thinking…
>
> Not being privy to your application and it’s architecture this may not
> work for you at all.
>
> I’ve had success in the past using rabbitmq to offload flow logs.
> We used it to deliver netflow to ELK stacks running on AWS, and it worked
> really well.
> It has the major advantage of operational resilience.
>
> Something like this:
>
> netflow sources -> nfacctd -> rabbitmq -> big long unreliable network path
> across public internet -> rabbitmq -> logstash -> elasticsearch
>
> as ever YMMV :-)
> Dariush
>
>
> On 25 Jun 2018, at 16:53, Alex K <rightkickt...@gmail.com> wrote:
>
>
>
> On Mon, Jun 25, 2018 at 6:49 PM, Rasto Rickardt <pho...@axfr.org> wrote:
>
>> Hello,
>>
>> If there is no more RAM available, i would test sqlite3 plugin, as
>> sqlite3 is more suited for limited resources usage. Youl will possibly
>> need to change your workflow to export sqlite3 files and load it
>> somewhere else, but it should be a lot cheaper memory wise.
>>
>> Thanx. Seems that I have to stick with mysql at least at this phase.
>
> This one is swapping pretty heavily, i would suggest to set to around 10
>> for 4GB RAM.
>>
>> sysctl vm.swappiness=10 so it will start using swap when available RAM
>> is around 400MB.
>>
> This is already done.
>
>
>> r.
>>
>> On 06/25/2018 05:35 PM, Alex K wrote:
>> > Thanks Dariush. Appreciate your feedback.
>> >
>> > I was testing several stripped down kernels by compiling and removing
>> > most of unused modules. The gain I had was in the range of few MB.
>> > Seems that I have to find a more elegant approach at uacctd
>> > configuration since with current setup I am loading several mysql
>> > plugins so as to be able to filter traffic with directions, networks and
>> > ports.
>> > I was testing an edge case where these plugins may rise up to 210
>> > instances thus leading to this memory usage. Normally this will not be
>> > done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
>> > Thus in normal cases seems that I am ok.
>> >
>> > Cheers
>> >
>> >
>> > On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
>> > <dari...@gravitas.co.uk <mailto:dari...@gravitas.co.uk>> wrote:
>> >
>> >     OK, so it’s effectively an embedded system scenario, with a fixed
>> >     config hardware, or similar
>> >     No silver bullets or one-liner fixes here :-\
>> >
>> >     You’ve a number of options to slim down your memory footprint
>> >     - Strip Debian of all the packages you don’t need, learn a lot about
>> >     kernel tuning, and tune the kernel to your needs.
>> >     - Start looking at one of the more embedded systems oriented
>> >     distros, rather than OOTB Debian. Although Debian is skinnier than
>> >     most, there’s a lot that can be done to minimise footprint. The fact
>> >     that you’ve got a 64bit Debian9 running on it makes me fairly
>> >     optimistic that you could run pretty much any Debian derivative.
>> >     - Profiling your userspace software stack and see if there are any
>> >     dependencies you don’t need, strip them out. Whether you can do this
>> >     will very much depend on the modularity of the components.
>> >
>> >     All of the above lead you down the path of maintaining forks and
>> >     compiling your own packages, to a greater or lesser extent. That’s a
>> >     maintenance overhead you want to avoid unless you have to.
>> >     Having said all that, if you’re looking at deploying thousands of
>> >     units, the economics of software maintenance may stack up for you.
>> >
>> >     HTH
>> >     Dariush
>> >
>> >
>> >>     On 25 Jun 2018, at 14:32, Alex K <rightkickt...@gmail.com
>> >>     <mailto:rightkickt...@gmail.com>> wrote:
>> >>
>> >>     Let me change the posting style... :)
>> >>
>> >>     On Mon, Jun 25, 2018 at 3:51 PM, Dariush
>> >>     Marsh-Mossadeghi <dari...@gravitas.co.uk
>> >>     <mailto:dari...@gravitas.co.uk>> wrote:
>> >>
>> >>         OK, so we’re moving from bottom-posting to top-posting…
>> >>         that’ll make it interesting for other readers ;-)
>> >>
>> >>         The output of free doesn’t look desperate, but it is starting
>> >>         to look a bit tight.
>> >>         You’ve got about a gig of buffer/cache, which the kernel will
>> >>         evict if it needs it.
>> >>         You’ve got 200M of genuinely free memory.
>> >>         What is potentially a little concerning is the gig of swap in
>> >>         use, that may or may not be a problem depending on what else
>> >>         is running on the box and how it’s memory use varies over time.
>> >>
>> >>         I’m not intimatley familiar with the internals of uacctd’s
>> >>         mysql plugins, but my advice would be:
>> >>
>> >>         - Monitor the memory usage. If it doesn’t vary much over time
>> >>         you could go on for years and be just fine. Keep a close watch
>> >>         on swap usage, if that varies a lot or grows over time you’ll
>> >>         want to do something about it.
>> >>
>> >>     Monitoring of last 2 weeks is showing that free mem is fluctuating
>> >>     form 50M to 220 MB and swaps remains steady. Thats why I am trying
>> >>     to find a solution.
>> >>
>> >>
>> >>         - Read up on OOM Killer, what it does and how it behaves. If
>> >>         you see OOM Killer entries in your logs, it’s time to think
>> again
>> >>
>> >>     I am aware of that. thanx
>> >>
>> >>
>> >>         - Can you put some more memory in the box? 8GB of RAM will
>> >>         cost you about £60. How much is your time worth ?
>> >>
>> >>     This is not an option. What I have is 4 GB. This is not just a
>> >>     personal project. There are going to be thousands of such
>> >>     installations
>> >>
>> >>     HTH
>> >>     Dariush
>> >>
>> >>
>> >>>     On 25 Jun 2018, at 12:32, Alex K <rightkickt...@gmail.com
>> >>>     <mailto:rightkickt...@gmail.com>> wrote:
>> >>>
>> >>>     Thanx for the reply.
>> >>>
>> >>>     The output of free is the following:
>> >>>
>> >>>     free
>> >>>                 total        used        free        shared
>> >>>     buff/cache   available
>> >>>     Mem:        4046572     2832576      152012      784240
>> >>>     1061984      204248
>> >>>     Swap:       3906556     1086080     2820476
>> >>>
>> >>>     While is as below when stopped:
>> >>>
>> >>>     free
>> >>>                   total        used        free      shared
>> >>>     buff/cache   available
>> >>>     Mem:        4046572      520352     3223884       14916
>> >>>     302336     3286952
>> >>>     Swap:       3906556      485040     3421516
>> >>>
>> >>>     Seems that mysql plugins are reserving quite some memory as they
>> >>>     list first in htop when sorted with memory.
>> >>>
>> >>>     Thanx,
>> >>>     Alex
>> >>>
>> >>>
>> >>>     On Mon, Jun 25, 2018 at 2:22 PM, Dariush
>> >>>     Marsh-Mossadeghi <dari...@gravitas.co.uk
>> >>>     <mailto:dari...@gravitas.co.uk>> wrote:
>> >>>
>> >>>
>> >>>>         On 25 Jun 2018, at 11:54, Alex K <rightkickt...@gmail.com
>> >>>>         <mailto:rightkickt...@gmail.com>> wrote:
>> >>>>
>> >>>>         Hi all,
>> >>>>
>> >>>>         I have a setup with uacctd monitoring traffic of several
>> >>>>         interfaces through NFLOG.
>> >>>>         With uacctd stopped I see that the server (a relatively
>> >>>>         small device with 4 GB of RAM) consumes 450MB of RAM. Once I
>> >>>>         start uacctd the mem usage goes up to 3.5 GB. I am using
>> >>>>         mysql plugin and this is running on Debian9 64 bit.
>> >>>>
>> >>>>         Is there any tweaks I can use to put a limit on the memory
>> >>>>         usage of uacctd.
>> >>>>
>> >>>>         Thanx,
>> >>>>         Alex
>> >>>>         _______________________________________________
>> >>>>         pmacct-discussion mailing list
>> >>>>         http://www.pmacct.net/#mailinglists
>> >>>>         <http://www.pmacct.net/#mailinglists>
>> >>>
>> >>>
>> >>>         tldr; post the output of the free command when uacctd is
>> >>>         running and I’ll do my best to interpret it for you :-)
>> >>>
>> >>>         Is it _really_ using it, or is it just not completely unused?
>> >>>
>> >>>         The linux kernel generally does pretty good job of keeping
>> >>>         stuff that might be useful in memory, but getting rid of it
>> >>>         very very quickly if the space is needed for something else.
>> >>>         The challenge you face is that most of the userspace tools
>> >>>         come with a long list of caveats about what they appear to
>> >>>         report vs what’s really happening, this is mainly due to the
>> >>>         way the kernel shares memory between processes, and its use
>> >>>         of as much memory as possible for various caches.
>> >>>
>> >>>         Some threads worth reading if you’re not familiar with the
>> >>>         in’s and out’s of linux kernel memory management...
>> >>>
>> >>>         https://www.linuxatemyram.com/
>> >>>         https://stackoverflow.com/questions/4802481/how-to-see-top-
>> processes-sorted-by-actual-memory-usage
>> >>>         <https://stackoverflow.com/questions/4802481/how-to-see-top
>> -processes-sorted-by-actual-memory-usage>
>> >>>         https://stackoverflow.com/questions/3784974/want-to-know-
>> whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/
>> >>>         <https://stackoverflow.com/questions/3784974/want-to-know-
>> whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-app/>
>> >>>
>> >>>         HTH
>> >>>         Dariush
>> >>
>> >
>> >
>> >     _______________________________________________
>> >     pmacct-discussion mailing list
>> >     http://www.pmacct.net/#mailinglists
>> >     <http://www.pmacct.net/#mailinglists>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > pmacct-discussion mailing list
>> > http://www.pmacct.net/#mailinglists
>> >
>>
>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to