Re: [pmacct-discussion] Redis support

[Rasto Rickardt]

Hello Paolo,

+1 for me, main reasons are the usual Redis ones:

Speed - 100k GET by_id per second
Data persistence across restarts
Multi node deployment is not a MUST like other NoSQL document stores
Lightweight

My use-case is storing time-based events from flows in InfluxDB (bps,
pps, flows per host/subnet) and details (tcp flags, packet sizes,
as_path etc.) in redis. Storing timestamp, tag and value is where Influx
is good, storing documentation is where redis shines.

Storing directly to redis can be usefull, because most of data amount
per flow goes there for me.

The same will apply for InfluxDB but i am interfacing it over http, so
there is bunch of overhead already, i do not think there will be major
performance improvement writing directly, and tag:value pairs are really
lightweight.

r.


On 11/14/2016 06:02 PM, Paolo Lucente wrote:
> 
> Hi Lorenzo,
> 
> You are the second person asking for this - i'd be curious to see on
> the list if there would be anybody else interested in this.
> 
> I have a question, maybe basic: would not Kafka or RabbitMQ be
> integrated the same way with Logstash and/or Telegraf?
> 
> Cheers, Paolo
> 
> On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
>> Hi Paolo, I'm writing to asking if you are thinking about
>> supporting Redis as plugin
>> 
>> The advantage to use Redis would be that both Logstash both
>> Telegraf support it in a native way, then it would be easy to store
>> pmacct data in native way in Elasticsearch or InfluxDB. Regards
>> 
>> 
>> digitel
>> 
>> Via della Fortezza 6 - 50129 Firenze 
>> www.digitelitalia.com - 800 901 669
>> 
>> Ing. Lorenzo Mainardi
>> 
>> Tel +39 055 4624933 Fax +39 055 4624 947 
>> l...@digitelitalia.com
>> 
>> 
> 
>> ___ pmacct-discussion
>> mailing list http://www.pmacct.net/#mailinglists
> 
> 
> ___ pmacct-discussion
> mailing list http://www.pmacct.net/#mailinglists
> 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] uacctd memory usage

[Rasto Rickardt]

Hello,

If there is no more RAM available, i would test sqlite3 plugin, as
sqlite3 is more suited for limited resources usage. Youl will possibly
need to change your workflow to export sqlite3 files and load it
somewhere else, but it should be a lot cheaper memory wise.

This one is swapping pretty heavily, i would suggest to set to around 10
for 4GB RAM.

sysctl vm.swappiness=10 so it will start using swap when available RAM
is around 400MB.

r.

On 06/25/2018 05:35 PM, Alex K wrote:
> Thanks Dariush. Appreciate your feedback.
> 
> I was testing several stripped down kernels by compiling and removing
> most of unused modules. The gain I had was in the range of few MB. 
> Seems that I have to find a more elegant approach at uacctd
> configuration since with current setup I am loading several mysql
> plugins so as to be able to filter traffic with directions, networks and
> ports.
> I was testing an edge case where these plugins may rise up to 210
> instances thus leading to this memory usage. Normally this will not be
> done so the normal case will be using 2.5 GB of RAM instead of 3.5 GB.
> Thus in normal cases seems that I am ok.
> 
> Cheers
> 
> 
> On Mon, Jun 25, 2018 at 6:07 PM, Dariush Marsh-Mossadeghi
> mailto:dari...@gravitas.co.uk>> wrote:
> 
> OK, so it’s effectively an embedded system scenario, with a fixed
> config hardware, or similar
> No silver bullets or one-liner fixes here :-\
> 
> You’ve a number of options to slim down your memory footprint
> - Strip Debian of all the packages you don’t need, learn a lot about
> kernel tuning, and tune the kernel to your needs.
> - Start looking at one of the more embedded systems oriented
> distros, rather than OOTB Debian. Although Debian is skinnier than
> most, there’s a lot that can be done to minimise footprint. The fact
> that you’ve got a 64bit Debian9 running on it makes me fairly
> optimistic that you could run pretty much any Debian derivative.
> - Profiling your userspace software stack and see if there are any
> dependencies you don’t need, strip them out. Whether you can do this
> will very much depend on the modularity of the components.
> 
> All of the above lead you down the path of maintaining forks and
> compiling your own packages, to a greater or lesser extent. That’s a
> maintenance overhead you want to avoid unless you have to.
> Having said all that, if you’re looking at deploying thousands of
> units, the economics of software maintenance may stack up for you.
> 
> HTH
> Dariush 
> 
> 
>> On 25 Jun 2018, at 14:32, Alex K > > wrote:
>>
>> Let me change the posting style... :)
>>
>> On Mon, Jun 25, 2018 at 3:51 PM, Dariush
>> Marsh-Mossadeghi > > wrote:
>>
>> OK, so we’re moving from bottom-posting to top-posting…
>> that’ll make it interesting for other readers ;-)
>>
>> The output of free doesn’t look desperate, but it is starting
>> to look a bit tight.
>> You’ve got about a gig of buffer/cache, which the kernel will
>> evict if it needs it.
>> You’ve got 200M of genuinely free memory.
>> What is potentially a little concerning is the gig of swap in
>> use, that may or may not be a problem depending on what else
>> is running on the box and how it’s memory use varies over time.
>>
>> I’m not intimatley familiar with the internals of uacctd’s
>> mysql plugins, but my advice would be:
>>
>> - Monitor the memory usage. If it doesn’t vary much over time
>> you could go on for years and be just fine. Keep a close watch
>> on swap usage, if that varies a lot or grows over time you’ll
>> want to do something about it.
>>
>> Monitoring of last 2 weeks is showing that free mem is fluctuating
>> form 50M to 220 MB and swaps remains steady. Thats why I am trying
>> to find a solution.
>>
>>
>> - Read up on OOM Killer, what it does and how it behaves. If
>> you see OOM Killer entries in your logs, it’s time to think again 
>>
>> I am aware of that. thanx
>>
>>
>> - Can you put some more memory in the box? 8GB of RAM will
>> cost you about £60. How much is your time worth ?
>>
>> This is not an option. What I have is 4 GB. This is not just a
>> personal project. There are going to be thousands of such
>> installations
>>
>> HTH
>> Dariush
>>
>>
>>> On 25 Jun 2018, at 12:32, Alex K >> > wrote:
>>>
>>> Thanx for the reply. 
>>>
>>> The output of free is the following: 
>>>
>>> free
>>> total    used    free    shared
>>> buff/cache   available
>>> Mem:    4046572 2832576  152012  784240
>>> 1061984  204248
>>> Swap:   3906556 1086080 2820476
>>>