With regard to you disk buffering question, I'm currently using this
configuration:
<Processor buffer>
Module pm_buffer
# 10GB buffer, cause we suck a lot of data (it would last roughly 30
minutes currently)
MaxSize 10000000
# If we've used 1% of it, that's still quite a bit and a potential sign
of trouble.
WarnLimit 100000
Type Disk
Directory /logs/current/buffer/
</Processor>
<Route 3>
Path in_json_tls => buffer => out_logstash
</Route>
It is important to determine how much time your buffer buys you with regard
to YOUR incoming logs, and size appropriately. In my environment, I'm
receiving a fairly massive amount in order to satisfy particular
requirements. If I prevent nxlog being able to send it out (in my case, to
logstash -- I do this by turning off logstash), and then watch the size of
/logs/current/buffer/ over time. I first see how much time I have before I
expect it to fill up; and then (which gives a useful insight into your
indexing speed for logstash), how long it takes to drain.
(logstash_etc_processing_rate = buffer_fill_rate - buffer_drain_rate)
Hope that helps,
Cameron
--
Cameron Kerr <[email protected]>
See my blog at http://distracted-it.blogspot.co.nz/ (previously
http://humbledown.org/)
Skype me on cameron.kerr.nz
On 14 August 2014 18:07, Shanks <[email protected]> wrote:
> Hello,
>
> First, Thank You for wonderful utility. Am using nxlog-ce-2.7.1191 on
> Debian. The nxlog binary is @ /usr/bin/nxlog.
>
> I've following 2 queries - please provide your view
> points/clarifications. Much appreciated.
>
> Q1. While using pm_buffer, when 'Type' is 'Disk', which partition will
> get used ?. /var ? or Are there any configurable param using which I can
> direct nxlog
> to use a specific partition ?.
>
> <Processor buffer>
> Module pm_buffer
> # 1Mb buffer
> MaxSize 1024
> Type Disk
> # warn at 512k
> WarnLimit 512
> </Processor>
>
> Q2. For below usage of im_file module, what is the impact of
> "SavePos TRUE" & "ReadFromLast FALSE" ?.
>
> <Input myfile2>
>
> Module im_file
> File "/home/mike/Downloads/test1.log"
> SavePos TRUE
> ReadFromLast FALSE
> Exec $TYPE = "testlog-2";
> </Input>
>
> I had opted for "SavePos TRUE", so that in case nxlog restarts, it
> remembers its last read position. Also, opted for 'ReadFromLast FALSE'
> because whenever use it for first time, I wanted nxlog to read entire
> log and not just start from last line. Thereafter wards, as and when new
> line gets
> written to log file, nxlog will ship them by default. And, as
> "SavePos TRUE" saves last-read position, it'll ensure double
> protection during nxlog restarts.
>
> Is above interpretations correct ?. Please advise.
>
> Thanks,
> Shanks
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> nxlog-ce-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
------------------------------------------------------------------------------
_______________________________________________
nxlog-ce-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
