On 8/14/2014 5:05 PM, Cameron Kerr wrote:
With regard to you disk buffering question, I'm currently using this
configuration:
<Processor buffer>
Module pm_buffer
# 10GB buffer, cause we suck a lot of data (it would last roughly
30 minutes currently)
MaxSize 10000000
# If we've used 1% of it, that's still quite a bit and a potential
sign of trouble.
WarnLimit 100000
Type Disk
Directory /logs/current/buffer/
</Processor>
<Route 3>
Path in_json_tls => buffer => out_logstash
</Route>
It is important to determine how much time your buffer buys you with
regard to YOUR incoming logs, and size appropriately. In my
environment, I'm receiving a fairly massive amount in order to satisfy
particular requirements. If I prevent nxlog being able to send it out
(in my case, to logstash -- I do this by turning off logstash), and
then watch the size of /logs/current/buffer/ over time. I first see
how much time I have before I expect it to fill up; and then (which
gives a useful insight into your indexing speed for logstash), how
long it takes to drain. (logstash_etc_processing_rate =
buffer_fill_rate - buffer_drain_rate)
Hope that helps,
Cameron
--
Cameron Kerr <cameron.kerr...@gmail.com
<mailto:cameron.kerr...@gmail.com>>
See my blog at http://distracted-it.blogspot.co.nz/ (previously
http://humbledown.org/)
Skype me on cameron.kerr.nz <http://cameron.kerr.nz>
On 14 August 2014 18:07, Shanks <certpurs...@hotmail.com
<mailto:certpurs...@hotmail.com>> wrote:
Hello,
First, Thank You for wonderful utility. Am using nxlog-ce-2.7.1191 on
Debian. The nxlog binary is @ /usr/bin/nxlog.
I've following 2 queries - please provide your view
points/clarifications. Much appreciated.
Q1. While using pm_buffer, when 'Type' is 'Disk', which partition will
get used ?. /var ? or Are there any configurable param using which
I can
direct nxlog
to use a specific partition ?.
<Processor buffer>
Module pm_buffer
# 1Mb buffer
MaxSize 1024
Type Disk
# warn at 512k
WarnLimit 512
</Processor>
Q2. For below usage of im_file module, what is the impact of
"SavePos TRUE" & "ReadFromLast FALSE" ?.
<Input myfile2>
Module im_file
File "/home/mike/Downloads/test1.log"
SavePos TRUE
ReadFromLast FALSE
Exec $TYPE = "testlog-2";
</Input>
I had opted for "SavePos TRUE", so that in case nxlog restarts, it
remembers its last read position. Also, opted for 'ReadFromLast FALSE'
because whenever use it for first time, I wanted nxlog to read entire
log and not just start from last line. Thereafter wards, as and
when new
line gets
written to log file, nxlog will ship them by default. And, as
"SavePos TRUE" saves last-read position, it'll ensure double
protection during nxlog restarts.
Is above interpretations correct ?. Please advise.
Thanks,
Shanks
------------------------------------------------------------------------------
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
<mailto:nxlog-ce-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
Thank You Cameron. I've had an oversight while reading through the nxlog
manual page.
Detailed note on how to arrive @ sizing for the buffer was insightful.
Thanks.
Shanks
------------------------------------------------------------------------------
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
