Personally, I send all in syslog format to logstash.
I convert the multi-lines logs into single line with:
'Exec if $raw_event =~ s/[\r\n]/ /g {}'
Below an extract of my log files.
Exec $hostname = '%HOSTNAME%';
Exec $SyslogSeverityValue = 5;
Exec if $raw_event =~ s/[\r\n]/ /g {}
Exec if $raw_event =~ s/^\*\*\* (.*) \[(.*)\] .* pid:([0-9]*)
thread:([0-9]*) *// { \
$SourceName = '%SOURCE%_' + $1; \
$ProcessID = $3; \
$tmp = $2; \
if ( $tmp =~ /[Ww]arning/ ) $SysLogSeverityValue = 4; \
if ( $tmp =~ /[Ee]rror/ ) $SysLogSeverityValue = 3; \
}
Exec if $raw_event =~ /^ *$/ drop();
Exec to_syslog_bsd();
On Tue, Sep 30, 2014 at 10:34 PM, Daniel Zorab
<daniel.zo...@derivco.co.uk> wrote:
> Hi Cameron and Botond
>
> Thanks for the advice here, this does make sense! Will test it out tomorrow
> and check the results.
>
> Regards
> Daniel
>
> Sent from my iPhone
>
>> On 30 Sep 2014, at 19:40, "Botond Botyanszki" <b...@nxlog.org> wrote:
>>
>> Hi,
>>
>> Cameron is correct. You should first verify that nxlog can parse the
>> multiline by writing a to a file after calling to_json().
>>
>> Since you are sending to logstash using om_tcp without any encapsulation,
>> logstash will treat each line in your multiline event as a separate
>> record, i.e. your multiline magic becomes effectively useless when it
>> reaches logstash.
>>
>> Regards,
>> Botond
>>
>> On Wed, 1 Oct 2014 07:31:07 +1300
>> Cameron Kerr <cameron.kerr...@gmail.com> wrote:
>>
>>> Youshould convert the format to something like JSON, and the on the
>>> LogStash receiver you read it in using the json_lines codec.
>>>
>>> Sent from my iPhone
>>>
>>>> On 1/10/2014, at 5:35 am, Daniel Zorab <daniel.zo...@derivco.co.uk> wrote:
>>>>
>>>> Hi
>>>>
>>>> I am attempting to create a proof of concept for visualizing log files by
>>>> leveraging nxlog -> logstash -> elasticsearch -> kibana. I am having
>>>> issues at the stage of filling up logtstash with information from a text
>>>> log file using nxlog and in particular the multiline portions of the log
>>>> file.
>>>>
>>>> I am using a Windows 7 x64 VM as a test machine with all the services and
>>>> applications localized to this VM.
>>>>
>>>> I have been able to send log file (single line) entries successfully from
>>>> the log file using nxlog through to logstash which then parses and stores
>>>> each event as it should. The issue comes in when attempting to
>>>> support/send multline entries through to logstash which is producing some
>>>> inconsistent results. (If I send through each as a single line it works
>>>> fine but obviously the multiline log entries get stored into their own
>>>> event) .Basically it looks like it is sending several single line entries
>>>> and logstash is viewing them as one single event (as per debugoutput.txt
>>>> eg: line 77 ->154 ) which can be compared against the input of
>>>> SampleLog.txt
>>>>
>>>> If using logstash on its own (no nxlog involvement) it is able to parse
>>>> and read multiline and single line inputs absoluytely fine as per the and
>>>> the logtash config file : Logstash.conf.
>>>> I have tried out multiple scenarios in the nxlog config by
>>>> enabling/disabling the xm_multiline module and utlising the HeaderLine and
>>>> EndLine. I have also tried disabling the multline config portion of the
>>>> logtstash unto no avail.
>>>>
>>>> Could anyone shed some more light on this issue or have I misunderstood
>>>> how to utilise the config for nxlog?
>>>>
>>>> Regards
>>>> Daniel
>>>> <Logstash.conf.txt>
>>>> <nxlog.conf.txt>
>>>> <rubydebug output.txt>
>>>> <SampleLog.txt>
>>>> ------------------------------------------------------------------------------
>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
>>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
>>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> nxlog-ce-users mailing list
>>>> nxlog-ce-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>>
>> ------------------------------------------------------------------------------
>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>> _______________________________________________
>> nxlog-ce-users mailing list
>> nxlog-ce-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
