Hi Botond,
Thanks for the advice, I followed it and changed nxlog.conf to
root@Debian607:~# cat /etc/nxlog/nxlog.conf
> ########################################
> # Global directives #
> ########################################
> User nxlog
> Group nxlog
> LogFile /var/log/nxlog/nxlog.log
> LogLevel INFO
> ########################################
> # Inputs #
> ########################################
> <Input internal>
> Module im_internal
> </Input>
>
> <Input confluence>
> Module im_file
> File "*/var/log/test.log*"
> SavePos TRUE
> Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> </Input>
>
> ########################################
> # Output #
> ########################################
>
> <Output confluence_out>
> Module om_tcp
> Host localhost
> Port 3516
> </Output>
> *<Output nxlog_out>
> Module om_file
> File "/var/log/nxlog/nxlog.out"
> </Output>
> *
> ########################################
> # Routes #
> ########################################
>
> <Route confluence>
> Path confluence => confluence_out, *nxlog_out*
> </Route>
However when adding a line "Test" to the */var/log/test.log *file I get in
the */var/log/nxlog/nxlog.out* file the truncated line, first character
missing. All is working in a vagrant vm with Debian 607, running in a Mac,
just in case that matters.
root@Debian607:~# tail /var/log/nxlog/nxlog.log
> ...
> 2013-07-25 14:32:07 INFO nxlog-ce-2.5.1089 started
> 2013-07-25 14:32:07 INFO connecting to localhost:3516
> root@Debian607:~# tail /var/log/nxlog/nxlog.out
> *est*|server:Debian607.vagrantup.com|
> root@Debian607:~#
I also enabled maximum debug level on logstash side and this is the first
notice of the event, with the message already truncated.
{:timestamp=>"2013-07-25T14:17:01.773000+0000", :message=>"heartbeat",
> :level=>:debug, :file=>
> "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb", :line=>
> "35", :method=>"debug"}
> {:timestamp=>"2013-07-25T14:17:03.779000+0000", :message=>"heartbeat",
> :level=>:debug, :file=>
> "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb", :line=>
> "35", :method=>"debug"}
> {:timestamp=>"2013-07-25T14:17:04.139000+0000", :message=>"Received newevent",
> :source=>"tcp://127.0.0.1:46544/", :event=>#<LogStash::Event:0x20c885fe
> @data={"@source"=>"tcp://127.0.0.1:46544/", "@tags"=>["nxlog"], "@fields"=>{},
> "@timestamp"=>"2013-07-25T14:17:04.135Z", "@source_host"=>"127.0.0.1",
> "@source_path"=>"/", "@message"=>"*est*|server:Debian607.vagrantup.com|\n",
> "@type"=>"confluence"}, @cancelled=false>, :level=>:debug, :file=>
> "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb", :line=>
> "35", :method=>"debug"}
> {:timestamp=>"2013-07-25T14:17:04.151000+0000", :message=>"est|server:
> Debian607.vagrantup.com|\n", :pattern=>"^20", :match=>false, :negate=>true,
> :level=>:debug, :file=>
> "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb", :line=>
> "35", :method=>"debug"}
Regards,
Enrique.
---------- Forwarded message ----------
From: Botond Botyanszki <b...@nxlog.org>
Date: Thu, Jul 25, 2013 at 3:56 AM
Subject: Re: [nxlog-ce-users] Truncated logs from nxlog
To: nxlog-ce-users@lists.sourceforge.net
Hi,
I suggest changing the output to om_file or using tcpdump/wireshark to
see what gets sent over the network.
The problem is most likely on the receiver (logstash) side.
Regards,
Botond
On Tue, 23 Jul 2013 17:55:35 -0400
Enrique Cadalso <enrique.cada...@blendedperspectives.com> wrote:
> Hi,
>
> I want to include the fqdn of the server sending logs in the raw_event. I
> am using this configuration
>
> <Input confluence>
> Module im_file
> File "/var/log/test.log"
> SavePos TRUE
> Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> </Input>
>
> Sending the log to logstash in the same host for testing using
>
> <Output confluence_out>
> Module om_tcp
> Host localhost
> Port 3516
> </Output>
>
> And I am receiving the log truncated, missing first character of the log
> line. Testing with "Test" string I receive
>
> {
> "@source" => "tcp://127.0.0.1:49388/",
> "@tags" => [
> [0] "nxlog",
> [1] "_jsonparsefailure",
> [2] "multiline",
> [3] "_grokparsefailure"
> ],
> "@fields" => {
> "log_level_alert" => [
> [0] "%{@fields.log_level}"
> ]
> },
> "@timestamp" => "2013-07-23T21:11:59.945Z",
> "@source_host" => "127.0.0.1",
> "@source_path" => "/",
> "@message" => "*est*|server:Debian607.vagrantup.com|\n",
> "@type" => "confluence"
> }
>
> What is truncating the first character of the log? Is there a better way
to
> include the fqdn in the event?
>
> Thanks
>
> Enrique.
>
>
>
>
> Full nxlog.conf (in the sender)
>
> root@Debian607:~# cat /etc/nxlog/nxlog.conf
> ########################################
> # Global directives #
> ########################################
>
> User nxlog
> Group nxlog
> LogFile /var/log/nxlog/nxlog.log
> LogLevel INFO
>
> ########################################
> # Inputs #
> ########################################
>
> <Input internal>
> Module im_internal
> </Input>
>
>
> <Input confluence>
> Module im_file
> File "/var/log/test.log"
> SavePos TRUE
> Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> </Input>
>
>
> ########################################
> # Output #
> ########################################
>
>
> <Output confluence_out>
> Module om_tcp
> Host localhost
> Port 3516
> </Output>
>
>
> ########################################
> # Routes #
> ########################################
>
>
> <Route confluence>
> Path confluence => confluence_out
> </Route>
>
>
-------------------------------------------------------------------------------------------------
> Full logstash.conf (in the receiver)
>
>
> input {
> tcp {
> type => "confluence"
> port => 3516
> format => 'json'
> tags => ["nxlog"]
> }
> }
>
> filter {
> multiline {
> type => "confluence"
> pattern => "^20"
> negate => true
> what => "previous"
> add_tag => ["multiline_confluence"]
> }
>
> grok {
> type => "confluence"
> pattern => "%{DATESTAMP},%{NUMBER} %{LOGLEVEL:log_level}
> \[%{JAVAFILE:thread}\] \[%{JAVACLASS:class}\]"
> add_tag => ["matched_confluence"]
> }
>
> }
>
> output {
> stdout {
> type => "confluence"
> debug => true
> }
> }
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
