Hi,
I found the solution. I was just doing
- nano /var/log/test.log [ENTER]
- [SCROLL DOWN TO THE END OF THE FILE]
- Test [ENTER]
- [Ctrl+O] [ENTER] (saving in nano)
I tried without modifying $raw_event and it still is truncating the line,
so problem was not in the line modifying the $raw_event. Then I discovered
using
echo 'Test' >> /var/log/test.log
It works! So my procedure using nano was the culprit. Why? I am not that
interested in knowing why because I wont be editing log files but it is
weird anyway. But what if an application makes something similar to
whatever nano and me were doing there?
Thanks for pointing me in the right direction.
Enrique.
On Fri, Jul 26, 2013 at 4:44 AM, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi,
>
> How are you "adding" a line to the file?
> I tested with this configuration and it works ok on debian.
>
> Regards,
> Botond
>
> On Thu, 25 Jul 2013 10:42:25 -0400
> Enrique Cadalso <enrique.cada...@blendedperspectives.com> wrote:
>
> > Hi Botond,
> >
> > Thanks for the advice, I followed it and changed nxlog.conf to
> >
> > root@Debian607:~# cat /etc/nxlog/nxlog.conf
> > > ########################################
> > > # Global directives #
> > > ########################################
> > > User nxlog
> > > Group nxlog
> > > LogFile /var/log/nxlog/nxlog.log
> > > LogLevel INFO
> > > ########################################
> > > # Inputs #
> > > ########################################
> > > <Input internal>
> > > Module im_internal
> > > </Input>
> > >
> > > <Input confluence>
> > > Module im_file
> > > File "*/var/log/test.log*"
> > > SavePos TRUE
> > > Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> > > </Input>
> > >
> > > ########################################
> > > # Output #
> > > ########################################
> > >
> > > <Output confluence_out>
> > > Module om_tcp
> > > Host localhost
> > > Port 3516
> > > </Output>
> > > *<Output nxlog_out>
> > > Module om_file
> > > File "/var/log/nxlog/nxlog.out"
> > > </Output>
> > > *
> > > ########################################
> > > # Routes #
> > > ########################################
> > >
> > > <Route confluence>
> > > Path confluence => confluence_out, *nxlog_out*
> > > </Route>
> >
> >
> >
> > However when adding a line "Test" to the */var/log/test.log *file I get
> in
> > the */var/log/nxlog/nxlog.out* file the truncated line, first character
> > missing. All is working in a vagrant vm with Debian 607, running in a
> Mac,
> > just in case that matters.
> >
> > root@Debian607:~# tail /var/log/nxlog/nxlog.log
> > > ...
> > > 2013-07-25 14:32:07 INFO nxlog-ce-2.5.1089 started
> > > 2013-07-25 14:32:07 INFO connecting to localhost:3516
> > > root@Debian607:~# tail /var/log/nxlog/nxlog.out
> > > *est*|server:Debian607.vagrantup.com|
> > > root@Debian607:~#
> >
> >
> >
> > I also enabled maximum debug level on logstash side and this is the first
> > notice of the event, with the message already truncated.
> >
> > {:timestamp=>"2013-07-25T14:17:01.773000+0000", :message=>"heartbeat",
> > > :level=>:debug, :file=>
> > > "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb",
> :line=>
> > > "35", :method=>"debug"}
> > > {:timestamp=>"2013-07-25T14:17:03.779000+0000", :message=>"heartbeat",
> > > :level=>:debug, :file=>
> > > "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb",
> :line=>
> > > "35", :method=>"debug"}
> > > {:timestamp=>"2013-07-25T14:17:04.139000+0000", :message=>"Received
> newevent",
> > > :source=>"tcp://127.0.0.1:46544/",
> :event=>#<LogStash::Event:0x20c885fe
> > > @data={"@source"=>"tcp://127.0.0.1:46544/", "@tags"=>["nxlog"],
> "@fields"=>{},
> > > "@timestamp"=>"2013-07-25T14:17:04.135Z", "@source_host"=>"127.0.0.1",
> > > "@source_path"=>"/", "@message"=>"*est*|server:Debian607.vagrantup.com
> |\n",
> > > "@type"=>"confluence"}, @cancelled=false>, :level=>:debug, :file=>
> > > "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb",
> :line=>
> > > "35", :method=>"debug"}
> > > {:timestamp=>"2013-07-25T14:17:04.151000+0000", :message=>"est|server:
> > > Debian607.vagrantup.com|\n", :pattern=>"^20", :match=>false,
> :negate=>true,
> > > :level=>:debug, :file=>
> > > "/opt/logstash/logstash-1.1.13-flatjar.jar!/logstash/logging.rb",
> :line=>
> > > "35", :method=>"debug"}
> >
> >
> > Regards,
> >
> > Enrique.
> >
> > ---------- Forwarded message ----------
> > From: Botond Botyanszki <b...@nxlog.org>
> > Date: Thu, Jul 25, 2013 at 3:56 AM
> > Subject: Re: [nxlog-ce-users] Truncated logs from nxlog
> > To: nxlog-ce-users@lists.sourceforge.net
> >
> >
> > Hi,
> >
> > I suggest changing the output to om_file or using tcpdump/wireshark to
> > see what gets sent over the network.
> >
> > The problem is most likely on the receiver (logstash) side.
> >
> > Regards,
> > Botond
> >
> >
> > On Tue, 23 Jul 2013 17:55:35 -0400
> > Enrique Cadalso <enrique.cada...@blendedperspectives.com> wrote:
> >
> > > Hi,
> > >
> > > I want to include the fqdn of the server sending logs in the
> raw_event. I
> > > am using this configuration
> > >
> > > <Input confluence>
> > > Module im_file
> > > File "/var/log/test.log"
> > > SavePos TRUE
> > > Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> > > </Input>
> > >
> > > Sending the log to logstash in the same host for testing using
> > >
> > > <Output confluence_out>
> > > Module om_tcp
> > > Host localhost
> > > Port 3516
> > > </Output>
> > >
> > > And I am receiving the log truncated, missing first character of the
> log
> > > line. Testing with "Test" string I receive
> > >
> > > {
> > > "@source" => "tcp://127.0.0.1:49388/",
> > > "@tags" => [
> > > [0] "nxlog",
> > > [1] "_jsonparsefailure",
> > > [2] "multiline",
> > > [3] "_grokparsefailure"
> > > ],
> > > "@fields" => {
> > > "log_level_alert" => [
> > > [0] "%{@fields.log_level}"
> > > ]
> > > },
> > > "@timestamp" => "2013-07-23T21:11:59.945Z",
> > > "@source_host" => "127.0.0.1",
> > > "@source_path" => "/",
> > > "@message" => "*est*|server:Debian607.vagrantup.com|\n",
> > > "@type" => "confluence"
> > > }
> > >
> > > What is truncating the first character of the log? Is there a better
> way
> > to
> > > include the fqdn in the event?
> > >
> > > Thanks
> > >
> > > Enrique.
> > >
> > >
> > >
> > >
> > > Full nxlog.conf (in the sender)
> > >
> > > root@Debian607:~# cat /etc/nxlog/nxlog.conf
> > > ########################################
> > > # Global directives #
> > > ########################################
> > >
> > > User nxlog
> > > Group nxlog
> > > LogFile /var/log/nxlog/nxlog.log
> > > LogLevel INFO
> > >
> > > ########################################
> > > # Inputs #
> > > ########################################
> > >
> > > <Input internal>
> > > Module im_internal
> > > </Input>
> > >
> > >
> > > <Input confluence>
> > > Module im_file
> > > File "/var/log/test.log"
> > > SavePos TRUE
> > > Exec $raw_event=$raw_event+'|server:'+hostname_fqdn()+'|';
> > > </Input>
> > >
> > >
> > > ########################################
> > > # Output #
> > > ########################################
> > >
> > >
> > > <Output confluence_out>
> > > Module om_tcp
> > > Host localhost
> > > Port 3516
> > > </Output>
> > >
> > >
> > > ########################################
> > > # Routes #
> > > ########################################
> > >
> > >
> > > <Route confluence>
> > > Path confluence => confluence_out
> > > </Route>
> > >
> > >
> >
> -------------------------------------------------------------------------------------------------
> > > Full logstash.conf (in the receiver)
> > >
> > >
> > > input {
> > > tcp {
> > > type => "confluence"
> > > port => 3516
> > > format => 'json'
> > > tags => ["nxlog"]
> > > }
> > > }
> > >
> > > filter {
> > > multiline {
> > > type => "confluence"
> > > pattern => "^20"
> > > negate => true
> > > what => "previous"
> > > add_tag => ["multiline_confluence"]
> > > }
> > >
> > > grok {
> > > type => "confluence"
> > > pattern => "%{DATESTAMP},%{NUMBER} %{LOGLEVEL:log_level}
> > > \[%{JAVAFILE:thread}\] \[%{JAVACLASS:class}\]"
> > > add_tag => ["matched_confluence"]
> > > }
> > >
> > > }
> > >
> > > output {
> > > stdout {
> > > type => "confluence"
> > > debug => true
> > > }
> > > }
> >
> >
> ------------------------------------------------------------------------------
> > See everything from the browser to the database with AppDynamics
> > Get end-to-end visibility with application monitoring from AppDynamics
> > Isolate bottlenecks and diagnose root cause in seconds.
> > Start your free trial of AppDynamics Pro today!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> > _______________________________________________
> > nxlog-ce-users mailing list
> > nxlog-ce-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
