Hi,
TCP syslog does not guarantee message because there is no application
level acknowledgement in the log transfer protocol.
When sending the data, this is first placed in the socket buffers of the
operating system. If there is a connection error, this data will be lost.
This is not a bug in nxlog. That's how it works in everything else
(rsyslog, logstash, whatever) because it is not possible to solve this
other than to use a different application level protocol full reliability
such as http, relp, etc.
Regards,
Botond
On Thu, 10 Jul 2014 15:52:26 +0200
Sławomir Musiał <slawomir.mus...@outlook.com> wrote:
>
>
>
> Hello,
>
> I use Nxlog in version 2.7.1191 and I want to send data using tcp between two
> nxlog hosts. I set a simple configuration:
>
> 1st host nxlog.conf modules:
> <Input in_file>
> Module im_file
> File "input.txt"
> SavePos TRUE
> </Input>
> <Output out_tcp>
> Module om_tcp
> Host {dst_addr}
> Port 514
> </Output>
> <Route route1>
> Path in_file => out_tcp
> </Route>
>
> 2nd host nxlog.conf modules:
> <Input in_tcp>
> Module im_tcp
> Host 0.0.0.0
> Port 514
> </Input>
> <Output out_file>
> Module om_file
> File "output.txt"
> </Output>
> <Route route2>
> Path in_tcp => out_file
> </Route>
>
> Normally, everything works fine - I write messages to input.txt, they are
> sent over tcp to remote host and saved into output.txt. The problem occurs
> when while sending the data connection between hosts is lost.
> While sending data I unplugged the network cable and plugged it again.
> Comparing input and output files it looks like data which was about to be
> sent right before disconnection was lost ( about 7kBs missing ).
>
> In
> https://www.mail-archive.com/nxlog-ce-users@lists.sourceforge.net/msg00106.html
> you wrote:
> "When om_tcp detects that it cannot send, the input module (im_msvistalog)
> will stop reading logs. When the tcp connection is resumed, the input module
> will resume reading from the same position and will properly collect all
> messages that were stored in the windows eventlog during that time. This
> works the same way with im_file."
>
> However, it looks like in my case after reconnecting, input module resumed
> sending data from the wrong position. The same situation occurs when ssl
> module is used.
> The problem appears mostly when big amount of data is sent. Maybe in this
> case information about connection loss is passed to input module too late,
> the position is already shifted, but data not sent. Therefore, after
> reconnection it starts from already shifted position.
>
> Here are logs from nxlog.log file ( from DEBUG mode ) that probably describe
> what happens in dissconnection moment:
>
> 2014-07-10 12:09:33 DEBUG worker 0 got signal for new job
> 2014-07-10 12:09:33 DEBUG worker 0 processing event 0x863f38
> 2014-07-10 12:09:33 DEBUG worker 1 waiting for new event
> 2014-07-10 12:09:33 DEBUG PROCESS_EVENT: POLL (out_nxlog)
> 2014-07-10 12:09:33 DEBUG nx_module_pollset_poll: out_nxlog
> 2014-07-10 12:09:33 DEBUG module out_nxlog got 1 poll events
> 2014-07-10 12:09:33 DEBUG Module out_nxlog can read
> 2014-07-10 12:09:33 DEBUG Module out_nxlog can write
> 2014-07-10 12:09:33 DEBUG nx_event_to_jobqueue: READ (out_nxlog)
> 2014-07-10 12:09:33 DEBUG event added to jobqueue
> 2014-07-10 12:09:33 DEBUG nx_event_to_jobqueue: WRITE (out_nxlog)
> 2014-07-10 12:09:33 DEBUG event added to jobqueue
> 2014-07-10 12:09:33 DEBUG worker 1 got signal for new job
> 2014-07-10 12:09:33 DEBUG worker 1 processing event 0x3d8720
> 2014-07-10 12:09:33 DEBUG worker 0 waiting for new event
> 2014-07-10 12:09:33 DEBUG PROCESS_EVENT: READ (out_nxlog)
> 2014-07-10 12:09:33 DEBUG om_tcp read
> 2014-07-10 12:09:33 DEBUG STOP: out_nxlog
> 2014-07-10 12:09:33 DEBUG om_tcp_stop
> 2014-07-10 12:09:33 DEBUG om_tcp closing socket
> 2014-07-10 12:09:33 DEBUG remove socket [25]
> 2014-07-10 12:09:33 DEBUG om_tcp_stop
> 2014-07-10 12:09:33 INFO reconnecting in 1 seconds
> 2014-07-10 12:09:33 DEBUG new event in event_thread [in_file:MODULE_SPECIFIC]
> 2014-07-10 12:09:33 ERROR [om_tcp.c:358/io_err_handler()] -;
> [om_tcp.c:206/om_tcp_read()] om_tcp detected a connection error; An existing
> connection was forcibly closed by the remote host.
> 2014-07-10 12:09:33 DEBUG new event in event_thread [out_nxlog:RECONNECT]
> 2014-07-10 12:09:33 DEBUG future event, event thread sleeping 796875ms in
> cond_timedwait
> 2014-07-10 12:09:33 DEBUG worker 1 processing event 0x85af28
> 2014-07-10 12:09:33 DEBUG PROCESS_EVENT: WRITE (out_nxlog)
> 2014-07-10 12:09:33 DEBUG om_tcp_write
> 2014-07-10 12:09:33 DEBUG module out_nxlog not running, not writing any more
> data
> 2014-07-10 12:09:33 DEBUG worker 1 waiting for new event
> 2014-07-10 12:09:34 DEBUG new event in event_thread [in_file:MODULE_SPECIFIC]
> 2014-07-10 12:09:34 DEBUG nx_event_to_jobqueue: MODULE_SPECIFIC (in_file)
> 2014-07-10 12:09:34 DEBUG event added to jobqueue
> 2014-07-10 12:09:34 DEBUG new event in event_thread [out_nxlog:RECONNECT]
> 2014-07-10 12:09:34 DEBUG future event, event thread sleeping 203125ms in
> cond_timedwait
> 2014-07-10 12:09:34 DEBUG worker 0 got signal for new job
> 2014-07-10 12:09:34 DEBUG worker 0 processing event 0x3d8758
> 2014-07-10 12:09:34 DEBUG PROCESS_EVENT: MODULE_SPECIFIC (in_file)
>
> Of course, I can attach the whole nxlog.log file if needed.
> This situation is repeatable. How can I solve the problem?
>
> Thank you for help in advance,
> Slawomir
>
>
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
