Hi,
I have an issue with nxlog, I'm sending all logs from my servers through
HAProxy servers to 3 nxlog servers using SSL with one port.
I'm trying to define a simple check in the HAProxy to check that port is
alive (basic TCP check). And after few minutes I'm not getting an answer
from nxlog server on the port (even with simple 'nc' command to the port),
so the HAProxy getting timeouts to the servers and declare they're 'DOWN'.
I've checked nxlog log and saw two 'weird' things:
1. the following lines repeat:
2016-05-05 07:27:23 INFO SSL connection accepted from <HAPROXY_IP>:36852
2016-05-05 07:27:23 ERROR SSL error, SSL_ERROR_SSL: retval -1, ssl
handshake failure,
2016-05-05 07:27:23 WARNING SSL connection closed from <HAPROXY_IP>:36854
These warnings are repeating for all HAProxy servers.
2. The following error:
2016-05-05 07:27:31 ERROR apr_file_write failed;Bad file descriptor
2016-05-05 07:27:31 ERROR last message repeated 97 times
Also sometimes I see the following error:
ERROR ### PANIC at line 2456 in module.c/nx_module_pollset_add_socket():
"failed to add descriptor to pollset: Operation not permitted
;backtrace:;/usr/bin/nxlog(nx_append_backtrace+0x66)
[0x437846];/usr/bin/nxlog(_nx_panic+0x15c)
[0x415ddf];/usr/bin/nxlog(nx_module_pollset_add_socket+0x27b)
[0x41ec1f];/usr/libexec/nxlog/modules/input/im_ssl.so(+0x411b)
[0x7fa0bcb3a11b];/usr/libexec/nxlog/modules/input/im_ssl.so(+0x4b5b)
[0x7fa0bcb3ab5b];/usr/libexec/nxlog/modules/input/im_ssl.so(+0x5e9c)
[0x7fa0bcb3be9c];/usr/bin/nxlog(nx_event_process+0x203)
[0x43e79c];/usr/bin/nxlog() [0x4524d7];/usr/bin/nxlog()
[0x445d00];/lib64/libpthread.so.0(+0x7dc5)
[0x7fa0bfb8fdc5];/lib64/libc.so.6(clone+0x6d) [0x7fa0bf8bcc9d]" ###
I've checked open files for root (nxlog runs under root) and limit is
60000, while 'sudo lsof | wc -l' showed only total of 1656.
Also, there's no high CPU, IO Wait or memory consumption. I'm using '
nxlog-ce.x86_64 2.9.1504-1' on amazon linux.
Does someone has any idea what's going on here? I don't think it's related
to HAProxy server, because I'm trying to use 'nc' command from any server
with no luck.
Thanks in advance,
Ben
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
