Kaushik Vankayala created LOG4J2-2926:
-----------------------------------------
Summary: Application OUTAGE due to Unable to write to stream TCP
Key: LOG4J2-2926
URL: https://issues.apache.org/jira/browse/LOG4J2-2926
Project: Log4j 2
Issue Type: Bug
Components: Appenders
Affects Versions: 2.13.3
Environment: Mulesoft, Linux, ELK (hosted service on AWS)
Reporter: Kaushik Vankayala
Fix For: 2.13.3
Hi Team, we have recently encountered an outage in our PRODUCTION application.
We have custom logging using log4j2 and the remote server was out of storage.
We suspect we got the issue because of the same reason and the ERROR we faced
is as below;
2020-08-30 22:23:04,686 Log4j2-TF-17-AsyncLoggerConfig-9 ERROR Unable to write
to stream
TCP:[api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com|http://api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com/]:8500
for appender SOCKET
org.apache.logging.log4j.core.appender.AppenderLoggingException: Error sending
to
TCP:[api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com|http://api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com/]:8500
for
[api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com/52.221.23.118:8500|http://api-manager-2623b9734249246e.elb.ap-southeast-1.amazonaws.com/52.221.23.118:8500]
at
org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:231)
at
org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:190)
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.writeByteArrayToManager(AbstractOutputStreamAppender.java:206)
at
org.apache.logging.log4j.core.appender.SocketAppender.directEncodeEvent(SocketAppender.java:459)
at
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
"http.listener.02 SelectorRunner" #76 prio=5 os_prio=0 tid=0x00007f314c52d800
nid=0xb19 waiting for monitor entry [0x00007f314a6fc000]
java.lang.Thread.State: BLOCKED (on object monitor) at
org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor.enqueue(AsyncLoggerConfigDisruptor.java:376)
- waiting to lock <0x0000000088b43a58> (a java.lang.Object) at
org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor.enqueueEvent(AsyncLoggerConfigDisruptor.java:330)
at
org.apache.logging.log4j.core.async.AsyncLoggerConfig.logInBackgroundThread(AsyncLoggerConfig.java:159)
at
org.apache.logging.log4j.core.async.EventRoute$1.logMessage(EventRoute.java:46)
We tried to follow the link
([https://help.mulesoft.com/s/article/Mule-instance-which-implements-a-log4j2-SocketAppender-complains-with-Broken-Pipe-Error]).
Unlike splunk we have ELK in our architecture. Our Socket appender looks like
below
{{<Socket name="SOCKET" host="${sys:tcp.host}" port="${sys:tcp.port}"
reconnectDelayMillis="30000" immediateFail="false" bufferedIo="true"
bufferSize="204800" protocol="TCP" immediateFlush="false">}}
We have couple of queries below if you could kindly address them;
# With the current Socket Appender what additional tags may be needed to
independently stream the logs irrestive of the remote destination status
# Our ELK server is a hosted servie. The first point after Cloudhub is a Load
Balancer after which there is an EC2 server where Logstash is running. Do we
need to configure any keep-alive configuration at the O/S level?
# Why should a storage issue at a remote destination cause an issue in the
socket appender and eventually fail the running of an application. Logging by
socket appender should ideally be an independent activithy.
Finally, we would request you to recommend a solution for the case where the
remote endpoint storage is exhausted or there may be any TCP sockets dead, and
how we can avoid the OUTAGE of MuleSoft application due to a logging problem by
Log4j2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
