[
https://issues.apache.org/jira/browse/LOG4J2-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiří Smolík updated LOG4J2-3425:
--------------------------------
Description:
Recently, we've had an issue with our syslog server (collector), where it was
suddenly unable to forward messages sent to it. This resulted in flooding the
server and ultimately, exhausting its resources (memory and allocated temporary
storage).
However, more interestingly, Log4j2 appender behaved as follows:
# At the moment of exhaustion, one thread (T1) entered the synchronized block
in TcpSocketManager (see
[here|[https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217]])
and tried to write the message (see
[here|[https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253|https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L219)),]]).
# The server didn't respond immediately - actually, it blocked T1 for hours
(it was most likely waiting for resources to be free).
# In the meantime, all threads attempting to write a message to the syslog
appender hanged at the synchronized block (they were waiting for T1 to finish).
Eventually, the connection to the server broke apart or some resources were
freed, and T1 finished. Interestingly, it didn't throw any exception, but that
is beside the point in this issue.
So, what happened is that the server got flooded and blocked one thread in our
application, but due to the way TcpSocketManager is implemented, our
application suffered a DoS.
As a precaution to such dreadful scenario, I would like to request adding a
read timeout setting to the TcpSocketManager (also syslog appender), which
would in case of the above described situation prevent T1 from blocking other
threads for a long time. At least, that's my idea of an easy fix. Connect and
reconnect timeout settings have a different purpose and were useless in this
case.
I am aware that the issue is also a matter of the syslog server configuration,
but application developers do not always have it under their control.
was:
Recently, we've had an issue with our syslog server (collector), where it was
suddenly unable to forward messages sent to it. This resulted in flooding the
server and ultimately, exhausting its resources (memory and allocated temporary
storage).
However, more interestingly, Log4j2 appender behaved as follows:
# At the moment of exhaustion, one thread (T1) entered the synchronized block
in TcpSocketManager (see
[here](https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217))
and tried to write the message (see
[here]([https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253)).|https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L219)),]
# The server didn't respond immediately - actually, it blocked T1 for hours
(it was most likely waiting for resources to be free).
# In the meantime, all threads attempting to write a message to the syslog
appender hanged at the synchronized block (they were waiting for T1 to finish).
Eventually, the connection to the server broke apart or some resources were
freed, and T1 finished. Interestingly, it didn't throw any exception, but that
is beside the point in this issue.
So, what happened is that the server got flooded and blocked one thread in our
application, but due to the way TcpSocketManager is implemented, our
application suffered a DoS.
As a precaution to such dreadful scenario, I would like to request adding a
read timeout setting to the TcpSocketManager (also syslog appender), which
would in case of the above described situation prevent T1 from blocking other
threads for a long time. At least, that's my idea of an easy fix. Connect and
reconnect timeout settings have a different purpose and were useless in this
case.
I am aware that the issue is also a matter of the syslog server configuration,
but application developers do not always have it under their control.
> Syslog appender lacks the 'readTimeoutMillis' setting
> -----------------------------------------------------
>
> Key: LOG4J2-3425
> URL: https://issues.apache.org/jira/browse/LOG4J2-3425
> Project: Log4j 2
> Issue Type: Improvement
> Components: Appenders
> Affects Versions: 2.12.1
> Reporter: Jiří Smolík
> Priority: Major
>
> Recently, we've had an issue with our syslog server (collector), where it was
> suddenly unable to forward messages sent to it. This resulted in flooding the
> server and ultimately, exhausting its resources (memory and allocated
> temporary storage).
> However, more interestingly, Log4j2 appender behaved as follows:
> # At the moment of exhaustion, one thread (T1) entered the synchronized
> block in TcpSocketManager (see
> [here|[https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L217]])
> and tried to write the message (see
> [here|[https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L253|https://github.com/apache/logging-log4j2/blob/e2ee2f4dd9327eb1841023c3269ba12f5673e694/log4j-core/src/main/java/org/apache/logging/log4j/core/net/TcpSocketManager.java#L219)),]]).
> # The server didn't respond immediately - actually, it blocked T1 for hours
> (it was most likely waiting for resources to be free).
> # In the meantime, all threads attempting to write a message to the syslog
> appender hanged at the synchronized block (they were waiting for T1 to
> finish).
> Eventually, the connection to the server broke apart or some resources were
> freed, and T1 finished. Interestingly, it didn't throw any exception, but
> that is beside the point in this issue.
> So, what happened is that the server got flooded and blocked one thread in
> our application, but due to the way TcpSocketManager is implemented, our
> application suffered a DoS.
> As a precaution to such dreadful scenario, I would like to request adding a
> read timeout setting to the TcpSocketManager (also syslog appender), which
> would in case of the above described situation prevent T1 from blocking other
> threads for a long time. At least, that's my idea of an easy fix. Connect and
> reconnect timeout settings have a different purpose and were useless in this
> case.
> I am aware that the issue is also a matter of the syslog server
> configuration, but application developers do not always have it under their
> control.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
