Maxence Dunnewind created AMQ-6473:
--------------------------------------
Summary: Old destination GC blocks other requests
Key: AMQ-6473
URL: https://issues.apache.org/jira/browse/AMQ-6473
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.14.1
Reporter: Maxence Dunnewind
Using ActiveMQ with 5.14.1, we create a lot of topics through a WSS + Stomp
setup. We have enabled :
* keepalive for consumers ( every 25s)
* GC for old destinations (topics)
The current relevant config is :
{quote}
<!-- Check for destination purge every minute (ie every 60 000 ms) -->
<broker xmlns="http://activemq.apache.org/schema/core";
brokerName="broker-main" dataDirectory="${activemq.data}"
schedulePeriodForDestinationPurge="60000">
<destinationPolicy>
<policyMap>
<policyEntries>
<!-- Remove topic inactive since 30 minutes (aka 30 * 600 s
aka 600000ms) -->
<policyEntry topic=">"
gcInactiveDestinations="true" inactiveTimoutBeforeGC="1800000">
<!-- The
constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block
producers and affect other consumers
by limiting the number of
messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy
limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
[...]
<transportConnectors>
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600?transport.closeAsync=false"/>
<transportConnector name="ws"
uri="ws://0.0.0.0:61614?maximumConnections=100000&wireFormat.maxFrameSize=104857600?transport.closeAsync=false"/>
<transportConnector name="wss"
uri="wss://0.0.0.0:61624?maximumConnections=100000&wireFormat.maxFrameSize=104857600?transport.closeAsync=false"/>
</transportConnectors>
{quote}
However, it seems that when a lot of topics are removed (I'm almost sure that
topic created for less than 30 minutes are removed btw, still need to check
this) , the whole broker is almost blocked. Has side effects :
* a lot of clients are disconnected with :
{quote}
Transport Connection to: wss://X.X.X.X:56088 failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive (no
connection attempt made) for too (>30000) long: wss://85.95.201.56:56088 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker
{quote}
* when I try to curl http://localhost:8161/admin/xml/topics.jsp I get an
exception in the logs :
{quote}
19: <c:forEach items="${requestContext.brokerQuery.topics}" var="row">
20: <topic name="<c:out value="${row.name}"/> ">
21:
22: <stats size="${row.queueSize}"
23: consumerCount="${row.consumerCount}"
Stacktrace:
at
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:568)[apache-jsp-8.0.9.M3.jar:2.3]
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)[apache-jsp-8.0.9.M3.jar:2.3]
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:405)[apache-jsp-8.0.9.M3.jar:2.3]
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:349)[apache-jsp-8.0.9.M3.jar:2.3]
at
org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:107)[apache-jsp-9.2.13.v20150730.jar:9.2.13.v20150730]
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)[tomcat-servlet-api-8.0.24.jar:]
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)[spring-web-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)[spring-web-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:45)[activemq-web-5.14.1.jar:5.14.1]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:102)[file:/opt/apache-activemq-5.14.1/webapps/admin/WEB-INF/classes/:]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.Server.handle(Server.java:499)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_66]
2016-10-21 14:05:01,241 | WARN | /admin/xml/topics.jsp |
org.eclipse.jetty.server.HttpChannel | qtp106297322-46
javax.servlet.ServletException: javax.servlet.ServletException:
org.apache.jasper.JasperException: An exception occurred processing JSP page
/xml/topics.jsp at line 20
17: --%>
18: <topics>
19: <c:forEach items="${requestContext.brokerQuery.topics}" var="row">
20: <topic name="<c:out value="${row.name}"/> ">
21:
22: <stats size="${row.queueSize}"
23: consumerCount="${row.consumerCount}"
Stacktrace:
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:130)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.Server.handle(Server.java:499)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_66]
2016-10-21 14:05:01,244 | WARN | Could not send response error 500:
javax.servlet.ServletException: javax.servlet.ServletException:
org.apache.jasper.JasperException: An exception occurred processing JSP page
/xml/topics.jsp at line 20
17: --%>
18: <topics>
19: <c:forEach items="${requestContext.brokerQuery.topics}" var="row">
20: <topic name="<c:out value="${row.name}"/> ">
21:
22: <stats size="${row.queueSize}"
23: consumerCount="${row.consumerCount}"
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
