Hi, thanks for the response. Some questions on these points from the troubleshooting.
1. *It contains a pending message for a destination or durable topic subscription* This seems a little flawed, if a consumer who I have little control of is mis-behaving then my ActiveMQ can end up shutting down and unrecoverable. Is there some way of timing this out or similar? *2. It contains an ack for a message which is in an in-use data file - the ack cannot be removed as a recovery would then mark the message for redelivery* Same comment as 1. *3. The journal references a pending transaction* I'm not using transactions, but are there transactions under the hood? *4. It is a journal file, and there may be a pending write to it* Why would this be the case? I'll see if I can change the logging settings, since the first occurrence the number of log files does not seem to have been an issue. I have it configured to keep messages for 7 days so regardless of the above conditions I would have thought that at that expiry the log would be cleaned up so we don't end up in such a situation where the system stops and cannot restart. On 14 September 2017 at 00:02, Tim Bain <tb...@alumni.duke.edu> wrote: > Have you run the troubleshooting procedures in > http://activemq.apache.org/why-do-kahadb-log-files- > remain-after-cleanup.html? > What did they indicate? > > On Sep 12, 2017 8:27 PM, "Lionel van den Berg" <lion...@gmail.com> wrote: > > Hi all, > ActiveMQ 5.14.4 running on Suse enterprise 11 sp4 > > I'm having trouble with the log files (like db-1.log) not being cleaned up > and after about 6 weeks of running ActiveMQ shuts down because there are > too many files in the directory (I think it was 3000), it cannot be > restarted we think due to a JVM limitation. We use a mix of topics and > queues, we can't guarantee that the consumer from the third party won't be > shutdown for an extended period. We are storing messages for 7 days, or at > least we think we are, I've copied the config further down. Don't assume I > know everything that we are doing with all of the config :). I've read > various posts but I'm not sure still whether there might be a parameter > that I can set that will trigger the cleanup. > > It looks similar to this https://issues.apache.org/jira/browse/AMQ-5695 > but > the last comment there is that it is resolved in 5.14.3, I'm running 5.14.4 > on the server side at least, client libraries we did not change. > > Does anyone have any ideas? > > > > <!-- > Licensed to the Apache Software Foundation (ASF) under one or more > contributor license agreements. See the NOTICE file distributed with > this work for additional information regarding copyright ownership. > The ASF licenses this file to You under the Apache License, Version 2.0 > (the "License"); you may not use this file except in compliance with > the License. You may obtain a copy of the License at > > http://www.apache.org/licenses/LICENSE-2.0 > > Unless required by applicable law or agreed to in writing, software > distributed under the License is distributed on an "AS IS" BASIS, > WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > implied. > See the License for the specific language governing permissions and > limitations under the License. > --> > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > > <!-- Allows us to use system properties as variables in this > configuration file --> > <bean > class="org.springframework.beans.factory.config. > PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.conf}/credentials.properties</value> > </property> > </bean> > > <!-- Allows accessing the server log --> > <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery" > lazy-init="false" scope="singleton" > init-method="start" destroy-method="stop"> > </bean> > > <!-- The <broker> element is used to configure the ActiveMQ broker. --> > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="AutomationServerThree" dataDirectory="${activemq.data}"> > > <destinationPolicy> > <policyMap> > <policyEntries> > <!-- policy for all topics --> > <policyEntry topic=">" producerFlowControl="true" > useCache="true" expireMessagesPeriod="86400000" alwaysRetroactive="true"> > <pendingMessageLimitStrategy> > <constantPendingMessageLimitStrategy limit="1000"/> > </pendingMessageLimitStrategy> > <!-- discard expired messages --> > <deadLetterStrategy> > <sharedDeadLetterStrategy processExpired="false"/> > </deadLetterStrategy> > </policyEntry> > <!-- policy for all queues --> > <policyEntry queue=">" producerFlowControl="true" > useCache="true" expireMessagesPeriod="86400000"> > <pendingMessageLimitStrategy> > <constantPendingMessageLimitStrategy limit="1000"/> > </pendingMessageLimitStrategy> > <!-- discard expired messages --> > <deadLetterStrategy> > <sharedDeadLetterStrategy processExpired="false"/> > </deadLetterStrategy> > </policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > > > <!-- > The managementContext is used to configure how ActiveMQ is > exposed in > JMX. By default, ActiveMQ uses the MBean server that is started > by > the JVM. For more information, see: > > http://activemq.apache.org/jmx.html > --> > <managementContext> > <managementContext createConnector="true"/> > </managementContext> > > <!-- > Configure message persistence for the broker. The default > persistence > mechanism is the KahaDB store (identified by the kahaDB tag). > For more information, see: > > http://activemq.apache.org/persistence.html > --> > <persistenceAdapter> > <kahaDB directory="/media/SharedStorage/Binaries/activemq" > indexCacheSize="40000" > checkForCorruptJournalFiles="true"> > <locker> > <shared-native-file-locker lockAcquireSleepInterval="5000" > /> > </locker> > </kahaDB> > </persistenceAdapter> > > > <!-- > The systemUsage controls the maximum amount of space the broker > will > use before disabling caching and/or slowing down producers. For > more information, see: > http://activemq.apache.org/producer-flow-control.html > --> > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70" /> > </memoryUsage> > <storeUsage> > <storeUsage limit="100 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="50 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > <!-- > The transport connectors expose ActiveMQ over a given protocol > to > clients and other brokers. For more information, see: > > http://activemq.apache.org/configuring-transports.html > --> > <transportConnectors> > <!-- DOS protection, limit concurrent connections to 1000 and > frame size to 100MB --> > <transportConnector name="openwire" uri="tcp:// > 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize= > 104857600 > "/> > <transportConnector name="amqp" uri="amqp:// > 0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > "/> > <transportConnector name="stomp" uri="stomp:// > 0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize= > 104857600 > "/> > <transportConnector name="mqtt" uri="mqtt:// > 0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > "/> > <transportConnector name="ws" uri="ws:// > 0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize= > 104857600 > "/> > </transportConnectors> > > <!-- destroy the spring context on shutdown to stop jetty --> > <shutdownHooks> > <bean xmlns="http://www.springframework.org/schema/beans"; > class="org.apache.activemq.hooks.SpringContextHook" /> > </shutdownHooks> > > </broker> > > <!-- > Enable web consoles, REST and Ajax APIs and demos > The web consoles requires by default login, you can disable this in > the jetty.xml file > > Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details > --> > <import resource="jetty.xml"/> > > </beans> >
