Re: jconsole using activemq

[Tim Bain]

Does that apply for a hostname of localhost like the OP was using?

And can you give more details about exactly what switched, and in what
version of the JVM?  I'd have expected hostname resolution to be handled by
the OS and/or a service like DNS, and that therefore the behavior would be
the same for all versions of the JVM, so I'd like to better understand what
changed so I'll know what behavior to expect from different versions.

Tim

On Sep 23, 2016 1:04 PM, "Matt Pavlovich"  wrote:

> The sounds like a hostname to IP resolution. Newer JDK's require a lookup
> for the hostname to an IP for JMX. Many cloud and VM instances do not set
> this by default.
>
> For example, if you have your hostname vm-123456
>
> 1. Add entry to /etc/hosts
> 127.0.0.1   vm-123456
>
> 2. Restart the JVM
>
> On 9/22/16 7:53 PM, mathewvino wrote:
>
>> activemq : apache-activemq-5.14.0
>> jconsole using jdk1.7
>>
>> I have started the activemq which comes with the jetty server. Minor
>> change
>> to make the jmx enable is  just change > createConnector="true"/> default is false(activemq.xml) --- I am using the
>> latest version 5.14.0
>>
>> when i start  i am seeing the message
>> JMX consoles can connect to
>> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
>>
>> So I start Jconsole  use remote process and use
>> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
>>
>> with no password : it says secure/insecure  connection failed: not sure
>> where the problem is
>>
>> Any help is really appreciated
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nab
>> ble.com/jconsole-using-activemq-tp4716819.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>

Re: Active MQ Shared File System Master Slave with Elastic File System

[Tim Bain]

Thanks for following up on this to make sure that one got created.

On Sep 23, 2016 9:34 AM, "wcrowell"  wrote:

> Thank you!!!  I was going to create one if it had not been done already.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818p4716853.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: jconsole using activemq

[Matt Pavlovich]

The sounds like a hostname to IP resolution. Newer JDK's require a 
lookup for the hostname to an IP for JMX. Many cloud and VM instances do 
not set this by default.


For example, if you have your hostname vm-123456

1. Add entry to /etc/hosts
127.0.0.1   vm-123456

2. Restart the JVM

On 9/22/16 7:53 PM, mathewvino wrote:

activemq : apache-activemq-5.14.0
jconsole using jdk1.7

I have started the activemq which comes with the jetty server. Minor  change
to make the jmx enable is  just change  default is false(activemq.xml) --- I am using the
latest version 5.14.0

when i start  i am seeing the message
JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

So I start Jconsole  use remote process and use
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

with no password : it says secure/insecure  connection failed: not sure
where the problem is

Any help is really appreciated









--
View this message in context: 
http://activemq.2283324.n4.nabble.com/jconsole-using-activemq-tp4716819.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ ReplicatedLevelDB corruption

[mlange]

Concluded the test now; No good news though.

After sending the 100.000 messages (which got consumed by services that in
turn also produced new messages and so had a flow from one queue to another
and another etc... resulting in about 1.500.000 messages in the few hours
that this test ran) I restarted the master broker. Catching up went fine, no
errors

Shortly thereafter I restarted the new master (the slaves had caught up, I
had made sure of that message); Now the new master is giving multiple times
the message: "2016-09-23 19:10:37,644 | WARN  | Invalid log position: 0 |
org.apache.activemq.leveldb.LevelDBClient | Thread-118"

While both slaves are spewing out many log messages along the lines of this:
"2016-09-23 19:15:33,892 | WARN  | No reader available for position: 0,
log_infos:
{209718314=LogInfo(/data/activemq/broker1-db/0c800c2a.log,209718314,104859264),
2831214880=LogInfo(/data/activemq/broker1-db/a8c0e920.log,2831214880,104859800),
3250651975=LogInfo(/data/activemq/broker1-db/c1c10347.log,3250651975,104862133),
3355514108=LogInfo(/data/activemq/broker1-db/c80114fc.log,3355514108,104859203),
4823542070=LogInfo(/data/activemq/broker1-db/00011f816936.log,4823542070,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-3
"

Question arises if LevelDB is a good choice or is it too easily corrupted?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-ReplicatedLevelDB-corruption-tp4716831p4716869.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ Shared File System Master Slave with Elastic File System

[wcrowell]

Did a JIRA issue get created on this?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-Elastic-File-System-tp4715818p4716847.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ Shared File System Master Slave with Elastic File System

[wcrowell]

Thank you!!!  I was going to create one if it had not been done already.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-Elastic-File-System-tp4715818p4716853.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: activemqmessageconsumer duplicate dispatch on connection

[MariscaJane]

  









  

  

  



i use your suggestion, but i still see a pending message :( still got the
same error, any other suggestion? thank you sir :D



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/activemqmessageconsumer-duplicate-dispatch-on-connection-tp4716779p4716821.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ Shared File System Master Slave with Elastic File System

[ishmeister]

Done: https://issues.apache.org/jira/browse/AMQ-6441

If anyone wants to update the ticket, feel free.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-Elastic-File-System-tp4715818p4716851.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ ReplicatedLevelDB corruption

[mlange]

possibly of note is that I had the sync option on the default (quorum_mem).

I have moved the data directories for each broker as an ".org" (for
safekeeping) and will try and see if quorum_disk is the way to prevent this
situation from happening. It will take a little while though to get definite
results.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-ReplicatedLevelDB-corruption-tp4716831p4716850.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ Shared File System Master Slave with Elastic File System

[ishmeister]

No not yet, but I will do so now.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-Elastic-File-System-tp4715818p4716848.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ ReplicatedLevelDB corruption

[mlange]

Yes, the three brokers are on separate (virtual) machines that write to their
own disk.
It's hard to determine (at this point) whether the original master got
corrupted, as it (at least) got corrupted after starting said broker again
(or was already corrupted to begin with)





--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-ReplicatedLevelDB-corruption-tp4716831p4716842.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ ReplicatedLevelDB corruption

[Tim Bain]

Just to confirm: your three brokers are writing their LevelDB files to
independent, separate disk locations, giving you three separate sets of
LevelDB files.  Right?

Is the original master's set of data files corrupted?  Or is it just the
two slaves for whom this happened?

On Sep 23, 2016 5:44 AM, "mlange"  wrote:

> Recently I installed Apache ActiveMQ in a few different ways. One of those
> is
> using ReplicatedLevelDB for a master/slave/slave setup.
>
> Yesterday I did a bit of loadtesting: sending 100.000 messages with 100
> threads producing the messages (used jmeter for that) (so each thread
> produced 1000 messages); I had another process moving the messages from one
> broker to another and back again (the queues had the same names across each
> broker, so that was easy moving) and then went about processing the
> messages
> which caused the messages to flow across various queues.
>
> All seemed fine, everything looked okay... until I stopped the active
> broker. (this is hours after the last message was consumed and procsesed):
>
> Then I notice a few bouncing brokers, one comes up but crashes on an
> EOFException; a bit later the other broker does the same.
>
> In the log I see many messages like this:
>
> [quote]
> 2016-09-23 13:38:52,950 | WARN  | No reader available for position: 0,
> log_infos:
> {11534500540=LogInfo(/data/activemq/broker1-db/0002af8282bc.log,
> 11534500540,104858130),
> 12163654223=LogInfo(/data/activemq/broker1-db/0002d502a24f.log,
> 12163654223,104858162),
> 12897666570=LogInfo(/data/activemq/broker1-db/000300c2c60a.log,
> 12897666570,104859912),
> 13002526482=LogInfo(/data/activemq/broker1-db/00030702cf12.log,
> 13002526482,104859038),
> 18455209795=LogInfo(/data/activemq/broker1-db/00044c042743.log,
> 18455209795,104859837),
> 22020442500=LogInfo(/data/activemq/broker1-db/000520854984.log,
> 22020442500,104859288),
> 23173898306=LogInfo(/data/activemq/broker1-db/00056545a042.log,
> 23173898306,104860684),
> 24641928389=LogInfo(/data/activemq/broker1-db/0005bcc5fcc5.log,
> 24641928389,0)}
> | org.apache.activemq.leveldb.RecordLog | Thread-1039
> [/quote]
>
> Then I see messages like this:
> [quote]
> 2016-09-23 13:38:46,324 | WARN  | Invalid log position: 11409726550 |
> org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
> BrokerService[broker1]
> Task-3
> [/quote]
>
> After that, the broker starts and logs a few messages like this:
> [quote]
> 2016-09-23 13:40:49,041 | WARN  | Invalid log position: 0 |
> org.apache.activemq.leveldb.LevelDBClient | Thread-1040
> [/quote]
>
> and then we get exception:
> [quote]
> 2016-09-23 13:41:09,748 | INFO  | Stopping BrokerService[broker1] due to
> exception, java.io.EOFException: File
> '/data/activemq/broker1-db/00030702cf12.log' offset: 110647192 |
> org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> handler.
> java.io.EOFException: File '/data/activemq/broker1-db/
> 00030702cf12.log'
> offset: 110647192
> [/quote]
>
> This is a rince and repeat situation; both living brokers are now
> alternating this sequence.
>
> It seems like the load I generated caused corruption on the database; but
> this should not be possible... What information can I provide to see how
> this situation can be avoided?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-ReplicatedLevelDB-corruption-tp4716831.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: jconsole using activemq

[Tim Bain]

If I remember correctly, you also can skip all the authentication checks by
starting JConsole (or JVisualVM) on the same machine as your broker,
running as the same user as the broker, and then selecting it from the
Local Processes list.  Sometimes that's an option, sometime it's not, but
it was in this case (you can see "activemq.jar start" in the screenshot).

Also, it's been a while since I've connected to a broker whose password
hadn't been changed, but I think the default credentials are admin/admin,
rather than blank.

Tim

On Sep 23, 2016 5:50 AM, "Steve Hill"  wrote:

> We ran into many problems getting JMX to work properly in our test
> environment.  We are running two instances on each server in our test
> environment.  We found the following changes got it to work and kept them
> using independent mbean servers.
>
> In the bin/env file add the following (We have it on port 11009)
>
> ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START
> -Dcom.sun.management.jmxremote.port=11009 
> -Dcom.sun.management.jmxremote.rmi.port=11009
> "
> ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START
> -Dcom.sun.management.jmxremote.ssl=false"
> ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START
> -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.authenticate=false
> -Djava.rmi.server.hostname="
>
> In the activemq.xml file, add useJmx="true" to the  tag and
> add/update the following element
>
> 
>  createConnector="false"/>
> 
>
> We do get the warning about secure/insecure however you click ok and it
> connects.
>
> Also check to make sure the port is open in your firewall, the following
> will open the port in iptables.
>
> -A INPUT -p tcp -m tcp --dport 11009 -j ACCEPT
>
> From jconsole you should be able to connect now using : in
> the remote connection box.
>
> Good luck!
> Steve.
>
> On 9/22/16 7:53 PM, mathewvino wrote:
>
>> activemq : apache-activemq-5.14.0
>> jconsole using jdk1.7
>>
>> I have started the activemq which comes with the jetty server. Minor
>> change
>> to make the jmx enable is  just change > createConnector="true"/> default is false(activemq.xml) --- I am using the
>> latest version 5.14.0
>>
>> when i start  i am seeing the message
>> JMX consoles can connect to
>> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
>>
>> So I start Jconsole  use remote process and use
>> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
>>
>> with no password : it says secure/insecure  connection failed: not sure
>> where the problem is
>>
>> Any help is really appreciated
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nab
>> ble.com/jconsole-using-activemq-tp4716819.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>

Re: jconsole using activemq

[Steve Hill]

We ran into many problems getting JMX to work properly in our test 
environment.  We are running two instances on each server in our test 
environment.  We found the following changes got it to work and kept 
them using independent mbean servers.


In the bin/env file add the following (We have it on port 11009)

ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START 
-Dcom.sun.management.jmxremote.port=11009 
-Dcom.sun.management.jmxremote.rmi.port=11009 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START 
-Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START 
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Djava.rmi.server.hostname="


In the activemq.xml file, add useJmx="true" to the  tag and 
add/update the following element



createConnector="false"/>



We do get the warning about secure/insecure however you click ok and it 
connects.


Also check to make sure the port is open in your firewall, the following 
will open the port in iptables.


-A INPUT -p tcp -m tcp --dport 11009 -j ACCEPT

From jconsole you should be able to connect now using : 
in the remote connection box.


Good luck!
Steve.

On 9/22/16 7:53 PM, mathewvino wrote:

activemq : apache-activemq-5.14.0
jconsole using jdk1.7

I have started the activemq which comes with the jetty server. Minor  change
to make the jmx enable is  just change  default is false(activemq.xml) --- I am using the
latest version 5.14.0

when i start  i am seeing the message
JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

So I start Jconsole  use remote process and use
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

with no password : it says secure/insecure  connection failed: not sure
where the problem is

Any help is really appreciated









--
View this message in context: 
http://activemq.2283324.n4.nabble.com/jconsole-using-activemq-tp4716819.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

ActiveMQ ReplicatedLevelDB corruption

[mlange]

Recently I installed Apache ActiveMQ in a few different ways. One of those is
using ReplicatedLevelDB for a master/slave/slave setup. 

Yesterday I did a bit of loadtesting: sending 100.000 messages with 100
threads producing the messages (used jmeter for that) (so each thread
produced 1000 messages); I had another process moving the messages from one
broker to another and back again (the queues had the same names across each
broker, so that was easy moving) and then went about processing the messages
which caused the messages to flow across various queues.

All seemed fine, everything looked okay... until I stopped the active
broker. (this is hours after the last message was consumed and procsesed):

Then I notice a few bouncing brokers, one comes up but crashes on an
EOFException; a bit later the other broker does the same. 

In the log I see many messages like this:

[quote]
2016-09-23 13:38:52,950 | WARN  | No reader available for position: 0,
log_infos:
{11534500540=LogInfo(/data/activemq/broker1-db/0002af8282bc.log,11534500540,104858130),
12163654223=LogInfo(/data/activemq/broker1-db/0002d502a24f.log,12163654223,104858162),
12897666570=LogInfo(/data/activemq/broker1-db/000300c2c60a.log,12897666570,104859912),
13002526482=LogInfo(/data/activemq/broker1-db/00030702cf12.log,13002526482,104859038),
18455209795=LogInfo(/data/activemq/broker1-db/00044c042743.log,18455209795,104859837),
22020442500=LogInfo(/data/activemq/broker1-db/000520854984.log,22020442500,104859288),
23173898306=LogInfo(/data/activemq/broker1-db/00056545a042.log,23173898306,104860684),
24641928389=LogInfo(/data/activemq/broker1-db/0005bcc5fcc5.log,24641928389,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-1039
[/quote]

Then I see messages like this:
[quote]
2016-09-23 13:38:46,324 | WARN  | Invalid log position: 11409726550 |
org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[broker1]
Task-3
[/quote]

After that, the broker starts and logs a few messages like this:
[quote]
2016-09-23 13:40:49,041 | WARN  | Invalid log position: 0 |
org.apache.activemq.leveldb.LevelDBClient | Thread-1040
[/quote]

and then we get exception:
[quote]
2016-09-23 13:41:09,748 | INFO  | Stopping BrokerService[broker1] due to
exception, java.io.EOFException: File
'/data/activemq/broker1-db/00030702cf12.log' offset: 110647192 |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.EOFException: File '/data/activemq/broker1-db/00030702cf12.log'
offset: 110647192
[/quote]

This is a rince and repeat situation; both living brokers are now
alternating this sequence.

It seems like the load I generated caused corruption on the database; but
this should not be possible... What information can I provide to see how
this situation can be avoided?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-ReplicatedLevelDB-corruption-tp4716831.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[CVE-2016-4978] Apache ActiveMQ Artemis: Deserialization of untrusted input vunerability

[Martyn Taylor]

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: Apache Artemis 1.0.0, 1.1.0, 1.2.0, 1.3.0

A class implementing the Serializable interface is free to implement
the “readObject(java.io.ObjectInputStream
in)” method however it chooses. This readObject method is used during the
deserialization process, when constructing a java object from a serialized
byte stream. It is possible to implement the method in such a way that can
result in java code being executed during the deserialization of an object
of this class (gadget class).

The JMS specification outlines a getObject() method on the
javax.jms.ObjectMessage
class. The Apache Artemis implementation of this method allows
deserialization of objects, from untrusted input. There are several places
where Apache Artemis uses this getObject() method. In the JMS Core client,
the Artemis broker and the Artemis REST component. These Artemis components
may therefore be vulnerable to a remote code execution attack. Successful
exploitations of this vulnerability rely on these "gadget classes"  being
present on the Artemis classpath and the sender of the untrusted input
being authenticated and authorized to send messages to the Artemis broker.

The code execution exploit may happen under the following circumstances:

· In the JMS client when consuming an object message.

· In the REST module when a REST client requests to consume a message that
was originally sent as an object message (cross protocol).

· In the Artemis management layer, when a client sends an object message to
a management address.

· On the broker when an AMQP client consumes a message that was originally
sent as an object message (cross protocol).

For this exploit to occur the sender of the compromised message needs to be
authenticated and authorized in order to send the message to the Artemis
broker and affected classes (gadget classes) present on the Artemis class
path.

Mitigation:
To secure the Apache Artemis broker and management layer:
** Upgrade to 1.4.0.

For the Apache Artemis REST module and Apache Artemis JMS client.
** Upgrade to Apache Artemis 1.4.0
** Configure the appropriate deserialization white/black lists as outlined
in the Artemis documentation.

Credit: This issue was discovered by Matthias Kaiser of Code White (
www.code-white.com)

Re: KAHADB clean up old log files

[lichtin]

https://issues.apache.org/jira/browse/AMQ-6440


Tim Bain wrote
> I was going to write a JIRA enhancement request for that when I had some
> free time, but if you have time to write it before I get to it, that would
> be fine.  Please put the link here if you do.
> 
> On Sep 22, 2016 9:00 AM, "lichtin" 

> lichtin@

>  wrote:
> 
>> Changing trace levels and using the debugger is not always an option in
>> production.
>> I'm wondering why we could not revive the "KahaDBJournalReader" tool
>> from https://issues.jboss.org/browse/MB-756 and ship it with ActiveMQ.
>>
>>
>> Tim Bain wrote
>> > Alternatively, you all could run the KahaDB file analysis tool (
>> > http://activemq.apache.org/why-do-kahadb-log-files-
>> remain-after-cleanup.html)
>> > with a debugger attached to try to see what destination has the
>> message(s)
>> > that are keeping 980 and 1438 alive.  (You say that 1328 shouldn't be
>> > there, but the log shows that it's there because it contains acks for
>> > messages in 980, so it should be there, at least until 980 is deleted.)
>> >
>> > Tim
>> >
>> > On Sep 22, 2016 6:39 AM, "Steve Hill" <
>>
>> > shill@.co
>>
>> > > wrote:
>> >
>> > After the comment about the TOPIC I decided to take a quick look at
>> that
>> > in
>> > the web interface (just to make sure nothing had gone awry)  In deed
>> one
>> > of
>> > our "queues" had been created as a topic, not sure why as we use the
>> same
>> > code to connect to the queues however once removed most of the data
>> files
>> > vanished.
>> >
>> > Under the topics there are several activemq.advisory.xyz topics. Could
>> > these be the source of the remaining kahadb log files that still
>> remain?
>> >
>> > I am now down to three data files that should not be there. (db-980,
>> > db-1329, and db-1438)
>> >
>> >
>> > -rw-r--r--. 1 activemq activemq 33554432 Aug  8 14:00 db-980.log
>> > -rw-r--r--. 1 activemq activemq   527645 Sep 19 19:11 db-1329.log
>> > drwxr-xr-x. 5 activemq activemq 4096 Sep 20 14:57 ..
>> > -rw-r--r--. 1 activemq activemq8 Sep 21 13:17 lock
>> > -rw-r--r--. 1 activemq activemq 33553518 Sep 21 21:52 db-1438.log
>> > drwxr-xr-x. 2 activemq activemq12288 Sep 22 08:20 .
>> > -rw-r--r--. 1 activemq activemq  3287296 Sep 22 08:34 db.redo
>> > -rw-r--r--. 1 activemq activemq 12357632 Sep 22 08:34 db.data
>> > -rw-r--r--. 1 activemq activemq 33554432 Sep 22 08:34 db-1456.log
>> >
>> > From kahadb logs
>> >
>> >  2016-09-22 08:31:07,503 [eckpoint Worker] TRACE MessageDatabase
>> > - gc candidates: [1329]
>> >  2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
>> > - ackMessageFileMap: {1329=[980], 1438=[], 1456=[]}
>> >  2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
>> > - not removing data file: 1329 as contained ack(s) refer to
>> referenced
>> > file: [980]
>> >  2016-09-22 08:31:07,505 [eckpoint Worker] TRACE MessageDatabase
>> > - Not yet time to check for compaction: 7 of 10 cycles
>> >  2016-09-22 08:31:07,505 [eckpoint Worker] DEBUG MessageDatabase
>> > - Checkpoint done.
>> >
>> > Thanks!
>> > Steve.
>> >
>> >
>> > On 9/22/16 8:14 AM, Steve Hill wrote:
>> >
>> >> Chris/Tim:
>> >>
>> >> Thanks for the ideas.  Unfortunately neither are the case in our
>> >> situation.  The web client shows 0 messages in any of our queues and
>> we
>> >> are
>> >> not using topics.  We did set up the remote debug port and downloaded
>> the
>> >> source code if there is anything useful to further troubleshoot that
>> >> could
>> >> be gleaned from there?
>> >>
>> >> Thanks!
>> >> Steve.
>> >>
>> >>
>> >> The most likely place for you to have old messages laying around is in
>> >> the
>> >> DLQ.  The next most likely place is on a topic for an offline (and
>> never
>> >> coming back) durable subscriber.
>> >>
>> >> Tim
>> >>
>> >> On Sep 20, 2016 2:02 PM, "Christopher Shannon" <
>> >>
>>
>> > christopher.l.shannon@
>>
>> >> wrote:
>> >>
>> >> Do you have old messages laying around?  KahaDB can't clean up old
>> >> journal
>> >> files if there are messages scattered throughout the logs.  Even a
>> single
>> >> unacked message means that a journal file (and the future files) can't
>> be
>> >> cleaned up. This situation tends to happen either when there are old
>> >> messages not being acked quick enough or there are different
>> destinations
>> >> with different speeds.  Ie if one queue has a very slow message rate
>> and
>> >> another queue has a high message rate.
>> >>
>> >> Assuming there isn't some sort of bug with the GC process then one way
>> to
>> >> help lower the usage would be to split up destinations into different
>> >> stores using multiKahaDB.  Take a look at Gary's blog post on this:
>> >> http://blog.garytully.com/2011/11/activemq-multiple-
>> kahadb-instances.html
>> >>
>> >>
>> >> On Tue, Sep 20, 2016 at 1:04 PM, Hill, Steve <
>>
>> > SHill@
>>
>> > > wrote:
>> >>
>> >> Hi fellow ActiveMQ users!
>> >>>
>> >>> I have searched all over the internet and can not find a