I have a 3 node 5.10 cluster with replicated levelDB as persistent store.
<persistenceAdapter>
<replicatedLevelDB
directory="activemq-data"
replicas="3"
bind="tcp://0.0.0.0:0"
zkAddress="queue1:2181,queue2:2181,queue3:2181"
zkPath="/activemq/leveldb-stores"
hostname="queue3"
sync="quorum_disk"
/>
</persistenceAdapter>
I'm Stresstesting the queue with around 5000 persistent msg/s.
1 Producer
2 Consumers
I get some warning messages on the log:
On Master:
2014-12-16 16:25:28,375 | INFO | Slave has disconnected:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-3
2014-12-16 16:25:28,761 | WARN | Unexpected session error:
java.io.FileNotFoundException:
/queue/activemq/conf/activemq-data/00000027b8d097b0.log (No such file or
directory) | org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-1
2014-12-16 16:25:28,761 | INFO | Slave has disconnected:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2
2014-12-16 16:25:29,603 | INFO | Slave has connected:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2
2014-12-16 16:25:30,034 | INFO | Slave has connected:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-1
2014-12-16 16:25:31,360 | INFO | Slave has now caught up:
6a84579c-77fe-41eb-a728-1be472c12894 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-4
2014-12-16 16:25:32,546 | INFO | Slave has now caught up:
db6c9a23-7025-4384-b02f-dcda763113c3 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-2
On the slaves
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
2014-12-16 16:30:00,287 | WARN | No reader available for position:
27fd52c562, log_infos:
{171757839210=LogInfo(/queue/activemq/conf/activemq-data/00000027fd90a36a.log,171757839210,0)}
| org.apache.activemq.leveldb.RecordLog | Thread-16882
It looks to me that from time to time the connection to the master drops.
Should I worry about it ? what No reader available means, is there data
loss risk?
It looks to me that the level db implementation is kind of fragile...
Is somebody using it in production ?
Regards
Christian
