Usually a restart of amq resolves this issue. Today is different.
After two restarts I see a few dozen messages dequeued then its all stop.
Nothing in the logs other than DEBUG or INFO. Other servers appear fine.
I can use stomp to send a message on but the script that subscribes and
dequeues them fails to receive any one the freeze occurs.
Only thing I see different is PECL stomp 1.0.5 on this server and older
versions elsewhere. Are there know issues here?
Incidentally I checked with hawtio but failed to see anything wrong. Memory
and file handles in particular well within limits.
Thanks,
James
On 21 August 2014 09:49, James Green <james.mk.gr...@gmail.com> wrote:
> One of our spoke machines shows evidence that it is not delivering the
> messages to consumers as expected. Of the queues in question is named
> "DeliveryNotifications.Inbound" and I've switched on debug logging to
> capture what I can.
>
> During the following log segment you can see there are messages for
> delivery (console shows 139) and an automated PHP script comes along via
> STOMP to collect them, but receives none.
>
> 2014-08-21 09:40:31,839 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-56
> 2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
> Broker[drax] Scheduler
> 2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages done. | org.apache.activemq.broker.region.Queue |
> ActiveMQ Broker[drax] Scheduler
> 2014-08-21 09:40:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-56
> 2014-08-21 09:40:38,699 | DEBUG | drax adding consumer:
> ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
> queue://DeliveryNotifications.Inbound |
> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
> tcp:///127.0.0.1:38070@61612
> 2014-08-21 09:40:38,700 | DEBUG | queue://DeliveryNotifications.Inbound
> add sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
> tcp:///127.0.0.1:38070@61612
> 2014-08-21 09:40:38,700 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-56
> 2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
> Broker[drax] Scheduler
> 2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages done. | org.apache.activemq.broker.region.Queue |
> ActiveMQ Broker[drax] Scheduler
> 2014-08-21 09:41:05,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-53
> 2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
> Broker[drax] Scheduler
> 2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages done. | org.apache.activemq.broker.region.Queue |
> ActiveMQ Broker[drax] Scheduler
> 2014-08-21 09:41:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-53
> 2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
> Broker[drax] Scheduler
> 2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages done. | org.apache.activemq.broker.region.Queue |
> ActiveMQ Broker[drax] Scheduler
> 2014-08-21 09:42:05,906 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-57
> 2014-08-21 09:42:18,775 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=drax,destinationType=Queue,destinationName=DeliveryNotifications.Inbound,endpoint=Consumer,clientId=ID_drax-35978-1408600805652-4_9348,consumerId=ID_drax-35978-1408600805652-4_9348_-1_1
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
> tcp:///127.0.0.1:38070@61612
> 2014-08-21 09:42:18,775 | DEBUG | drax removing consumer:
> ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
> queue://DeliveryNotifications.Inbound |
> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
> tcp:///127.0.0.1:38070@61612
> 2014-08-21 09:42:18,775 | DEBUG | queue://DeliveryNotifications.Inbound
> remove sub: QueueSubscription:
> consumer=ID:drax-35978-1408600805652-4:9348:-1:1, destinations=1,
> dispatched=0, delivered=0, pending=0, lastDeliveredSeqId: 0, dequeues: 4,
> dispatched: 4, inflight: 0 | org.apache.activemq.broker.region.Queue |
> ActiveMQ Transport: tcp:///127.0.0.1:38070@61612
> 2014-08-21 09:42:18,776 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-57
> 2014-08-21 09:42:27,943 | DEBUG | drax adding consumer:
> ID:drax-35978-1408600805652-4:9368:-1:1 for destination:
> queue://DeliveryNotifications.Inbound |
> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
> tcp:///127.0.0.1:38178@61612
> 2014-08-21 09:42:27,944 | DEBUG | queue://DeliveryNotifications.Inbound
> add sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
> tcp:///127.0.0.1:38178@61612
> 2014-08-21 09:42:27,944 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-56
> 2014-08-21 09:42:35,906 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
> Broker[drax] Scheduler
> 2014-08-21 09:42:35,907 | DEBUG | queue://DeliveryNotifications.Inbound
> expiring messages done. | org.apache.activemq.broker.region.Queue |
> ActiveMQ Broker[drax] Scheduler
> 2014-08-21 09:42:35,907 | DEBUG | DeliveryNotifications.Inbound toPageIn:
> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
> Task-56
>
> Any ideas what's happening here? Or what we can do to debug further?
>
> Thanks,
>
> James
>
>
