Alistair. I just tried the same with an embedded activemq instance and it
works great.
Maybe this has to do with broker settings? I can setup a tiny test project
for you if you want to try that?
On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
<alistair.yo...@uhi.ac.uk>wrote:
> I can now reproduce this every time. Sending 100 messages in very quick
> succession to a camel route. JMX:
>
> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
> <from uri="activemq:topic:matrix"/>
> <to uri="activemq:topic:edirectory"/>
> </route>
>
> EnqueueCount = 100
> DispatchCount = 100
> InFlightCount = 100
> DequeueCount = 1
>
> only 1 message ever gets through the route. That's 99 percent message loss,
> every time. This happens on two servers with the same config.
>
> This happens with the producer on the same machine as the broker as well as
> a remote broker.
>
> Is there anything I should be looking at to see where the other 99 messages
> are? JMX says they're in flight but all dead letter queues are empty and
> there are no errors anywhere.
>
> Alistair
>
> --
> mov eax,1
> mov ebx,0
> int 80h
>
>
>
>
> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>
> > getting somewhere.
> >
> > <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
> > <from uri="activemq:topic:matrix"/>
> > <transacted />
> > <process ref="matrixProcessor" />
> > <to uri="activemq:topic:edirectory"/>
> > </route>
> >
> > - producer clock is 2mins ahead of broker clock, timestampplugin enabled
> on broker
> > - no delay between messages at the producer = 62 out of 100 get through
> the route, consistently
> > - 3sec delay at producer, around 88 - 97 get through. That's as good as
> it gets
> > - 5sec delay at producer, 93 made it through the route
> > - remove the route and send direct to topic in activemq, no delay, no
> message loss, consistently
> >
> > so the messages are being lost in the route. No matter what the delay in
> sending messages, some are always lost and vanish. There are no errors,
> nothing in any dead letter queue. They simply vanish. They don't even make
> it as far as the <process ref="matrixProcessor" />.
> >
> > sending from a ruby producer that blats them out far quicker than the
> java producer is even worse. Only 1 - 3 ever get through the route. Removing
> the route and sending to the activemq topic instead, all messages get
> through no matter how fast they come.
> >
> > Alistair
> >
> > --
> > mov eax,1
> > mov ebx,0
> > int 80h
> >
> >
> >
> >
> > On 29 Sep 2011, at 18:10, Alistair Young wrote:
> >
> >> no connection pool. What was disturbing was the first message sent to
> the broker after a restart and clean out of the data dir, disappeared.
> There's a similar route on the broker that works fine. The only difference
> is the producer for the wonky route is on windows and is up to 1min ahead of
> the broker's clock. Would have thought the timestamplugin would take care of
> that though. Can see it changing the timestamp in the logs.
> >>
> >> Alistair
> >>
> >> --------------
> >> mov eax,1
> >> mov ebx,0
> >> int 80
> >>
> >> On 29 Sep 2011, at 17:31, Tim wrote:
> >>
> >>> Well with only 17 you definitely aren't hitting any prefetch limits or
> >>> anything like that.
> >>> Are you using a connection pool?
> >>>
> >>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
> alistair.yo...@uhi.ac.uk>wrote:
> >>>
> >>>> I think this way madness lies.
> >>>>
> >>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
> >>>> topicB enqueueCount = 12
> >>>>
> >>>> so 17 came in, 12 made it through, of the 5 that went missing it
> claims to
> >>>> have sent 3 to topicB but they never arrived and the last 2 just
> simply
> >>>> vanished completely.
> >>>>
> >>>> What on earth?
> >>>>
> >>>> Alistair
> >>>>
> >>>> --------------
> >>>> mov eax,1
> >>>> mov ebx,0
> >>>> int 80
> >>>>
> >>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
> >>>>
> >>>>> nup - cleaned out the data dir and restarted the broker. First
> message in
> >>>> vanished. Wasn't persisted. So something is fundamentally broken.
> >>>>>
> >>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
> >>>>> topicB is completely empty
> >>>>>
> >>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
> just
> >>>> vanished from the broker.
> >>>>>
> >>>>> Alistair
> >>>>>
> >>>>> --------------
> >>>>> mov eax,1
> >>>>> mov ebx,0
> >>>>> int 80
> >>>>>
> >>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
> >>>>>
> >>>>>> route goes from topicA -> topicB, transacted.
> >>>>>> topicA inflightCount = 96 and increases on each batch of incoming
> >>>> messages
> >>>>>> topicB dispatchCount = enqueueCount
> >>>>>>
> >>>>>> wondering if the missing messages are connected to topicA
> inflightCount.
> >>>> I noticed there are two consumers for topicB. The main consumer gets
> its
> >>>> messages fine. Wonder if the second consumer is a durable topic
> consumer and
> >>>> therefore activemq is persisting its messages but it hasn't connected
> in a
> >>>> very long time. Would that cause the topic to get too big? i.e.
> messages go
> >>>> into the topic until the limit is reached. Main consumer pulls
> messages off
> >>>> and messages are able to go onto topicB again. Before consumer pulls
> and
> >>>> after limit reached, messages can't get from topicA -> topicB, hence
> the
> >>>> topicA inflightCount not zero?
> >>>>>>
> >>>>>> Alistair
> >>>>>>
> >>>>>> --------------
> >>>>>> mov eax,1
> >>>>>> mov ebx,0
> >>>>>> int 80
> >>>>>>
> >>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
> >>>>>>
> >>>>>>> Sorry you might have tried this since I haven't been following this
> >>>> thread.
> >>>>>>> But can you check your jmx console.
> >>>>>>> In particular check 2 things.. the route to see if the number of
> >>>> exchanges
> >>>>>>> match what you think and how if any exchanges failed.
> >>>>>>> Also check the JMX console on activemq for the queue or topic in
> >>>> question
> >>>>>>> and see how many were enqueued vs dispatched.
> >>>>>>> Check your deadletter queue from there too
> >>>>>>>
> >>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
> >>>>>>> <alistair.yo...@uhi.ac.uk>wrote:
> >>>>>>>
> >>>>>>>> dunno - nothing works. Random messages are just vanishing once
> they
> >>>> reach
> >>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
> vanishing.
> >>>> I've
> >>>>>>>> removed <transacted /> and <process> but it doesn't help. The
> producer
> >>>> is a
> >>>>>>>> few secs behind the broker:
> >>>>>>>>
> >>>>>>>> sent : 11:25:26
> >>>>>>>> arrived : 11:24:57
> >>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
> >>>> presumably
> >>>>>>>> the timestampplugin doing this
> >>>>>>>> message vanishes
> >>>>>>>>
> >>>>>>>> but all messages display this clock behaviour and not all vanish.
> >>>>>>>>
> >>>>>>>> Alistair
> >>>>>>>>
> >>>>>>>> --------------
> >>>>>>>> mov eax,1
> >>>>>>>> mov ebx,0
> >>>>>>>> int 80
> >>>>>>>>
> >>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
> >>>>>>>>
> >>>>>>>>> just saw your info about transacted being before from - will
> change
> >>>> that
> >>>>>>>> and monitor again.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>>
> >>>>>>>>> Alistair
> >>>>>>>>>
> >>>>>>>>> --------------
> >>>>>>>>> mov eax,1
> >>>>>>>>> mov ebx,0
> >>>>>>>>> int 80
> >>>>>>>>>
> >>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
> >>>>>>>>>
> >>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
> and
> >>>>>>>> another single message vanished. tracer logged nothing. No errors,
> >>>> dead
> >>>>>>>> letter queue empty.
> >>>>>>>>>>
> >>>>>>>>>> One thing that happens is another machine polls the stats topic
> in
> >>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
> stats on
> >>>> the
> >>>>>>>> matrix topic, which is part of the transacted route.
> >>>>>>>>>>
> >>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
> >>>>>>>> Topic:ActiveMQ.Advisory.Connection
> >>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
> >>>> [null]:
> >>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> >>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
> >>>>>>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
> >>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
> >>>>>>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
> >>>>>>>> vm://matrixBroker#285920
> >>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
> >>>>>>>> {commandId = 1, responseRequired = true, connectionId =
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
> =
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
> =
> >>>> null,
> >>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
> false,
> >>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
> >>>>>>>>>>
> >>>>>>>>>> Alistair
> >>>>>>>>>>
> >>>>>>>>>> --------------
> >>>>>>>>>> mov eax,1
> >>>>>>>>>> mov ebx,0
> >>>>>>>>>> int 80
> >>>>>>>>>>
> >>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> <transacted/> Should be after <from>
> >>>>>>>>>>> it is after from - do you mean it should be before?
> >>>>>>>>>>> <route id="eDirSuccessBroadcast">
> >>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
> >>>>>>>>>>> <transacted />
> >>>>>>>>>>> <process ref="groupwiseProcessor" />
> >>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
> >>>>>>>>>>> </route>
> >>>>>>>>>>>
> >>>>>>>>>>> thanks for the dead letter tips, will apply them.
> >>>>>>>>>>>
> >>>>>>>>>>> Alistair
> >>>>>>>>>>>
> >>>>>>>>>>> --------------
> >>>>>>>>>>> mov eax,1
> >>>>>>>>>>> mov ebx,0
> >>>>>>>>>>> int 80
> >>>>>>>>>>>
> >>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi
> >>>>>>>>>>>>
> >>>>>>>>>>>> <transacted/> Should be after <from>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
> >>>>>>>>>>>> <alistair.yo...@uhi.ac.uk> wrote:
> >>>>>>>>>>>>>> Do you use message expiry?
> >>>>>>>>>>>>> no
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> timestamp plugin
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> using that
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> activemq 5.5.0
> >>>>>>>>>>>>> camel 2.8.0
> >>>>>>>>>>>>> spring 3.0.5
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
> is
> >>>>>>>> logging so hopefully I can see any errors.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <route id="matrix"
> >>>> errorHandlerRef="matrixDeadLetterErrorHandler">
> >>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
> >>>>>>>>>>>>> <process ref="matrixProcessor" />
> >>>>>>>>>>>>> <transacted />
> >>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
> >>>>>>>>>>>>> </route>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="jmsConnectionFactory"
> >>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
> >>>>>>>> depends-on="matrixBrokerID">
> >>>>>>>>>>>>> <property name="brokerURL"
> >>>>>>>> value="vm://matrixBroker?create=false"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="jmsTransactionManager"
> >>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
> >>>>>>>>>>>>> <property name="connectionFactory"
> >>>>>>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="activemq"
> >>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
> >>>>>>>>>>>>> <property name="connectionFactory"
> >>>>>>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>>>>> <property name="transacted" value="true"/>
> >>>>>>>>>>>>> <property name="transactionManager"
> >>>>>>>> ref="jmsTransactionManager"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
> >>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
> >>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
> >>>>>>>>>>>>> <property name="redeliveryPolicy"
> >>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
> >>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
> >>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
> >>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --------------
> >>>>>>>>>>>>> mov eax,1
> >>>>>>>>>>>>> mov ebx,0
> >>>>>>>>>>>>> int 80
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you use message expiry?
> >>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
> >>>>>>>> possible.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There is a timestamp plugin
> >>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And do you use queue or topic.
> >>>>>>>>>>>>>> What version of AMQ and Camel are you using?
> >>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
> >>>> context?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
> >>>> taar...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>> Hi
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
> your
> >>>>>>>> logger and
> >>>>>>>>>>>>>>> how you configured it.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
> logging
> >>>> FAQ[2]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]
> http://camel.apache.org/how-do-i-enable-debug-logging.html
> >>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Taariq
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
> >>>>>>>> alistair.yo...@uhi.ac.uk>wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
> >>>>>>>> camelTracer and
> >>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
> >>>> tried
> >>>>>>>> them all but
> >>>>>>>>>>>>>>>> no log appears.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> mov eax,1
> >>>>>>>>>>>>>>>> mov ebx,0
> >>>>>>>>>>>>>>>> int 80h
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
> your
> >>>>>>>> route.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> regards, Marco
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
> >>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
> losing
> >>>> 9
> >>>>>>>> out of 10
> >>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
> nothing.
> >>>>>>>> Verified the
> >>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Claus Ibsen
> >>>>>>>>>>>>>> -----------------
> >>>>>>>>>>>>>> FuseSource
> >>>>>>>>>>>>>> Email: cib...@fusesource.com
> >>>>>>>>>>>>>> Web: http://fusesource.com
> >>>>>>>>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Claus Ibsen
> >>>>>>>>>>>> -----------------
> >>>>>>>>>>>> FuseSource
> >>>>>>>>>>>> Email: cib...@fusesource.com
> >>>>>>>>>>>> Web: http://fusesource.com
> >>>>>>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >
>
>
