Re: ActiveMQ with KahaDB as persistent store becomes very slow (almost unresponsive) after creating large no (25000+) of Topics

[Tim Bain]

How are you able to tell that a single call to updateIndex() is taking 3.5
minutes?  It's not possible to determine that from thread dumps alone, so
what other tool are you using to prove that assertion?

A profiler such as JVisualVM would help you determine where precisely the
broker is spending its time, though of course it may not tell you exactly
why.  Stepping through the code with a debugger may help you figure out the
why if it's not obvious from the profiler output.  Luckily this is an open
source project and you can download the source to make that possible.

Tim
On Apr 8, 2016 11:58 AM, "Shobhana"  wrote:

> Hi Tim,
>
> I said indexing was the point of contention after seeing that Thread
> "ActiveMQ NIO Worker 169" was still working on
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex even after >
> 3.5 minutes.
>
> These are full thread dumps. I guess the lock (read lock) is held by
> threads
> "ActiveMQ NIO Worker 169" and "ActiveMQ NIO Worker 171". Since the read
> lock
> is already held by other threads, the thread "ActiveMQ Broker[localhost]
> Scheduler" is waiting to acquire write lock. Since there is already a
> thread
> waiting to acquire write lock, other threads which are waiting to acquire
> read lock are still waiting.
>
> What could be the reason for updateIndex not completing even after 3.5
> minutes?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-with-KahaDB-as-persistent-store-becomes-very-slow-almost-unresponsive-after-creating-large-s-tp4709985p4710533.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

RE: MasterSlave config ActiveMQ

[Tim Bain]

Is hdp132 the master or the slave?  And what URI does your client use to
connect?

Do you see that message once, or six times?
On Apr 8, 2016 12:32 PM, "Natarajan, Rajeswari" 
wrote:

> Thanks Tim for the detailed email.  When I do failover I see below
> exception in our app logs.
>
> org.apache.activemq.transport.failover.FailoverTransport
> Transport (tcp://hdp132.lab1.ariba.com:61616) failed, attempting to
> automatically reconnect java.io.EOFException: null
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:267)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
>
> The message redelivery attempt is default  in the activemq conf, so it is
> 6 times.
>
>
> I just send 1 message to the queue ,receive it and don't ack it ,then do
> failover . There is a thread which listens for receiving messages in the
> queue ,I don't see any exception in there .  So not sure  from where the
> transport exception is thrown.
> In the produce send  , receive and ack code the JMS
> Is there a way to track failover happened.  May be some sort of listener
> ,then the session can be recovered.
>
> Regards,
> Rajeswari
>
> -Original Message-
> From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain
> Sent: Thursday, April 07, 2016 8:53 PM
> To: ActiveMQ Users 
> Subject: Re: MasterSlave config ActiveMQ
>
> I apologize, I misunderstood what you meant by "I made sure the messages
> will not be acknowledged."
>
> When either the broker or a consumer goes down while a consumer is actively
> consuming a message, the broker will consider that a filed delivery and try
> to redeliver the message.  The maximumRedeliveries property of the
> Redelivery Policy (http://activemq.apache.org/redelivery-policy.html)
> controls how many times the message will be redelivered before being put
> into the DLQ.  The default value is 6; is this the value you're using, or
> have you explicitly set a different value?  With the default value of 6,
> your consumer should fail over to the new master, the message should be
> redelivered (because this is redelivery attempt #1 and that's less than 6),
> and your consumer should start processing the message for the second time.
> As long as that doesn't throw any exceptions or fail in any other way (have
> you confirmed this?), the second redelivery attempt should eventually
> result in a successful processing of the message.
>
> However, it sounds like that's not happening for you, and
>
> http://stackoverflow.com/questions/8576821/cant-get-activemq-to-resend-my-messages#comment35122357_8576821
> seems to indicate that when using INDIVIDUAL_ACKNOWLEDGE mode, if you don't
> close the Connection or recover() the Session, the messages will be
> considered duplicates and ignored by the consumer.  I haven't worked with
> INDIVIDUAL_ACKNOWLEDGE mode myself; can anyone else on the list shed more
> light on whether that comment is accurate?
>
> If so, it would sound like clients who use INDIVIDUAL_ACKNOWLEDGE mode
> shouldn't use the failover transport, because they need the connection to
> die so they can recreate it.  If that were true, I'd expect we'd have a
> warning somewhere on the wiki saying that those two features aren't
> compatible with one another, but I haven't seen one in my searching this
> evening...
>
> Tim
>
> On Thu, Apr 7, 2016 at 12:02 PM, Natarajan, Rajeswari <
> rajeswari.natara...@sap.com> wrote:
>
> > I am trying to understand, sorry if I sound , I don't believe you or
> > skeptic. I have an application with the broker url defined as below
> (with
> > shared file system)
> >
> > failover:(tcp://:61616,tcp://:61616.
> >
> > When I failover I see the web console and see one consumer on the fail
> > over host ,as the application is having a consumer and got a message ,but
> > did not acknowledge (delaying it on purpose for a use case) during
> > failover.. In our use case, there will be messagse sent to the queue and
> we
> > do some processing ,but not acknowledging the messages ,unless we get a
> > condition satisfied ,which might take some time. So when a failover
> happens
> > within that time ,all such messages will be moved to DLQ?
> > Is there any way to  have them in the same state as before in  the fail
> > over host.
> >
> > Thank you,
> > Rajeswari
> >
> > -Original Message-
> > From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain
> > Sent: Thursday, April 07, 2016 5:41 AM
> > To: ActiveMQ Users 
> > Subject: RE: MasterSlave config ActiveMQ
> >
> > Quoting from what I originally wrote you: "or if there's a consumer
> > 

Re: Pooled connection factory does not work

[Tim Bain]

Hopefully resolving the errors you see when setting the prefetch size will
fix your problem.  If not, let us know what the behavior is with a prefetch
value of 1, 2, and 10, once that works.
On Apr 8, 2016 9:28 AM, "Michele"  wrote:

> Hi Tim,
>
> exactly... According to http://camel.apache.org/jms.html section
> Concurrent
> Consumin, we can configure concurrentConsumers option in one of the
> following ways:
> On the JmsComponent,
> On the endpoint URI or,
> By invoking setConcurrentConsumers() directly on the JmsEndpoint.
>
> However, I tried to add concurrentConsumers=10 as option in endpoint uri,
> but always only one works.
>
> I also tried with prefetchSize = 1 but with errors... I'm investigating.
>
> Any suggestions?
>
> Thanks in advance.
>
> Michele
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Pooled-connection-factory-does-not-work-tp4710421p4710525.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Performance issue in FilePendingMessageCursor.isEmpty()

[Tim Bain]

Looks like your attachment from YourKit was stripped off.

But your suggestion seems like a reasonable improvement; submit a JIRA
enhancement request.  Even better, a patch or a pull request!
On Apr 9, 2016 11:51 AM, "Martin Lichtin"  wrote:

With a test sending 5000 messages to a queue, I see heavy CPU use on the
broker side (v 5.12.2).
Using Yourkit it shows that the hot spot methods are



FilePendingMessageCursor.isEmpty() iterates over in-memory messages and
therefore (as it is a prioritized queue) uses
PrioritizedPendingListIterator which uses OrderedPendingList.getAsList()
which turns out to be an expensive method as it converts the self-managed
linked list to a Java ArrayList.

I wonder if anyone else has seen this issue?

Perhaps PrioritizedPendingListIterator could be improved to walk the
priority lists via OrderedPendingList iterators, as these are implemented
efficiently.

- Martin

Performance issue in FilePendingMessageCursor.isEmpty()

[Martin Lichtin]

With a test sending 5000 messages to a queue, I see heavy CPU use on the broker 
side (v 5.12.2).
Using Yourkit it shows that the hot spot methods are



FilePendingMessageCursor.isEmpty() iterates over in-memory messages and 
therefore (as it is a prioritized queue) uses PrioritizedPendingListIterator 
which uses OrderedPendingList.getAsList() which turns out to be an expensive 
method as it converts the self-managed linked list to a Java ArrayList.

I wonder if anyone else has seen this issue?

Perhaps PrioritizedPendingListIterator could be improved to walk the priority 
lists via OrderedPendingList iterators, as these are implemented efficiently.

- Martin