Re: ActiveMQ with KahaDB as persistent store becomes very slow (almost unresponsive) after creating large no (25000+) of Topics

[Shobhana]

Hi Tim & Christopher,

I tried with 5.13.2 version but as you suspected, it did not solve my
problem.

We don't have any wildcard subscriptions. Most of the Topics have a maximum
of 8 subscriptions (Ranges between 2 and 8) and a few topics (~25-30 so far)
have more than 8 (this is not fixed, it depends on no of users interested in
these specific topics; the max I have seen is 40).

Btw, I just realized that I have set a very low value for destination
inactivity (30 secs) and hence many destinations are getting removed very
early. Later when there is any message published to the same destination, it
would result in destination getting created again. I will correct this by
increasing this time out to appropriate values based on each destination
(varies from 1 hour to 1 day)

Today after upgrading to 5.13.2 version in my test env, I tried with
different configurations to see if there is any improvement. In particular,
I disabled journal disk sync (since many threads were waiting at KahaDB
level operations) and also disabled metadata update. With these changes, the
contention moved to a different level (KahaDB update index .. see attached
thread dumps)

ThreadDump1.txt
  
ThreadDump2.txt
  

I will test again by increasing the index cache size (current value is set
to the default of 1) to 10 and see if it makes any improvement.

Also histo reports showed a huge number (1393177) of
org.apache.activemq.management.CountStatisticImpl instances and 1951637
instances of java.util.concurrent.locks.ReentrantLock$NonfairSync. See
attached histo for complete report.

histo.txt   

What are these org.apache.activemq.management.CountStatisticImpl instances?
Is there any way to avoid them?

Thanks,
Shobhana






--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-with-KahaDB-as-persistent-store-becomes-very-slow-almost-unresponsive-after-creating-large-s-tp4709985p4710055.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ with KahaDB as persistent store becomes very slow (almost unresponsive) after creating large no (25000+) of Topics

[Christopher Shannon]

Also, the bug fix should at least help the performance on start up but may
not help with the producer timing out problem.  The main issue is that the
message has to be dispatched to all subscriptions before the ACK can be
sent back to the producer, so if you have a lot of subscriptions (such as a
wildcard subscription on a lot of topics) this could cause ACKs to be slow.

On Tue, Mar 29, 2016 at 8:53 PM, Shobhana  wrote:

> Thank you Christopher for your suggestion. I'll check this with 5.13.2
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-with-KahaDB-as-persistent-store-becomes-very-slow-almost-unresponsive-after-creating-large-s-tp4709985p4710043.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: activemq-all jar tight coupling with log4j

[Christopher Shannon]

I didn't realize that config was added recently (looks like 5.12.0).  It
also looks like the dependency jcl-over-slf4j was added as part of that
commit as well so not sure if that is also causing you problems.

I'm not entirely sure why it was just added (probably just to make it easy
to provide for a default logging configuration) but I can ping Dejan, who
made the change, and ask if it could be provided separately if that is the
only thing causing your issue and since it was a recent change in behavior.

Either way, I would still recommend not relying on that jar in the future
so that you have more control over your dependencies.  The documentation
should be updated to make it more obvious the limitations of the
activemq-all jar.  It's mentioned in some places but it isn't as clear as
it could be so I can work on updating that.

On Tue, Mar 29, 2016 at 8:19 PM, Paul Gale  wrote:

> The file log4j.properties in activemq-all was added on 2/12/15 (not sure
> what release of ActiveMQ that correlates to). That's the kicker. When we
> upgraded recently from 5.11.1 to 5.13.1 is when the problem became
> apparent. You can see why. It's a non-trivial exercise to coordinate
> multiple teams to switch their client-side jar to be something different.
> The schedules of those teams are out of our hands. If it was trivial this
> would be moot. Activemq-all was an uber jar at 5.11.1 and this was not an
> issue. I didn't think that requesting a log4j.properties file be removed
> would such a big ask in order to indulge the wider user community, compared
> to the narrower needs of the testing community.  Given the great pain
> that's caused by making such a switch the consequences of this change could
> have been better publicized with the appropriate warning in the release
> notes stating THIS WILL BREAK YOU IN PROD - similar to how the recent
> security serialization bug was handled. You guys request feedback - we
> provided it. Ignore at will.
>
>
> On Tue, Mar 29, 2016 at 4:31 PM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Nothing has been changed so I don't know what you are referring to. That
> > log4j dependency has been included in that jar for a while, since
> > AMQ-3715.  The main point of that jar file is it is an easy way to spin
> up
> > a broker for testing and it is supposed to contain concrete dependencies
> of
> > things so that it runs out of the box using a single jar file.
> >
> > Yes it is a legitimate way to connect to the broker but it doesn't mean
> > it's the best way.  You are certainly welcome to use activemq-all however
> > you want but if you do you will get the default dependencies of
> everything
> > needed to run including the default versions of spring, camel, log4j, etc
> > that are included.  Since it is an uber jar nothing can be customized and
> > that is the idea.  I don't see any reason to change this as people have
> > been expecting this behavior for  the past 4 years.
> >
> > In my opinion nothing needs to be done here because there is already a
> > solution.  You can use individual jars, such as activemq-client, and with
> > transitive dependencies you can customize what you want in the classpath
> > very easily if you don't like it.  For example,  activemq-client jar
> > doesn't have a dependency on log4j (I just double checked) and only slf4j
> > so you are free to easily plug in whatever implementation of logging you
> > feel like.
> >
> >
> >
> > On Tue, Mar 29, 2016 at 3:51 PM, Paul Gale 
> wrote:
> >
> > > Per the online documentation, using activemq-all is a legitimate way to
> > > connect to a broker. For those that were happily using it up until now,
> > > whether others consider it 'overkill' or not, they are now screwed by
> > this
> > > change. Regardless, stating that one can 'fine tune' dependencies by
> > > switching away from activemq-all is a poor justification for including
> a
> > > concrete dependency on log4j (or anything else one fancies) within
> > > activemq-all. Doing so prevents those users from using a logging
> > framework
> > > of their choice.
> > >
> > > Why not just remove the log4j dependency from activemq-all and keep the
> > > entire codebase logging agnostic, hence the purpose of slf4j? Why is a
> > > dependency on log4j needed outside of a testing context?
> > >
> > > Thanks,
> > > Paul
> > >
> > > On Tue, Mar 29, 2016 at 3:28 PM, Timothy Bish 
> > wrote:
> > >
> > > > On 03/29/2016 02:43 PM, stak wrote:
> > > >
> > > >> So are you saying that activemq-all.jar is not a production grade
> jar
> > to
> > > >> be
> > > >> used as a client jar when they want to connect to an external
> activemq
> > > as
> > > >> a
> > > >> client?.  Is the purpose of that jar really only for a quick unit
> test
> > > of
> > > >> sorts?
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> View this message in context:
> > > >>
> > >
> >
> 

Re: ActiveMQ with KahaDB as persistent store becomes very slow (almost unresponsive) after creating large no (25000+) of Topics

[Tim Bain]

AbstractRegion.lookup()

has two copies of the following code:

455destinationsLock.readLock().lock();
456try {
457dest = destinations.get(destination);
458} finally {
459destinationsLock.readLock().unlock();
460}

destinations is a ConcurrentHashMap.  Why are we using a single external
lock around a ConcurrentHashMap???  We've got lots of similar code
elsewhere in AbstractRegion.  I suspect that the primary goal is to make
sure that addDestination() and receiveDestination() don't create or remove
the same destination twice, but there are other ways to do that (for
example, having the map be ConcurrentHashMap and putting the AtomicReference into the map
immediately and then later populating the reference once the object is
constructed).

I've submitted https://issues.apache.org/jira/browse/AMQ-6226 to capture
that proposed improvement.

BTW, although I'm sure that the bug Chris referenced can't hurt, it doesn't
appear to be related to the apparent root cause of your issue, so I'd
expect you'll see little or no improvement in 5.13.2.  But it's definitely
worth testing to be sure.

Tim

On Mar 30, 2016 5:17 AM, "Christopher Shannon" <
christopher.l.shan...@gmail.com> wrote:

> Also, the bug fix should at least help the performance on start up but may
> not help with the producer timing out problem.  The main issue is that the
> message has to be dispatched to all subscriptions before the ACK can be
> sent back to the producer, so if you have a lot of subscriptions (such as a
> wildcard subscription on a lot of topics) this could cause ACKs to be slow.
>
> On Tue, Mar 29, 2016 at 8:53 PM, Shobhana  wrote:
>
> > Thank you Christopher for your suggestion. I'll check this with 5.13.2
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-with-KahaDB-as-persistent-store-becomes-very-slow-almost-unresponsive-after-creating-large-s-tp4709985p4710043.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>