Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker

[Tim Bain]

Neon,

Since 5.15.3 was just released and you'd no longer need to compile from
source, would it be possible to upgrade to it to see if Gary's hunch about
the root cause of the problem was correct and the problem is fixed in
5.15.3?

Tim

On Tue, Jan 16, 2018 at 4:02 AM, Gary Tully  wrote:

> Hi TIm,
> the regression caused references to batches of serialised messages destined
> for the journal to be retained by the page cache, potentially leading to an
> OOM.
> The temp store used to cursor non persistent messages uses the same journal
> logic and would be effected in the same way.
> The trigger for my hunch was that it only ocurred on 5.12.x
>
> On Fri, 12 Jan 2018 at 13:38 Tim Bain  wrote:
>
> > Gary,
> >
> > Can you elaborate on what about this situation looks like that problem?
> > Because based on the information given so far, it looked to me like the
> OP
> > was simply overdriving the storage to which the temp files are written.
> >
> > Tim
> >
> > On Jan 12, 2018 4:04 AM, "Gary Tully"  wrote:
> >
> > > this looks like an instance of the regression from
> > > https://issues.apache.org/jira/browse/AMQ-6815
> > >
> > > On Wed, 10 Jan 2018 at 19:42 neon18  wrote:
> > >
> > > > We run the broker with max heap of 4G and initial of 1G (-Xmx4G
> > -Xms1G).
> > > > We use non-persistent messages on these particular queues (3 of them
> in
> > > > this
> > > > test).
> > > > The number of messages sent to the broker in my last "flood gate"
> test
> > > was
> > > > around 40,000 (40k) in 5 minutes or about 8K msgs/min. After this
> flood
> > > of
> > > > messages, the producers send messages at a much much lower rate. I
> have
> > > > pretty much the factory default activemq.xml with
> > > > systemUsage/memoryUsage/percentOfJvmHeap=70 and queuePrefetch=20 on
> > > these 3
> > > > queues.
> > > >
> > > > So I have seen two different scenarios when lots of non-persistent
> > > messages
> > > > are put on queue.
> > > > 1. Async Writer Thread Shutdown errors (with no prior
> warnings/errors),
> > > > then
> > > > OutOfMemoryErrors
> > > > 2. INFO PListStore: ...tmp_storage initialized, ~10 seconds later
> WARN:
> > > > IOException: OutOfMemoryError: GC overhead limit exceeded ...
> ActiveMQ
> > > > Transport: tcp:..., then that repeats and other errors follow. Also
> > there
> > > > is
> > > > no warnings/errors prior to the tmp_storage init info log msg. FYI:
> the
> > > web
> > > > console was responsive until I saw the tmp_storage initialized
> (KahaDB)
> > > > INFO
> > > > msg (~4.5 minutes into my test), then it stops responding. The last
> > count
> > > > of
> > > > messages on queues via web console is ~30K msgs under ActiveMQ 5.15.2
> > > > broker. Under the 5.14.5 broker, I was able to see the flood of ~40K
> > msgs
> > > > added to the 3 queues in ~6 minutes.
> > > >
> > > > In more controlled testing in the past 2 days, where I clear the
> > AMQ_DATA
> > > > dir before each test run, I have not seen issue #1 (Async Writer
> Thread
> > > > Shutdown / OutOfMemoryError). I see issue #2 (OutOfMemoryError) a few
> > > > seconds after KahaDB tmp_storage is initialized, then the web console
> > > stop
> > > > responding and lots of OoMerrors and other errors in the
> activemq.log.
> > > >
> > > > Running with ActiveMQ 5.14.5 and 5.12.2 brokers, we do not have any
> > > > OutOfMemoryErrors with this same load or higher load vs running under
> > > > ActiveMQ 5.15.2. Running with 5.15.2 broker it seems like there might
> > be
> > > an
> > > > issue with throttling the producers of the queue when the JVM hit's
> the
> > > > configured memoryUsage (default of 70%).
> > > >
> > > > Running on that thought, I did another test with
> > > > systemUsage/memoryUsage/percentOfJvmHeap=50 %, but same thing
> (except
> > > that
> > > > the OoM error occurs about 20 seconds after the tmp_storage init info
> > > log.
> > > >
> > > > So, I ran the test again with systemUsage/memoryUsage to 20%, same
> > thing,
> > > > except the OoM error occurs about 40 seconds after the tmp_storage
> init
> > > > info
> > > > log. This time, I also monitored the memory percent used and temp
> > memory
> > > > percent used via the web console. right after I see the tmp_storage
> > init
> > > > info log, I can see memUsed=39% tempUsed=1%, ~10 seconds later
> > > memUsed=56%
> > > > tempUsed=2%, ~10 seconds later memUsed=69% tempUsed=2%, next refresh
> > > failed
> > > > and of course in the activemq.log I see the OutOfMemoryErrors and
> other
> > > > warnings and errors appearing in the log.
> > > >
> > > > Also, I grepped my old logs for "Journal failed" and did see some
> > > results,
> > > > but they happen after the a few OutOfMemoryErrors, so I did not
> include
> > > > them
> > > > in this thread.
> > > >
> > > > I can pretty reliably recreate the problem in about 6 minutes (with a
> > > clean
> > > > amq_data_dir) running ActiveMQ 5.15.2 broker and no 

Re: what value of networkTTL need to use for synchronous request and reply in network connectors

[Tim Bain]

In any mesh network, I'd always recommend setting
decreaseNetworkConsumerPriority=true. If you do that, and if clients will
only ever connect to a single broker, then a networkTTL of 2 is fine,
because the message will always traverse at most two brokers.

However, if your consumers can disconnect and reconnect to different
brokers in the mesh (e.g. via a failover URL), then it's possible in a
pathologically bad scenario for the messages to chase the consumer around
the network as it connects to and immediately disconnects from the various
brokers in the network, all the while increasing the networkTTL. Again,
this is in the pathologically bad case, where the consumer disconnects
without doing any work and then connects to a different broker, causing
messages flow to that next broker but the consumer then disconnects without
consuming any of them. In a more normal scenario, message will have to make
at most a few hops before finally getting consumed, so I'd suggest a
networkTTL that's higher than 2; a value like 5 is probably safe in most
cases, but you could use 10, or 100, if you wanted to be even safer.

The more important setting in that case, though, is the
replayWhenNoConsumers setting described in the Stuck Messages section at
the bottom of http://activemq.apache.org/networks-of-brokers.html. Because
if a message is produced on A and the consumer is on C, and the message is
routed to C but the consumer disconnects before it can be consumed and then
reconnects to A, the default configuration will prevent the message from
going back to A and therefore the message will be "lost" (until a consumer
connects to C or to B). That's a far more likely scenario than the one that
would cause a networkTTL of 5/10/100 to be hit, and the
replayWhenNoConsumers attribute allows you to address that situation.

Tim

Tim

On Thu, Feb 8, 2018 at 9:13 AM, Rajesh Malla  wrote:

> We are using n/w connectors between 3 brokers
> is
> A->B
> A->C
>
> B->A
> B->C
>
> C->A
> C->B
>
> also we are using synchronous request and reply pattern using camel to send
> and receive message in temp queue.
> we have question is it good to use networkTTL = 2 in above configuration ?
> will it work properly or
> is there any chance of losing message ?
> for example
>
> A   B   C
> tmpQ  tmpQ  tmpQ
>
> what if producer send message to A and waiting on tmpQ, consumer C receives
> message and reply to  tmpQ [ of C], ?
>
>  A,B,C are producers and consumers on tmpQ
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Re: Deadlock in ActiveMQ server between BrokerFilter and Producer creation with VM connection

[Tim Bain]

I see how #380 is blocked by #308 on lock object 0xfffe833683f0, but it
looks like #308 is blocked waiting on lock object 0xfffe81003250, which
isn't held by any of the three threads that are in the partial thread dump
you provided. What's the thread dump of whatever thread is the one that's
holding that lock?

Also, you may not find much interest in tracking down a deadlock in code
that's over 4 years old, because it's possible that the code has changed
since 5.9.0 in ways that would fix or avoid this problem. Is upgrading to a
current (5.14 or later) version and seeing if the problem reoccurs an
option?

Tim

On Wed, Feb 7, 2018 at 4:01 AM, zyc1984  wrote:

> Hi,
>
> We have identified a potential "deadlock" between BrokerFilter and Producer
> creation with VM connection
> This block is random, it happened on one given machine but not the others.
>
> ActiveMQ version 5.9.0
>
> Thread #310 : notified by BrokerFilter.addConsumer() and it creates a
> Producer with local VM connection
> Thread #308 : notifed by BrokerFilter.messageConsumed() and it tries to get
> Consumer info in our application with read lock
> Thread 380 : it is treating the Producer creation command but it's blocked
> by Thread #308 a lock inside held by ActiveMQ BrokerFilter stack.
>
>
> As this issue is not always happening, i guess it is related to the
> Threading model in ActiveMQ.
> And I would like to know in which case VM connection handling thread will
> be
> blocked by BrokerFilter and in which case it is not. Is it related to jvm
> arguments ? because the configuration of ActiveMQ is hard coded in our
> case.
>
> Thanks,
> zyc1984
>
>
>
>
> "ActiveMQ Transport: tcp:///22.0.195.149:64315@64078" #310 daemon prio=5
> os_prio=64 tid=0x00010274e000 nid=0x14d waiting on condition
> [0xfffe76ffd000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0xfffe8335fb60> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java
> :175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$Condit
> ionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> org.apache.activemq.transport.FutureResponse.getResult(Futur
> eResponse.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(Res
> ponseCorrelator.java:87)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(Active
> MQConnection.java:1394)
> at
> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQS
> ession.java:1925)
> at
> org.apache.activemq.ActiveMQMessageProducer.(
> ActiveMQMessageProducer.java:125)
> at
> org.apache.activemq.ActiveMQSession.createProducer(ActiveMQS
> ession.java:969)
> at EventFeeder.start(GenericEventFeeder.java:63)
> at DefaultConsumerManager.startFeed(DefaultConsumerManager.
> java:192)
> at
> DefaultConsumerAdministration.consumerConnected(DefaultConsu
> merAdministration.java:249)
> at ConsumerListener.onConsumerConnect(ConsumerListener.java:31)
> at ActiveMQInterceptor.addConsumer(ActiveMQInterceptor.java:80)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(M
> utableBrokerFilter.java:102)
> at
> org.apache.activemq.broker.TransportConnection.processAddCon
> sumer(TransportConnection.java:587)
> at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.
> java:347)
> at
> org.apache.activemq.broker.TransportConnection.service(Trans
> portConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(
> TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(Mutex
> Transport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand
> (WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCo
> mmand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(Tra
> nsportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran
> sport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> "ActiveMQ Transport: tcp:///22.0.195.149:64313@64078" #308 daemon prio=5
> os_prio=64 tid=0x0001028d5000 nid=0x14b waiting on condition
> [0xfffe773fd000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0xfffe81003250> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java

Re: partially durable queue?

[Tim Bain]

Sorry this took a while (I've not had much free time to look through the
source code recently) and you may well have already found what you need in
the meantime, but in case you haven't yet, I think you'd want to be looking
in the activemq-kahadb-store project and particularly
org.apache.activemq.store.kahadb.plist.PListImpl or PListStoreImpl. Based
on what you've said, I think you'll be looking to have the PListImpl store
messages in memory without writing to the PageFile until some threshold is
reached, and then begin writing messages into the PageFile at that time.

Since I've not done very much poking in the KahaDB code, it's possible that
I'm off the mark here on where best to implement this, but it's where I'd
start looking if I were you.

Tim

On Mon, Jan 29, 2018 at 3:53 AM, Noel Grandin  wrote:

> Thanks for all the answers. It sounds like I can't quite achieve what I
> want out of the box. (Which is not a surprise)
>
> I'll have a look at the internal code and see how hard it will be to build
> something custom that is basically a combination of a small non-persistent
> in-memory queue, and a larger on-disk persistent queue.
>
> Any ideas where to start looking in the codebase?
>

Re: Message Dequeued count increases by a lot after a message is failed to send to a subscriber

[Tim Bain]

5.15.0 is recent enough; I just wanted to make sure that you weren't using
a version 5.10 or earlier, because there were a number of
statistics-related bugs in the 5.6-5.10 range and I'd hate to be chasing
something that's already been solved.

Since it sounds like a bug, please submit a bug in JIRA for it. If
possible, please include anything you can that would help someone reliably
reproduce the problem: config files, test driver scripts or programs, etc.
Even better would be to create a JUnit test that demonstrates the problem
(and then could be used to confirm the fix once one is made),  if that's
possible. If not, then at least provide whatever you think we'd need to
reproduce the problem in a manual fashion, and the implementor of the fix
will implement the unit test as well.

Tim

On Mon, Feb 5, 2018 at 8:30 AM, kj 
wrote:

> I'm using version 5.15.0. I will download 5.15.2 and test whether I can
> reproduce this. Will keep you updated.
>
> Thanks
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805
> .html
>

Re: Scheduling(delayed delivery) a message AMQP.NETLIBRARY broker: ActiveMQ Windows platform C#

[Timothy Bish]

On 02/12/2018 01:58 AM, bittusrivastava wrote:

Thanks for response. I couldn't find where i can set those header in AMQP.net
library.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

They would be set in the MessageAnnotations on the message that you want 
to be delayed


https://github.com/Azure/amqpnetlite/blob/master/src/Message.cs#L42


--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/

Re: ARTEMIS - Selector modification

[Justin Bertram]

In Artemis when a JMS consumer creates a subscription on a topic it creates
a queue on the broker.  If the subscription has a selector then the queue
on the broker is created with a corresponding "filter".  The filter on a
queue is immutable so there's no way to change the subscription's
selector.  You'd have to create a new subscription with the new selector
and *then* drop the old one so that you don't miss any messages.


Justin

On Mon, Feb 12, 2018 at 10:39 AM, adirubin  wrote:

> Hi, I'm using Artemis 2.3 via JMS.
> I have a durable shared subscription to a topic, with some selector
> (filter). I would like to change the selector at runtime without
> re-subscribing.
>
> I've found some forum threads regarding ActiveMQ (not artemis) that suggest
> it is possible using jmx:
> http://activemq.2283324.n4.nabble.com/Changing-Message-
> Selector-td2347256.html
>
> Is there any such api for artemis? Or any other solution?
>
>
> My use case in detail: I have a highly-available application subscribed to
> a
> topic (a cluster of 2 or more webservers, using shared subscription). The
> app would like to receive only a subset of the messages, but may be
> upgraded
> to handle additional types. The application must be constantly on-line, so
> the servers are upgraded one at a time. Once a server is upgraded, it'll
> attempt to create a durable sared consumer, but with a different selector,
> and will fail.
>
> Thanks,
> Adi Rubin
>
>
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

ARTEMIS - Selector modification

[adirubin]

Hi, I'm using Artemis 2.3 via JMS.
I have a durable shared subscription to a topic, with some selector
(filter). I would like to change the selector at runtime without
re-subscribing.

I've found some forum threads regarding ActiveMQ (not artemis) that suggest
it is possible using jmx:
http://activemq.2283324.n4.nabble.com/Changing-Message-Selector-td2347256.html

Is there any such api for artemis? Or any other solution?


My use case in detail: I have a highly-available application subscribed to a
topic (a cluster of 2 or more webservers, using shared subscription). The
app would like to receive only a subset of the messages, but may be upgraded
to handle additional types. The application must be constantly on-line, so
the servers are upgraded one at a time. Once a server is upgraded, it'll
attempt to create a durable sared consumer, but with a different selector,
and will fail.

Thanks,
Adi Rubin







--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[ANNOUNCE] Apache ActiveMQ 5.15.3 Released

[Christopher Shannon]

Hi everyone,

The ActiveMQ team is pleased to announce that Apache ActiveMQ 5.15.3
has been released.

A list of issues resolved in this release is available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?
projectId=12311210=12341947

The Wiki page for the release is here:
http://activemq.apache.org/activemq-5153-release.html

A big thanks to everyone who contributed to this release.