Re: Migratiion of Queue data from HornetQ to Artemis 2.x
I debug lot but not able to find why this error comes if all things fine 2018-02-02 14:10:45,226 ERROR rejectedExecution Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated - DKumar -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Activemq to azure service bus or event hub
I've never tried to do this myself, but I believe that an ActiveMQ broker can use the AMQP protocol (which Azure Service Bus supports) to connect to a remote broker, so I think you'd be able to set up a networkConnector using the AMQP transport and connect to an Azure Service Bus broker. If you try it, please let us know your experience so others will know if it works. Tim On Feb 2, 2018 12:47 PM, "Tanahara"wrote: > Is there any way to enable all topics to to broadcast to either of these or > do I need to write a subscriber to do it. > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Consumers hanging on a queue although there are messages in it
If you're able to analyze the snapshot, that's great. If you want me to, I can. If you're using any custom broker plugins, it's possible that your package, class, and method names will appear in the stack trace information captured by the snapshot, so if you're using custom plugins, make sure you consider whether any of that is sensitive before sharing it publicly on this list. Ideally you'd only capture the snapshot while the problem is occurring; the more time you capture from when everything is normal, the harder it will be to zero in on the behavior from when things are bad. So I recommend capturing for a while, then if the problem hasn't started occurring after an hour or two, stop the sampling session and start a new one. Tim On Feb 2, 2018 7:57 AM, "Stefanic"wrote: > I will try to remember doing so, are you interested in the CPU sampling > snapshot? And if so over what amount of time? > > (I'm not very experienced with profiling/sampling, only really done > heapdump > and threaddump analysis) > > > Tim Bain wrote > > Would you be able to use JVisualVM to perform CPU sampling while this is > > occurring, to find out what those threads are doing? **WARNING** Do NOT > do > > CPU *profiling*, which would slow your broker significantly; sampling is > > lightweight and generally held to be safe to do against an operational > > process, but profiling is very heavyweight. Make sure you're on the right > > tab in JVisualVM. > > > > Tim > > > > On Feb 2, 2018 12:05 AM, "Stefanic" > > > snicodem@ > > > wrote: > > > >> Hi Tim, > >> > >> There are no selectors, so there are no messages left behind. > >> We have seen this three times now in production and every time when the > >> queue is empty (after about 30 minutes of refreshing) the problem goes > >> away. > >> > >> Threads behavior is rather strange, when the blocking issue is on-going > >> we > >> see 1 thread constantly running (100%) in visualvm, and all other > threads > >> (31 in our case) almost seem synchronized on the same object because all > >> of > >> them start running at the same time and go into timed_waiting state at > >> the > >> same time. > >> That results in 33% running of all other threads and that is not enough > >> to > >> keep our queue empty. > >> > >> Reproducing this behavior is nearly impossible for us, it occurs > randomly > >> within about a week. > >> > >> Here are the things we tried so far: > >> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1 > >> but > >> the problem occurred again on that version so we assume it is not the > >> broker/server > >> - Yesterday we downgraded both ActiveMQ and Camel client libraries: > >> ActiveMQ > >> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather > >> only downgraded 1 at a time but it's our production environment so > cannot > >> play around too much) > >> > >> I will reply here if we experience the blocked state again. > >> > >> > >> > >> -- > >> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > >> f2341805.html > >> > > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Migratiion of Queue data from HornetQ to Artemis 2.x
Thanks Enrich for the update and please let us know when 2.5.x version will be available. Apart from it I ran artemis server and it suddenly down after 10 hours running with below exception 13:16:22,551 ERROR [org.apache.activemq.artemis.core.server] AMQ224018: Failed to create session: ActiveMQSessionCreationException[errorType=SESSION_CREATION_REJECTED message=AMQ119034: Server not started] at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQPacketHandler.handleCreateSession(ActiveMQPacketHandler.java:138) [artemis-server-2.4.0.jar:2.4.0] Please suggest why this is occuring frequently. -Thanks Dharmendra - DKumar -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Performance issue with Oracle backed AMQ
I'm glad you found a solution (one that's already in the codebase, even). Is the driver one that we should be recognizing as an Oracle driver? Should we be writing a bug against the fact that we didn't recognize the driver? Or are you using a driver that's custom or exotic in some way? Tim On Feb 2, 2018 12:45 PM, "Nathan Wray"wrote: > To wrap this up, this issue is known and has been fixed via the > OracleJDBCAdapter class. > > The problem we're seeing is due to AMQ not recognizing our driver as an > Oracle driver, which is not unheard of. > We should be able to force the correct adapter with something like: > > > > > dataDirectory="${activemq.base}/activemq-data" dataSource="#oracle-ds" > lockKeepAlivePeriod="3000"> > > > > > which will override limitQuery with an inner select and a ROWNUM clause: > > @Override > public String limitQuery(String query) { > return "SELECT * FROM (" + query + ") WHERE ROWNUM <= " + > getMaxRows(); > } > > Thank you again Gary and Tim for your help. > > Nathan > > > > On Fri, Feb 2, 2018 at 11:29 AM, Nathan Wray wrote: > > > Gary, great leads, thank you. > > > > I'm planning to verify the performance of replacing the stock query: > > > > SELECT > > ID > > ,MSG > > FROM > > table > > WHERE > > CONTAINER=? > > AND ID < ? > > AND ID > ? > > AND XID IS NULL > > ORDER BY > > ID > > > > With something oracle-specific that will limit the rows in the database, > > and only perform the inner query on ID. > > I'm assuming ID is the PK and is unique w/o container, I'll verify. > > (The inner query below should be less expensive as a constraint without > > the MSG column, it's required in order to not break the order by > semantic.) > > > > SELECT > > t1.ID > > ,t1.MSG > > FROM > > table t1 > > WHERE > > t1.ID IN ( > > SELECT t2.ID FROM table t2 WHERE t2.CONTAINER=? t2.ID < ? AND > > t2.ID > ? AND t2.XID IS NULL ORDER BY t2.ID > > ) > > AND ROWNUM < maxRows > > > > BR > > Nathan > > > > > > > > > > On Fri, Feb 2, 2018 at 10:09 AM, Gary Tully > wrote: > > > >> there was some work on limiting queries: > >> https://issues.apache.org/jira/browse/AMQ-6049 > >> > >> also - the statements can be configured - so you can provide your own > >> statement via configuration. > >> > >> http://activemq.2283324.n4.nabble.com/Statements-in-Activemq > >> -Xml-Jdbcpersistence-adapter-td4668983.html > >> > >> On Fri, 2 Feb 2018 at 14:27 Nathan Wray wrote: > >> > >> > Tim, thanks for writing. > >> > > >> > Does AMQ support DB-speciifc queries? I could likely write the patch > >> > myself if pointed in the right direction. > >> > > >> > DefaultJDBCAdapter.doRecoverNextMessages looks like a method that > >> would be > >> > called at start-up, can you verify? > >> > Or, under what circumstances is that invoked? > >> > > >> > The default install had an index on ID; we added an index on Container > >> plus > >> > ID that changed the plan: > >> > > >> > > >> > *Temp Space* > >> > > >> > *Plan* > >> > > >> > *SELECT STATEMENT *ALL_ROWS Cost: 4 > >> > > >> > *3 * > >> > > >> > > >> > > >> > *3 **FILTER * > >> > > >> > *2 * > >> > > >> > > >> > > >> > > >> > > >> > *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 4 > >> > Bytes: 1,802 Cardinality: 1 > >> > > >> > *1 * > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3 > >> > Cardinality: 1 > >> > > >> > > >> > > >> > *Temp Space* > >> > > >> > *Plan* > >> > > >> > *SELECT STATEMENT *ALL_ROWS Cost: 7 > >> > > >> > *4 * > >> > > >> > > >> > > >> > *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4 > >> > > >> > *3 * > >> > > >> > > >> > > >> > > >> > > >> > *3 **FILTER * > >> > > >> > *2 * > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE *ACTIVEMQ.ACTIVEMQ_MSGS > >> > Cost: > >> > 6 Bytes: 7,208 Cardinality: 4 > >> > > >> > *1 * > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > *1 **INDEX RANGE SCAN INDEX *ACTIVEMQ.ACTIVEMQ_MSGS_CIDX Cost: 4 > >> > Cardinality: 4 > >> > > >> > > >> > > >> > Thank you > >> > Nathan > >> > > >> > > >> > > >> > > >> > On Fri, Feb 2, 2018 at 9:08 AM, Tim Bain > wrote: > >> > > >> > > I've never heard of this particular problem being reported, but it > >> should > >> > > be fairly straightforward to turn it into a top-N query by doing a > >> > > sub-select as described in > >> > > http://www.oracle.com/technetwork/issue-archive/ > >> > > 2006/06-sep/o56asktom-086197.html. > >> > > Would you please submit an enhancement request in JIRA for this > >> change? > >> > > > >> > > BTW, have you had your DBA ensure that an optimal index is in place > on > >> > the > >> > > table and that it's in good repair (Oracle indexes get cluttered > with > >> > > deleted rows over time
Disable the Statistics from the Admin GUI
I have upgrade from 5.11.1 to 5.15.0 for activemq. Now, when logged into the GUI console, there are alot of new "queues" that have ActiveMQ.Statistics.Destination in the name. I would like to know how to remove this from the GUI and it is alot of noise for me. What I have found looking through the archives it to disable the statistics plug in the activemq.xml file. I don't see that in there. Thanks, -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Activemq to azure service bus or event hub
Is there any way to enable all topics to to broadcast to either of these or do I need to write a subscriber to do it. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Can i Use ActiveMq for .net application over amqp protocol
Hi, I am investigating as how can i use activeMQ for .net application. I want to use *AMQP* protocol as well. Can anyone please suggest any pointers as how to do it. there are many implementation but they are connecting to uri "tcp://localhost:61616" but i want to connect to "amqp://localhost:5672". but it throws error as IConnection factory doesnt have implementation for amqp://localhost:5672 at line IConnectionFactory factory= new NMSConnectionFactory("amqp://localhost:5672"); IConnection connection = factory.CreateConnection(); Thanks -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Performance issue with Oracle backed AMQ
To wrap this up, this issue is known and has been fixed via the OracleJDBCAdapter class. The problem we're seeing is due to AMQ not recognizing our driver as an Oracle driver, which is not unheard of. We should be able to force the correct adapter with something like: which will override limitQuery with an inner select and a ROWNUM clause: @Override public String limitQuery(String query) { return "SELECT * FROM (" + query + ") WHERE ROWNUM <= " + getMaxRows(); } Thank you again Gary and Tim for your help. Nathan On Fri, Feb 2, 2018 at 11:29 AM, Nathan Wraywrote: > Gary, great leads, thank you. > > I'm planning to verify the performance of replacing the stock query: > > SELECT > ID > ,MSG > FROM > table > WHERE > CONTAINER=? > AND ID < ? > AND ID > ? > AND XID IS NULL > ORDER BY > ID > > With something oracle-specific that will limit the rows in the database, > and only perform the inner query on ID. > I'm assuming ID is the PK and is unique w/o container, I'll verify. > (The inner query below should be less expensive as a constraint without > the MSG column, it's required in order to not break the order by semantic.) > > SELECT > t1.ID > ,t1.MSG > FROM > table t1 > WHERE > t1.ID IN ( > SELECT t2.ID FROM table t2 WHERE t2.CONTAINER=? t2.ID < ? AND > t2.ID > ? AND t2.XID IS NULL ORDER BY t2.ID > ) > AND ROWNUM < maxRows > > BR > Nathan > > > > > On Fri, Feb 2, 2018 at 10:09 AM, Gary Tully wrote: > >> there was some work on limiting queries: >> https://issues.apache.org/jira/browse/AMQ-6049 >> >> also - the statements can be configured - so you can provide your own >> statement via configuration. >> >> http://activemq.2283324.n4.nabble.com/Statements-in-Activemq >> -Xml-Jdbcpersistence-adapter-td4668983.html >> >> On Fri, 2 Feb 2018 at 14:27 Nathan Wray wrote: >> >> > Tim, thanks for writing. >> > >> > Does AMQ support DB-speciifc queries? I could likely write the patch >> > myself if pointed in the right direction. >> > >> > DefaultJDBCAdapter.doRecoverNextMessages looks like a method that >> would be >> > called at start-up, can you verify? >> > Or, under what circumstances is that invoked? >> > >> > The default install had an index on ID; we added an index on Container >> plus >> > ID that changed the plan: >> > >> > >> > *Temp Space* >> > >> > *Plan* >> > >> > *SELECT STATEMENT *ALL_ROWS Cost: 4 >> > >> > *3 * >> > >> > >> > >> > *3 **FILTER * >> > >> > *2 * >> > >> > >> > >> > >> > >> > *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 4 >> > Bytes: 1,802 Cardinality: 1 >> > >> > *1 * >> > >> > >> > >> > >> > >> > >> > >> > *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3 >> > Cardinality: 1 >> > >> > >> > >> > *Temp Space* >> > >> > *Plan* >> > >> > *SELECT STATEMENT *ALL_ROWS Cost: 7 >> > >> > *4 * >> > >> > >> > >> > *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4 >> > >> > *3 * >> > >> > >> > >> > >> > >> > *3 **FILTER * >> > >> > *2 * >> > >> > >> > >> > >> > >> > >> > >> > *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE *ACTIVEMQ.ACTIVEMQ_MSGS >> > Cost: >> > 6 Bytes: 7,208 Cardinality: 4 >> > >> > *1 * >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > *1 **INDEX RANGE SCAN INDEX *ACTIVEMQ.ACTIVEMQ_MSGS_CIDX Cost: 4 >> > Cardinality: 4 >> > >> > >> > >> > Thank you >> > Nathan >> > >> > >> > >> > >> > On Fri, Feb 2, 2018 at 9:08 AM, Tim Bain wrote: >> > >> > > I've never heard of this particular problem being reported, but it >> should >> > > be fairly straightforward to turn it into a top-N query by doing a >> > > sub-select as described in >> > > http://www.oracle.com/technetwork/issue-archive/ >> > > 2006/06-sep/o56asktom-086197.html. >> > > Would you please submit an enhancement request in JIRA for this >> change? >> > > >> > > BTW, have you had your DBA ensure that an optimal index is in place on >> > the >> > > table and that it's in good repair (Oracle indexes get cluttered with >> > > deleted rows over time when your use pattern is frequent insertions >> and >> > > deletions, so semi-regular rebuilds may be necessary.) >> > > >> > > Tim >> > > >> > > On Feb 1, 2018 10:00 AM, "nathanwray" wrote: >> > > >> > > > We recently had over 1M messages back up in a container. >> > > > >> > > > For reasons that aren't completely clear, AMQ executed the query >> found >> > in >> > > > Statements.getFindNextMessagesStatement mid-morning: >> > > > >> > > > SELECT ID, MSG >> > > > FROM activemq.ACTIVEMQ_MSGS >> > > >WHERE CONTAINER = :1 AND ID > :2 AND ID < :3 AND XID IS NULL >> > > > ORDER BY ID; >> > > > >> > > > With the parameters: >> > > > 1 queue://generic_createContract >> > > > 2 -1 >> > > > 3 183893253 >> > > > >> > > > This call appears to originate from >> > > >
Re: Performance issue with Oracle backed AMQ
Gary, great leads, thank you. I'm planning to verify the performance of replacing the stock query: SELECT ID ,MSG FROM table WHERE CONTAINER=? AND ID < ? AND ID > ? AND XID IS NULL ORDER BY ID With something oracle-specific that will limit the rows in the database, and only perform the inner query on ID. I'm assuming ID is the PK and is unique w/o container, I'll verify. (The inner query below should be less expensive as a constraint without the MSG column, it's required in order to not break the order by semantic.) SELECT t1.ID ,t1.MSG FROM table t1 WHERE t1.ID IN ( SELECT t2.ID FROM table t2 WHERE t2.CONTAINER=? t2.ID < ? AND t2.ID > ? AND t2.XID IS NULL ORDER BY t2.ID ) AND ROWNUM < maxRows BR Nathan On Fri, Feb 2, 2018 at 10:09 AM, Gary Tullywrote: > there was some work on limiting queries: > https://issues.apache.org/jira/browse/AMQ-6049 > > also - the statements can be configured - so you can provide your own > statement via configuration. > > http://activemq.2283324.n4.nabble.com/Statements-in- > Activemq-Xml-Jdbcpersistence-adapter-td4668983.html > > On Fri, 2 Feb 2018 at 14:27 Nathan Wray wrote: > > > Tim, thanks for writing. > > > > Does AMQ support DB-speciifc queries? I could likely write the patch > > myself if pointed in the right direction. > > > > DefaultJDBCAdapter.doRecoverNextMessages looks like a method that would > be > > called at start-up, can you verify? > > Or, under what circumstances is that invoked? > > > > The default install had an index on ID; we added an index on Container > plus > > ID that changed the plan: > > > > > > *Temp Space* > > > > *Plan* > > > > *SELECT STATEMENT *ALL_ROWS Cost: 4 > > > > *3 * > > > > > > > > *3 **FILTER * > > > > *2 * > > > > > > > > > > > > *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 4 > > Bytes: 1,802 Cardinality: 1 > > > > *1 * > > > > > > > > > > > > > > > > *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3 > > Cardinality: 1 > > > > > > > > *Temp Space* > > > > *Plan* > > > > *SELECT STATEMENT *ALL_ROWS Cost: 7 > > > > *4 * > > > > > > > > *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4 > > > > *3 * > > > > > > > > > > > > *3 **FILTER * > > > > *2 * > > > > > > > > > > > > > > > > *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE *ACTIVEMQ.ACTIVEMQ_MSGS > > Cost: > > 6 Bytes: 7,208 Cardinality: 4 > > > > *1 * > > > > > > > > > > > > > > > > > > > > *1 **INDEX RANGE SCAN INDEX *ACTIVEMQ.ACTIVEMQ_MSGS_CIDX Cost: 4 > > Cardinality: 4 > > > > > > > > Thank you > > Nathan > > > > > > > > > > On Fri, Feb 2, 2018 at 9:08 AM, Tim Bain wrote: > > > > > I've never heard of this particular problem being reported, but it > should > > > be fairly straightforward to turn it into a top-N query by doing a > > > sub-select as described in > > > http://www.oracle.com/technetwork/issue-archive/ > > > 2006/06-sep/o56asktom-086197.html. > > > Would you please submit an enhancement request in JIRA for this change? > > > > > > BTW, have you had your DBA ensure that an optimal index is in place on > > the > > > table and that it's in good repair (Oracle indexes get cluttered with > > > deleted rows over time when your use pattern is frequent insertions and > > > deletions, so semi-regular rebuilds may be necessary.) > > > > > > Tim > > > > > > On Feb 1, 2018 10:00 AM, "nathanwray" wrote: > > > > > > > We recently had over 1M messages back up in a container. > > > > > > > > For reasons that aren't completely clear, AMQ executed the query > found > > in > > > > Statements.getFindNextMessagesStatement mid-morning: > > > > > > > > SELECT ID, MSG > > > > FROM activemq.ACTIVEMQ_MSGS > > > >WHERE CONTAINER = :1 AND ID > :2 AND ID < :3 AND XID IS NULL > > > > ORDER BY ID; > > > > > > > > With the parameters: > > > > 1 queue://generic_createContract > > > > 2 -1 > > > > 3 183893253 > > > > > > > > This call appears to originate from > > > > DefaultJDBCAdapter.doRecoverNextMessages, which calls setMaxRows on > the > > > > PreparedStatement with (apparently) 200 rows. > > > > > > > > However it appears the Oracle thin driver makes no attempt to limit > the > > > > result set based on the max rows value; instead it selects and sorts > > the > > > > entire 1M row plus values and makes them available to the client, > which > > > > stops creating objects after reading the first 200. > > > > > > > > The net result was that our Oracle server spiked to 90%+ on this one > > > query > > > > and caused a complete AMQ failure in production, knocking over a > number > > > of > > > > critical systems. > > > > > > > > We wound up stopping all of our brokers and manually dropping the > > > messages > > > > in this container in order to resolve the 3 hour outage. > > > > > > > > Is there a known issue with using Oracle persistence
Re: Performance issue with Oracle backed AMQ
there was some work on limiting queries: https://issues.apache.org/jira/browse/AMQ-6049 also - the statements can be configured - so you can provide your own statement via configuration. http://activemq.2283324.n4.nabble.com/Statements-in-Activemq-Xml-Jdbcpersistence-adapter-td4668983.html On Fri, 2 Feb 2018 at 14:27 Nathan Wraywrote: > Tim, thanks for writing. > > Does AMQ support DB-speciifc queries? I could likely write the patch > myself if pointed in the right direction. > > DefaultJDBCAdapter.doRecoverNextMessages looks like a method that would be > called at start-up, can you verify? > Or, under what circumstances is that invoked? > > The default install had an index on ID; we added an index on Container plus > ID that changed the plan: > > > *Temp Space* > > *Plan* > > *SELECT STATEMENT *ALL_ROWS Cost: 4 > > *3 * > > > > *3 **FILTER * > > *2 * > > > > > > *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 4 > Bytes: 1,802 Cardinality: 1 > > *1 * > > > > > > > > *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3 > Cardinality: 1 > > > > *Temp Space* > > *Plan* > > *SELECT STATEMENT *ALL_ROWS Cost: 7 > > *4 * > > > > *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4 > > *3 * > > > > > > *3 **FILTER * > > *2 * > > > > > > > > *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE *ACTIVEMQ.ACTIVEMQ_MSGS > Cost: > 6 Bytes: 7,208 Cardinality: 4 > > *1 * > > > > > > > > > > *1 **INDEX RANGE SCAN INDEX *ACTIVEMQ.ACTIVEMQ_MSGS_CIDX Cost: 4 > Cardinality: 4 > > > > Thank you > Nathan > > > > > On Fri, Feb 2, 2018 at 9:08 AM, Tim Bain wrote: > > > I've never heard of this particular problem being reported, but it should > > be fairly straightforward to turn it into a top-N query by doing a > > sub-select as described in > > http://www.oracle.com/technetwork/issue-archive/ > > 2006/06-sep/o56asktom-086197.html. > > Would you please submit an enhancement request in JIRA for this change? > > > > BTW, have you had your DBA ensure that an optimal index is in place on > the > > table and that it's in good repair (Oracle indexes get cluttered with > > deleted rows over time when your use pattern is frequent insertions and > > deletions, so semi-regular rebuilds may be necessary.) > > > > Tim > > > > On Feb 1, 2018 10:00 AM, "nathanwray" wrote: > > > > > We recently had over 1M messages back up in a container. > > > > > > For reasons that aren't completely clear, AMQ executed the query found > in > > > Statements.getFindNextMessagesStatement mid-morning: > > > > > > SELECT ID, MSG > > > FROM activemq.ACTIVEMQ_MSGS > > >WHERE CONTAINER = :1 AND ID > :2 AND ID < :3 AND XID IS NULL > > > ORDER BY ID; > > > > > > With the parameters: > > > 1 queue://generic_createContract > > > 2 -1 > > > 3 183893253 > > > > > > This call appears to originate from > > > DefaultJDBCAdapter.doRecoverNextMessages, which calls setMaxRows on the > > > PreparedStatement with (apparently) 200 rows. > > > > > > However it appears the Oracle thin driver makes no attempt to limit the > > > result set based on the max rows value; instead it selects and sorts > the > > > entire 1M row plus values and makes them available to the client, which > > > stops creating objects after reading the first 200. > > > > > > The net result was that our Oracle server spiked to 90%+ on this one > > query > > > and caused a complete AMQ failure in production, knocking over a number > > of > > > critical systems. > > > > > > We wound up stopping all of our brokers and manually dropping the > > messages > > > in this container in order to resolve the 3 hour outage. > > > > > > Is there a known issue with using Oracle persistence with AMQ that > > > precludes > > > having more than thousands of messages? If the "max rows" approach > with > > > the > > > thin driver works as it appears to, we can't be the first to have seen > > this > > > problem. > > > > > > Any insight would be appreciated. > > > > > > Thank you > > > Nathan > > > > > > > > > > > > > > > > > > -- > > > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > > > f2341805.html > > > > > >
Re: Performance issue with Oracle backed AMQ
Tim, thanks for writing. Does AMQ support DB-speciifc queries? I could likely write the patch myself if pointed in the right direction. DefaultJDBCAdapter.doRecoverNextMessages looks like a method that would be called at start-up, can you verify? Or, under what circumstances is that invoked? The default install had an index on ID; we added an index on Container plus ID that changed the plan: *Temp Space* *Plan* *SELECT STATEMENT *ALL_ROWS Cost: 4 *3 * *3 **FILTER * *2 * *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 4 Bytes: 1,802 Cardinality: 1 *1 * *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3 Cardinality: 1 *Temp Space* *Plan* *SELECT STATEMENT *ALL_ROWS Cost: 7 *4 * *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4 *3 * *3 **FILTER * *2 * *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE *ACTIVEMQ.ACTIVEMQ_MSGS Cost: 6 Bytes: 7,208 Cardinality: 4 *1 * *1 **INDEX RANGE SCAN INDEX *ACTIVEMQ.ACTIVEMQ_MSGS_CIDX Cost: 4 Cardinality: 4 Thank you Nathan On Fri, Feb 2, 2018 at 9:08 AM, Tim Bainwrote: > I've never heard of this particular problem being reported, but it should > be fairly straightforward to turn it into a top-N query by doing a > sub-select as described in > http://www.oracle.com/technetwork/issue-archive/ > 2006/06-sep/o56asktom-086197.html. > Would you please submit an enhancement request in JIRA for this change? > > BTW, have you had your DBA ensure that an optimal index is in place on the > table and that it's in good repair (Oracle indexes get cluttered with > deleted rows over time when your use pattern is frequent insertions and > deletions, so semi-regular rebuilds may be necessary.) > > Tim > > On Feb 1, 2018 10:00 AM, "nathanwray" wrote: > > > We recently had over 1M messages back up in a container. > > > > For reasons that aren't completely clear, AMQ executed the query found in > > Statements.getFindNextMessagesStatement mid-morning: > > > > SELECT ID, MSG > > FROM activemq.ACTIVEMQ_MSGS > >WHERE CONTAINER = :1 AND ID > :2 AND ID < :3 AND XID IS NULL > > ORDER BY ID; > > > > With the parameters: > > 1 queue://generic_createContract > > 2 -1 > > 3 183893253 > > > > This call appears to originate from > > DefaultJDBCAdapter.doRecoverNextMessages, which calls setMaxRows on the > > PreparedStatement with (apparently) 200 rows. > > > > However it appears the Oracle thin driver makes no attempt to limit the > > result set based on the max rows value; instead it selects and sorts the > > entire 1M row plus values and makes them available to the client, which > > stops creating objects after reading the first 200. > > > > The net result was that our Oracle server spiked to 90%+ on this one > query > > and caused a complete AMQ failure in production, knocking over a number > of > > critical systems. > > > > We wound up stopping all of our brokers and manually dropping the > messages > > in this container in order to resolve the 3 hour outage. > > > > Is there a known issue with using Oracle persistence with AMQ that > > precludes > > having more than thousands of messages? If the "max rows" approach with > > the > > thin driver works as it appears to, we can't be the first to have seen > this > > problem. > > > > Any insight would be appreciated. > > > > Thank you > > Nathan > > > > > > > > > > > > -- > > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > > f2341805.html > > >
Re: Performance issue with Oracle backed AMQ
I've never heard of this particular problem being reported, but it should be fairly straightforward to turn it into a top-N query by doing a sub-select as described in http://www.oracle.com/technetwork/issue-archive/2006/06-sep/o56asktom-086197.html. Would you please submit an enhancement request in JIRA for this change? BTW, have you had your DBA ensure that an optimal index is in place on the table and that it's in good repair (Oracle indexes get cluttered with deleted rows over time when your use pattern is frequent insertions and deletions, so semi-regular rebuilds may be necessary.) Tim On Feb 1, 2018 10:00 AM, "nathanwray"wrote: > We recently had over 1M messages back up in a container. > > For reasons that aren't completely clear, AMQ executed the query found in > Statements.getFindNextMessagesStatement mid-morning: > > SELECT ID, MSG > FROM activemq.ACTIVEMQ_MSGS >WHERE CONTAINER = :1 AND ID > :2 AND ID < :3 AND XID IS NULL > ORDER BY ID; > > With the parameters: > 1 queue://generic_createContract > 2 -1 > 3 183893253 > > This call appears to originate from > DefaultJDBCAdapter.doRecoverNextMessages, which calls setMaxRows on the > PreparedStatement with (apparently) 200 rows. > > However it appears the Oracle thin driver makes no attempt to limit the > result set based on the max rows value; instead it selects and sorts the > entire 1M row plus values and makes them available to the client, which > stops creating objects after reading the first 200. > > The net result was that our Oracle server spiked to 90%+ on this one query > and caused a complete AMQ failure in production, knocking over a number of > critical systems. > > We wound up stopping all of our brokers and manually dropping the messages > in this container in order to resolve the 3 hour outage. > > Is there a known issue with using Oracle persistence with AMQ that > precludes > having more than thousands of messages? If the "max rows" approach with > the > thin driver works as it appears to, we can't be the first to have seen this > problem. > > Any insight would be appreciated. > > Thank you > Nathan > > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Message Dequeued count increases by a lot after a message is failed to send to a subscriber
That sounds like a bug. What version of ActiveMQ are you using? If it's anything before 5.14.x, can you please try a newer version (5.15.2, ideally) to confirm that it still occurs in a current version? Thanks, Tim On Feb 1, 2018 11:23 AM, "kj"wrote: > This is the story: > > I am testing whether messages are sent to the DLQ if a subscriber has > failed > to receive a message. > > I start off with the following settings. > > Topic = topic1 > Messages Enqueued = 0 > Messages Dequeued = 0 > Number of subscribers to topic1 = 1 > Retry policy count = 0 > > I post a message to topic1 using the ActiveMQ web console. The subscriber > receives the message as expected and it throws a runtime exception as > expected. I except the message to be sent to the DLQ and that happens as > expected. After this all happens I check the number of messages enqueued > and > it says 1, which is expected. I check the number of messages dequeued and > it > says 2, which is what I don't understand. I'm under the assumption that it > should say 1. > > Could someone please clarify whether why the count is 2? > > Thank You > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
Re: Static network of brokers with not all brokers running (expected)
Are you looking to use the discovery transport for the client connections, or both? I've never used it, but my understanding is that discovery is like failover (connect to one broker in this list) rather than like static (connect to all the brokers in this list), so it would be reasonable to use on the clients but probably not what you want for connecting the brokers to each other. Tim
Re: Consumers hanging on a queue although there are messages in it
Would you be able to use JVisualVM to perform CPU sampling while this is occurring, to find out what those threads are doing? **WARNING** Do NOT do CPU *profiling*, which would slow your broker significantly; sampling is lightweight and generally held to be safe to do against an operational process, but profiling is very heavyweight. Make sure you're on the right tab in JVisualVM. Tim On Feb 2, 2018 12:05 AM, "Stefanic"wrote: > Hi Tim, > > There are no selectors, so there are no messages left behind. > We have seen this three times now in production and every time when the > queue is empty (after about 30 minutes of refreshing) the problem goes > away. > > Threads behavior is rather strange, when the blocking issue is on-going we > see 1 thread constantly running (100%) in visualvm, and all other threads > (31 in our case) almost seem synchronized on the same object because all of > them start running at the same time and go into timed_waiting state at the > same time. > That results in 33% running of all other threads and that is not enough to > keep our queue empty. > > Reproducing this behavior is nearly impossible for us, it occurs randomly > within about a week. > > Here are the things we tried so far: > - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1 but > the problem occurred again on that version so we assume it is not the > broker/server > - Yesterday we downgraded both ActiveMQ and Camel client libraries: > ActiveMQ > down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather > only downgraded 1 at a time but it's our production environment so cannot > play around too much) > > I will reply here if we experience the blocked state again. > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
RE: ActiveMQ custom plugin
OK, great. In that case you would probably want to make a custom processor in Camel that transforms the message however you need it to, and insert the processor in the route before the . Tim On Feb 2, 2018 1:23 AM, "Gupta Bharat"wrote: Yes I am using 5.x currently. Using Apache Camel I am routing messages from ActiveMQ to IBM MQ. http://camel.apache.org/schema/blueprint;> Regards, Bharat Gupta Development & Support | Platform Technician, IXCB | Scania IT Mobile: +46700869007, +91-9718483802 bharat.gu...@scania.com -Original Message- From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain Sent: den 2 februari 2018 07:06 To: ActiveMQ Users Subject: RE: ActiveMQ custom plugin Let's go back to basics. Are you using 5.x or Artemis? You say you're routing messages from ActiveMQ to IBM MQ. How is that being done? Tim On Feb 1, 2018 7:23 AM, "Gupta Bharat" wrote: I don't wish to add headers in the message I pasted. This message I am getting from user and it is content. Even the tag header is part of content. But when I route the message from Active MQ to IBM MQ certain attributes like destination topic name and message id need to be added. Also I am using MQTT client to publish a message on ActiveMQ specifying the topic name and message. I need to add that topic name as destination and also want to remove JWTtoken from message. Publication: Topic: Demo Message: {"header":{"messageId":"fcbef565-18ea-429a-836f- 7e24d9b53743","jwtToken":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9. eyJ1bmlxdWVfbmFtZSI6IndpaW5pa3B2dCIsInN1YiI6IndpaW5pa3B2dCIs ImtpZCI6ImMxMzFhMWI0LWU0YWUtNGUwNC1hMGYwLWViNzViYThjYTFjMyIs IkV4dGVybmFsU3RhZmZSZWZlcmVuY2UiOiJmZWNiMDQ1Ny05MmNhLTRmMDAt OTIwNC1lZjAwNTYzZDgxZGIiLCJFeHRlcm5hbEN1c3RvbWVyUmVmZXJlbmNl IjoiYTkxZWYzMTYtNTJhMi00ZWE0LTk3ZDMtNDI5MDhhMGNjYjQ3IiwiQWRB Y2NvdW50IjoibndpN2E0IiwiRnVuY3Rpb25hbFBlcm1pc3Npb25zIjoiNDkx MDA3IiwiVXNlclBlcm1pc3Npb25zIjpbIjI0NDUxMTQ3NTAiLCI5OTIiXSwi aXNzIjoiaHR0cHM6Ly9mbXNkZXZhdXRoLnNjYW5pYS5jb20vIiwiYXVkIjoi MmMxNzk2NzktN2VjMS00MGQxLTkxNjAtNjk3NjEyOTg0ZDg1IiwiZXhwIjox NTE1Njc5Njk4LCJuYmYiOjE1MTU2NzYwOTh9.Olt1zXeDA1sndO1d4M1y07uSBzj7Bx A6Plvmf8TCLfeQhh3-Fyx-wMIyHhXWSNFgI6tRDVL0heJftxATcGbe_ 0u1nVp0vwnrTO7PuGuKzEhWYXFlGs83IasN7hIidOp1qP9hXzi3i_ VKVNa0wILUsuBMza7nkPd7xuTgCqacuFDpBZowyt-V1PUvqyj2Gd50ADh_ouuazO3akN_ xHCFENJGDXLx91ABuJbGGiUsPI16uDcxAzDmfeKgnwbF5Zs8dw3TE_ LiiBCcICMKx81rp6L_RTkPuoe5Umv-Y2ihfcnDt1trp36TvpUsSa- TqtcvZntwkDBcCNKhoyEB5FokMxatK5xgnpjqpj1kwZmHqfzfsJwFDBHY9Cd 73UQpigmarLc7RgyWL3O_QZXxDeVP1vxO4V2NK1im3Jc0BjL4vs rcIqrWUn_v4pJCMQMFtSZ2hkg8Yw_VRnrvJToF4z9OJa_yDN__bIC5_ Ezl4HvvJ6jTrkr1sqQTJKq4ipyx-4pG9Ns5KPZezjgEraBMENcAHzCcIpI vI53Bmu6aPneFgEjOAQaJ5Dgzu-Cr4fesjEgIyBVt2bHPGa-ztAA7pfop-3BXsVjwXPsbc_7- tQanCr_gANHtZXQYXvjxM14PFuZAjc-GERg9IkCokgRd3x8xYlNWhRtiCZ47k flNNwt1_QVo","correlId":"","initialMessageid":"fcbef565- 18ea-429a-836f-7e24d9b53743","environment":"st5","producerUTCTimestamp": 1515677099337,"contractId":"SCCO3822","producerVer":"1.0","messageType":" SendChatMessageInConversationMessage","appId":"fma","messageTypeVer":"1.0"," receivers":[]},"body":{"conversationId":"2b575462- 97bd-4e9a-8ac4-6fec793c836a","chatMessageId":"098ee4e7-8976- 4c21-8f0a-521873fd6ac5","content":"grte","attachments":[]}} I need *Demo* to be added in headers and remove jwtToken from message. I removed it. But what method I can use in ActiveMQ to publish it on Topic Demo instead of message with jwtToken value. I Hope I am able to explain you better :) Regards, Bharat Gupta -Original Message- From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain Sent: den 1 februari 2018 15:02 To: ActiveMQ Users Cc: users-ow...@activemq.apache.org Subject: RE: ActiveMQ custom plugin Oh, and I missed your second question. I would expect that you could add headers similar to how the code you posted removed some. Does your question mean that you tried that and it didn't work? If so, how did it fail? Tim On Feb 1, 2018 6:59 AM, "Tim Bain" wrote: > Bharat, > > Is this in 5.x, or did you switch to Artemis? > > Your code is operating on a message, but what called your code? We'd need > some context on how you got that message to know what you should do with it > next. > > Tim > > On Feb 1, 2018 5:01 AM, "Gupta Bharat" wrote: > >> Hello Tim, >> >> I am getting following message from client using MQTT >> >> { >> "header": { >> "messageId": "fcbef565-18ea-429a-836f-7e24d9b53743", >> "jwtToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSU
RE: ActiveMQ custom plugin
Yes I am using 5.x currently. Using Apache Camel I am routing messages from ActiveMQ to IBM MQ. http://camel.apache.org/schema/blueprint;> Regards, Bharat Gupta Development & Support | Platform Technician, IXCB | Scania IT Mobile: +46700869007, +91-9718483802 bharat.gu...@scania.com -Original Message- From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain Sent: den 2 februari 2018 07:06 To: ActiveMQ UsersSubject: RE: ActiveMQ custom plugin Let's go back to basics. Are you using 5.x or Artemis? You say you're routing messages from ActiveMQ to IBM MQ. How is that being done? Tim On Feb 1, 2018 7:23 AM, "Gupta Bharat" wrote: I don't wish to add headers in the message I pasted. This message I am getting from user and it is content. Even the tag header is part of content. But when I route the message from Active MQ to IBM MQ certain attributes like destination topic name and message id need to be added. Also I am using MQTT client to publish a message on ActiveMQ specifying the topic name and message. I need to add that topic name as destination and also want to remove JWTtoken from message. Publication: Topic: Demo Message: {"header":{"messageId":"fcbef565-18ea-429a-836f- 7e24d9b53743","jwtToken":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9. eyJ1bmlxdWVfbmFtZSI6IndpaW5pa3B2dCIsInN1YiI6IndpaW5pa3B2dCIs ImtpZCI6ImMxMzFhMWI0LWU0YWUtNGUwNC1hMGYwLWViNzViYThjYTFjMyIs IkV4dGVybmFsU3RhZmZSZWZlcmVuY2UiOiJmZWNiMDQ1Ny05MmNhLTRmMDAt OTIwNC1lZjAwNTYzZDgxZGIiLCJFeHRlcm5hbEN1c3RvbWVyUmVmZXJlbmNl IjoiYTkxZWYzMTYtNTJhMi00ZWE0LTk3ZDMtNDI5MDhhMGNjYjQ3IiwiQWRB Y2NvdW50IjoibndpN2E0IiwiRnVuY3Rpb25hbFBlcm1pc3Npb25zIjoiNDkx MDA3IiwiVXNlclBlcm1pc3Npb25zIjpbIjI0NDUxMTQ3NTAiLCI5OTIiXSwi aXNzIjoiaHR0cHM6Ly9mbXNkZXZhdXRoLnNjYW5pYS5jb20vIiwiYXVkIjoi MmMxNzk2NzktN2VjMS00MGQxLTkxNjAtNjk3NjEyOTg0ZDg1IiwiZXhwIjox NTE1Njc5Njk4LCJuYmYiOjE1MTU2NzYwOTh9.Olt1zXeDA1sndO1d4M1y07uSBzj7Bx A6Plvmf8TCLfeQhh3-Fyx-wMIyHhXWSNFgI6tRDVL0heJftxATcGbe_ 0u1nVp0vwnrTO7PuGuKzEhWYXFlGs83IasN7hIidOp1qP9hXzi3i_ VKVNa0wILUsuBMza7nkPd7xuTgCqacuFDpBZowyt-V1PUvqyj2Gd50ADh_ouuazO3akN_ xHCFENJGDXLx91ABuJbGGiUsPI16uDcxAzDmfeKgnwbF5Zs8dw3TE_ LiiBCcICMKx81rp6L_RTkPuoe5Umv-Y2ihfcnDt1trp36TvpUsSa- TqtcvZntwkDBcCNKhoyEB5FokMxatK5xgnpjqpj1kwZmHqfzfsJwFDBHY9Cd 73UQpigmarLc7RgyWL3O_QZXxDeVP1vxO4V2NK1im3Jc0BjL4vs rcIqrWUn_v4pJCMQMFtSZ2hkg8Yw_VRnrvJToF4z9OJa_yDN__bIC5_ Ezl4HvvJ6jTrkr1sqQTJKq4ipyx-4pG9Ns5KPZezjgEraBMENcAHzCcIpI vI53Bmu6aPneFgEjOAQaJ5Dgzu-Cr4fesjEgIyBVt2bHPGa-ztAA7pfop-3BXsVjwXPsbc_7- tQanCr_gANHtZXQYXvjxM14PFuZAjc-GERg9IkCokgRd3x8xYlNWhRtiCZ47k flNNwt1_QVo","correlId":"","initialMessageid":"fcbef565- 18ea-429a-836f-7e24d9b53743","environment":"st5","producerUTCTimestamp": 1515677099337,"contractId":"SCCO3822","producerVer":"1.0","messageType":" SendChatMessageInConversationMessage","appId":"fma","messageTypeVer":"1.0"," receivers":[]},"body":{"conversationId":"2b575462- 97bd-4e9a-8ac4-6fec793c836a","chatMessageId":"098ee4e7-8976- 4c21-8f0a-521873fd6ac5","content":"grte","attachments":[]}} I need *Demo* to be added in headers and remove jwtToken from message. I removed it. But what method I can use in ActiveMQ to publish it on Topic Demo instead of message with jwtToken value. I Hope I am able to explain you better :) Regards, Bharat Gupta -Original Message- From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain Sent: den 1 februari 2018 15:02 To: ActiveMQ Users Cc: users-ow...@activemq.apache.org Subject: RE: ActiveMQ custom plugin Oh, and I missed your second question. I would expect that you could add headers similar to how the code you posted removed some. Does your question mean that you tried that and it didn't work? If so, how did it fail? Tim On Feb 1, 2018 6:59 AM, "Tim Bain" wrote: > Bharat, > > Is this in 5.x, or did you switch to Artemis? > > Your code is operating on a message, but what called your code? We'd need > some context on how you got that message to know what you should do with it > next. > > Tim > > On Feb 1, 2018 5:01 AM, "Gupta Bharat" wrote: > >> Hello Tim, >> >> I am getting following message from client using MQTT >> >> { >> "header": { >> "messageId": "fcbef565-18ea-429a-836f-7e24d9b53743", >> "jwtToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSU >> zI1NiJ9.eyJ1bmlxdWVfbmFtZSI6IndpaW5pa3B2dCIsInN1YiI6IndpaW5p >> a3B2dCIsImtpZCI6ImMxMzFhMWI0LWU0YWUtNGUwNC1hMGYwLWViNzViYThj >> YTFjMyIsIkV4dGVybmFsU3RhZmZSZWZlcmVuY2UiOiJmZWNiMDQ1Ny05MmNh >> LTRmMDAtOTIwNC1lZjAwNTYzZDgxZGIiLCJFeHRlcm5hbEN1c3RvbWVyUmVm >>