date:20180203

Re: Performance issue with Oracle backed AMQ

2018-02-03 Thread Nathan Wray

I'm going to look into the driver detection Monday and see if I can
determine why the driver isn't recognized, if I can articulate the problem
I'm create a bug.
We're using the stock ojdbc7.jar Oracle thin driver so I'm not aware of any
reason why detection should not work.
I found the resource folder with the driver-named files that refer to the
correct resource adapter class but I haven't found where the correct
filename is generated from the driver class.  I'm assuming it's reflection
somewhere.


On Fri, Feb 2, 2018 at 9:15 PM, Tim Bain  wrote:

> I'm glad you found a solution (one that's already in the codebase, even).
>
> Is the driver one that we should be recognizing as an Oracle driver? Should
> we be writing a bug against the fact that we didn't recognize the driver?
> Or are you using a driver that's custom or exotic in some way?
>
> Tim
>
> On Feb 2, 2018 12:45 PM, "Nathan Wray"  wrote:
>
> > To wrap this up, this issue is known and has been fixed via the
> > OracleJDBCAdapter class.
> >
> > The problem we're seeing is due to AMQ not recognizing our driver as an
> > Oracle driver, which is not unheard of.
> > We should be able to force the correct adapter with something like:
> >
> >
> > 
> >
> >  > dataDirectory="${activemq.base}/activemq-data" dataSource="#oracle-ds"
> > lockKeepAlivePeriod="3000">
> >
> >
> >
> >
> > which will override limitQuery with an inner select and a ROWNUM clause:
> >
> > @Override
> > public String limitQuery(String query) {
> > return "SELECT * FROM (" + query + ") WHERE ROWNUM <= " +
> > getMaxRows();
> > }
> >
> > Thank you again Gary and Tim for your help.
> >
> > Nathan
> >
> >
> >
> > On Fri, Feb 2, 2018 at 11:29 AM, Nathan Wray 
> wrote:
> >
> > > Gary, great leads, thank you.
> > >
> > > I'm planning to verify the performance of replacing the stock query:
> > >
> > > SELECT
> > > ID
> > > ,MSG
> > > FROM
> > > table
> > > WHERE
> > > CONTAINER=?
> > > AND ID < ?
> > > AND ID > ?
> > > AND XID IS NULL
> > > ORDER BY
> > > ID
> > >
> > > With something oracle-specific that will limit the rows in the
> database,
> > > and only perform the inner query on ID.
> > > I'm assuming ID is the PK and is unique w/o container, I'll verify.
> > > (The inner query below should be less expensive as a constraint without
> > > the MSG column, it's required in order to not break the order by
> > semantic.)
> > >
> > > SELECT
> > > t1.ID
> > > ,t1.MSG
> > > FROM
> > > table t1
> > > WHERE
> > > t1.ID IN (
> > > SELECT t2.ID FROM table t2 WHERE t2.CONTAINER=? t2.ID < ? AND
> > > t2.ID > ? AND t2.XID IS NULL ORDER BY t2.ID
> > > )
> > > AND ROWNUM < maxRows
> > >
> > > BR
> > > Nathan
> > >
> > >
> > >
> > >
> > > On Fri, Feb 2, 2018 at 10:09 AM, Gary Tully 
> > wrote:
> > >
> > >> there was some work on limiting queries:
> > >> https://issues.apache.org/jira/browse/AMQ-6049
> > >>
> > >> also - the statements can be configured - so you can provide your own
> > >> statement via configuration.
> > >>
> > >> http://activemq.2283324.n4.nabble.com/Statements-in-Activemq
> > >> -Xml-Jdbcpersistence-adapter-td4668983.html
> > >>
> > >> On Fri, 2 Feb 2018 at 14:27 Nathan Wray  wrote:
> > >>
> > >> > Tim, thanks for writing.
> > >> >
> > >> > Does AMQ support DB-speciifc queries?  I could likely write the
> patch
> > >> > myself if pointed in the right direction.
> > >> >
> > >> > DefaultJDBCAdapter.doRecoverNextMessages looks like a method that
> > >> would be
> > >> > called at start-up, can you verify?
> > >> > Or, under what circumstances is that invoked?
> > >> >
> > >> > The default install had an index on ID; we added an index on
> Container
> > >> plus
> > >> > ID that changed the plan:
> > >> >
> > >> >
> > >> > *Temp Space*
> > >> >
> > >> > *Plan*
> > >> >
> > >> > *SELECT STATEMENT *ALL_ROWS Cost: 4
> > >> >
> > >> > *3 *
> > >> >
> > >> >
> > >> >
> > >> > *3 **FILTER *
> > >> >
> > >> > *2 *
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *2 **TABLE ACCESS BY INDEX ROWID TABLE *ACTIVEMQ.ACTIVEMQ_MSGS
> Cost: 4
> > >> > Bytes: 1,802 Cardinality: 1
> > >> >
> > >> > *1 *
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *1 **INDEX RANGE SCAN INDEX (UNIQUE) *ACTIVEMQ.SYS_C0010280 Cost: 3
> > >> > Cardinality: 1
> > >> >
> > >> >
> > >> >
> > >> > *Temp Space*
> > >> >
> > >> > *Plan*
> > >> >
> > >> > *SELECT STATEMENT *ALL_ROWS Cost: 7
> > >> >
> > >> > *4 *
> > >> >
> > >> >
> > >> >
> > >> > *4 **SORT ORDER BY *Cost: 7 Bytes: 7,208 Cardinality: 4
> > >> >
> > >> > *3 *
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *3 **FILTER *
> > >> >
> > >> > *2 *
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *2 **TABLE ACCESS BY INDEX ROWID BATCHED TABLE
> *ACTIVEMQ.ACTIVEMQ_MSGS
> > >> > Cost:
> > >> > 6 Bytes: 7,208 Cardinality: 4
> > >> >

RE: Thousands of "Transport Connection failed" exceptions

2018-02-03 Thread Andrei Shakirin

Hi,

The reason of an issue is identified.
It was large exception string throwing in JMS handler.
AMQ client tried to send it in dlqDeliveryFailureCause property.

Issue can be easily reproduced using the handler code:

public void onMessage(Message message) {
...
StringBuffer bigBuffer = new StringBuffer(Short.MAX_VALUE);
...
throw new RuntimeException(bigBuffer.toString());
}

This client kills the server using failover protocol

I see two AMQ problems here:
1) Exception message have to be controlled and limited before set in 
dlqDeliveryFailureCause: some exceptions coming from thirdparty and not under 
client handler control
2) Failover reconnection by EVERY IOException is IMO very dangerous

Details in https://issues.apache.org/jira/browse/AMQ-6894

Regards,
Andrei.

> -Original Message-
> From: Andrei Shakirin [mailto:ashaki...@talend.com]
> Sent: Montag, 29. Januar 2018 21:51
> To: users@activemq.apache.org
> Subject: RE: Thousands of "Transport Connection failed" exceptions
> 
> Hi Frizz,
> 
> I have absolutely the same symptoms described in
> http://activemq.2283324.n4.nabble.com/Excessive-number-of-connections-by-
> failover-transport-td4735849.html
> 
> Also using randomize=false=true. After some time of work,
> client spams the server with connections and after short time server becomes
> out of resources.
> Version of client and server is 5.14.5.
> 
> @activemq community: any ideas, workarounds?
> 
> Regards,
> Andrei.
> 
> 
> > -Original Message-
> > From: Frizz [mailto:frizzthe...@googlemail.com]
> > Sent: Montag, 29. Januar 2018 20:12
> > To: users@activemq.apache.org
> > Subject: Thousands of "Transport Connection failed" exceptions
> >
> > My clients connect to AMQ with this connection string:
> > (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=tru
> > e
> >
> > It works - for some time. But sooner or later my AMQ server becomes
> > unresponsive because the host it runs on runs out of resources (threads).
> > The AMQ server basically kills it.
> >
> > activemq.log shows lots of entries like this:
> > ...
> > 2018-01-29 16:50:50,800 | WARN  | Transport Connection to: tcp://
> > 172.13.2.145:45958 failed: java.net.SocketException: Connection reset
> > | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.145:45958@61616 ...
> > 2018-01-29 16:50:52,894 | WARN  | Failed to register MBean
> >
> org.apache.activemq:type=Broker,brokerName=amq1,connector=clientConnec
> > t
> > ors,connectorName=default,connectionViewType=clientId,connectionName=I
> > D
> > _e325fbc8d9c2-41743-1517236130397-0_22
> > ...
> > ...
> >
> > And then I get spammed with thousands of lines like this:
> >
> > 2018-01-29 18:14:40,374 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51089 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51089@61616
> > 2018-01-29 18:14:40,455 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51091 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51091@61616
> > 2018-01-29 18:14:40,537 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51093 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51093@61616
> > 2018-01-29 18:14:40,617 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51095 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51095@61616
> > 2018-01-29 18:14:40,698 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51097 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51097@61616
> > 2018-01-29 18:14:40,780 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51099 failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///172.13.2.150:51099@61616
> > 2018-01-29 18:14:40,860 | WARN  | Transport Connection to: tcp://
> > 172.13.2.150:51101 failed:
> > ...
> >
> > Why is the AMQ server trying to connect to every single port on one of
> > the client machines?

Re: Performance issue with Oracle backed AMQ

RE: Thousands of "Transport Connection failed" exceptions

2 matches

Site Navigation

Mail list logo

Footer information