Harm
Can you issue a "kill - 3 <pid_of_esb>" and get me a thread dump?
Evanthika, can you test this scenario tomorrow to try to reproduce this
issue?
asankha
Harm Verhagen wrote:
Hoi,
I'm experiencing something that looks like a deadlock in wso2esb.
I have a sequence with a number of database accesses. (using mysql).
Basicaly I want to store every message in a database.
When stress testing the esb sometimes locks up, not processing any
messages anymore.
Only stop/start seems to help
The logging seems to indicate its related to the databasepool management.
*ESB logging*
2008-09-25 17:31:56,199 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBReportMediator End : DBReport mediator
2008-09-25 17:31:56,199 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Start : DBLookup mediator
2008-09-25 17:31:56,199 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Getting a connection from DataSource
jdbc:mysql://localhost:3306/esb and preparing statement : select id
from transaction_uuid where uuid = ?
2008-09-25 17:31:56,200 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Setting as parameter : 1 value :
urn:uuid:953256D70E1C3B31A21222356730663 as JDBC Type : 12(see
java.sql.Types for valid types)
2008-09-25 17:31:56,200 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Successfully prepared statement : select id
from transaction_uuid where uuid = ? against DataSource :
jdbc:mysql://localhost:3306/esb
2008-09-25 17:31:56,200 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Processing the first row returned : select id
from transaction_uuid where uuid = ?
2008-09-25 17:31:56,200 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator Column : 1 returned value : 6004 Setting this
as the message property : transaction_id
2008-09-25 17:31:56,201 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBLookupMediator End : DBLookup mediator
2008-09-25 17:31:56,201 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBReportMediator Start : DBReport mediator
2008-09-25 17:31:56,201 [127.0.0.1-vloeki_v01] [HttpServerWorker-5]
DEBUG DBReportMediator Getting a connection from DataSource
jdbc:mysql://localhost:3306/esb and preparing statement : delete from
transaction_uuid where id = ?
...
Here it hangs.. I can get more debug prints out of it by pressing
Configuration -> save but _no_ messages are processed.
When the ESB hangs, Mysql is still up & running fine, I can happily
view the database with mysql querybrowser, make new connections etc.
Any Idea what goes on ?
*how to reproduce*
issue soap request with a stress test tool, 5 threads simultaneously
doing a medium load (30-60 request/second)
Its pretty reproducable, after a few thousens requests I get wso2 to
hang. so withing a few minutues.
*The sequence:*
<syn:sequence name="md_seq">
<syn:property name="Authorization" value="Basic
ZnJlZGRpZTpoZWluZWtlbg==" scope="transport"/>
<syn:log level="headers" separator=",">
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; name="type"
expression="get-property('axis2','messageType')"/>
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; name="test"
expression="get-property('transport', 'Authorization')"/>
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
name="RemoteIP" expression="get-property('axis2','REMOTE_ADDR')"/>
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; name="time"
expression="get-property('SYSTEM_DATE', 'yyyy-MM-dd HH:mm:ss z')"/>
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; name="uid"
expression="get-property('transport', 'MessageID')"/>
<syn:property name="text" value="XXXXXX"/>
<syn:property xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; name="uid"
expression="get-property('MessageID')"/>
</syn:log>
<syn:dblookup>
<syn:connection>
<syn:pool>
<syn:password>esb</syn:password>
<syn:user>esb</syn:user>
<syn:url>jdbc:mysql://localhost:3306/esb</syn:url>
<syn:driver>com.mysql.jdbc.Driver</syn:driver>
<syn:property name="poolstatements" value="true"/>
</syn:pool>
</syn:connection>
<syn:statement>
<syn:sql><![CDATA[select * from customer c where token
= '00000000-00';]]></syn:sql>
<syn:result name="customer_id" column="id"/>
</syn:statement>
</syn:dblookup>
<syn:dbreport>
<syn:connection>
<syn:pool>
<syn:password>esb</syn:password>
<syn:user>esb</syn:user>
<syn:url>jdbc:mysql://localhost:3306/esb</syn:url>
<syn:driver>com.mysql.jdbc.Driver</syn:driver>
<syn:property name="autocommit" value="false"/>
</syn:pool>
</syn:connection>
<syn:statement>
<syn:sql><![CDATA[insert into transaction (
customer_id, token, ip, transaction_type ) values ( ?, ?, ?,1
);]]></syn:sql>
<syn:parameter
xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
expression="get-property('customer_id')" type="INTEGER"/>
<syn:parameter value="A0000000-02BBDDE" type="VARCHAR"/>
<syn:parameter
xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
expression="get-property('axis2','REMOTE_ADDR')" type="VARCHAR"/>
</syn:statement>
<syn:statement>
<syn:sql><![CDATA[insert into transaction_uuid(id,
uuid) values(LAST_INSERT_ID(),?)]]></syn:sql>
<syn:parameter
xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
expression="get-property('MessageID')" type="VARCHAR"/>
</syn:statement>
</syn:dbreport>
<syn:dblookup>
<syn:connection>
<syn:pool>
<syn:password>esb</syn:password>
<syn:user>esb</syn:user>
<syn:url>jdbc:mysql://localhost:3306/esb</syn:url>
<syn:driver>com.mysql.jdbc.Driver</syn:driver>
</syn:pool>
</syn:connection>
<syn:statement>
<syn:sql><![CDATA[select id from transaction_uuid
where uuid = ?]]></syn:sql>
<syn:parameter
xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
expression="get-property('MessageID')" type="VARCHAR"/>
<syn:result name="transaction_id" column="1"/>
</syn:statement>
</syn:dblookup>
<syn:dbreport>
<syn:connection>
<syn:pool>
<syn:password>esb</syn:password>
<syn:user>esb</syn:user>
<syn:url>jdbc:mysql://localhost:3306/esb</syn:url>
<syn:driver>com.mysql.jdbc.Driver</syn:driver>
<syn:property name="autocommit" value="false"/>
</syn:pool>
</syn:connection>
<syn:statement>
<syn:sql><![CDATA[delete from transaction_uuid where
id = ?]]></syn:sql>
<syn:parameter
xmlns:ns1="http://org.apache.synapse/xsd";
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
expression="get-property('transaction_id')" type="INTEGER"/>
</syn:statement>
</syn:dbreport>
<syn:send>
<syn:endpoint>
<syn:address
uri="http://127.0.0.1:8180/boost-platform/services/messageDelivery"/>
</syn:endpoint>
</syn:send>
</syn:sequence>
*version
*OS: debian
wso2 v1.7.1
mysql 5.0
mysql-connector-java-5.0.4
------------------------------------------------------------------------
_______________________________________________
Esb-java-user mailing list
[email protected]
http://mailman.wso2.org/cgi-bin/mailman/listinfo/esb-java-user
_______________________________________________
Esb-java-user mailing list
[email protected]
http://mailman.wso2.org/cgi-bin/mailman/listinfo/esb-java-user
