Hi Kasun,
I first tried the JMS transport reconnection mechanism with AS 5.1.0 and
then i tried with the newly built AS 5.2.0 pack. However with both these
packs the reconnection works fine where it waits for the JMSConnection to
come alive by retrying in exponentially increasing time intervals. Please
see the AS console log i get in the this situation below.
[2013-08-19 15:28:02,454] INFO {org.wso2.andes.client.AMQConnection} -
Unable to connect to broker at tcp://localhost:5673
org.wso2.andes.transport.TransportException: Could not open connection
at
org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
at
org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
at
org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
at
org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
at
org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
at
org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
at
org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
at
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-19 15:28:02,454] ERROR
{org.apache.axis2.transport.jms.ServiceTaskManager} - Error acquiring a
JMS connection to : QueueConnectionFactory using JNDI properties :
{java.naming.factory.initial=org.wso2.andes.jndi.PropertiesFileInitialContextFactory,
java.naming.provider.url=repository/conf/jndi.properties,
transport.jms.ConnectionFactoryType=queue,
ServiceClass=org.wso2.carbon.mashup.javascript.stubgenerator.JSStubGenerator
}
javax.jms.JMSException: Error creating connection: Could not open connection
at
org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:325)
at
org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
at
org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
at
org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
at
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.wso2.andes.AMQConnectionFailureException: Could not open
connection
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
at
org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
... 10 more
Caused by: org.wso2.andes.transport.TransportException: Could not open
connection
at
org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
at
org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
at
org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
at
org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
... 11 more
[2013-08-19 15:28:07,451] ERROR
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
: 2 for service : HelloService failed. Next retry in 40 seconds
[2013-08-19 15:28:07,451] ERROR
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
: 2 for service : JavaScriptStubGeneratorService failed. Next retry in 40
seconds
[2013-08-19 15:28:47,452] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
: 3 for service : HelloService
[2013-08-19 15:28:47,452] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
: 3 for service : JavaScriptStubGeneratorService
[2013-08-19 15:28:52,453] WARN
{org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
all polling tasks of service : HelloService
[2013-08-19 15:28:52,453] WARN
{org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
all polling tasks of service : JavaScriptStubGeneratorService
[2013-08-19 15:28:52,453] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
service : HelloService shutdown
[2013-08-19 15:28:52,454] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
service : JavaScriptStubGeneratorService shutdown
[2013-08-19 15:28:52,454] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
service : HelloService [re-]initialized
[2013-08-19 15:28:52,454] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
service : JavaScriptStubGeneratorService [re-]initialized
[2013-08-19 15:28:53,455] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
attempt: 3 for service: HelloService was successful!
[2013-08-19 15:28:53,455] INFO
{org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
attempt: 3 for service: JavaScriptStubGeneratorService was successful!
Therefore the exponential reconnection attempts already work with AS 5.2.0
as i observed. Please let us know if this is not the error you see in the
console when MB server is down and AS is trying to reconnect.
Thanks!
Ishara
On Mon, Aug 19, 2013 at 10:15 AM, Kasun Gajasinghe <kas...@wso2.com> wrote:
> Hi,
>
> The error log that repeats is the following. The error logs stop when the
> activemq server is started.
>
> TID: [0] [AS] [2013-08-19 10:12:43,835] ERROR
> {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught
> exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
> org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a JMS
> connection to : QueueConnectionFactory using JNDI properties :
> {java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory,
> org.wso2.xkms2.service.crypto.persistence.enabled=true,
> org.wso2.xkms2.service.crypto.keystore.location=keystore.jks,
> org.wso2.xkms2.service.crypto.server.key.password=password,
> org.wso2.xkms2.service.crypto.keystore.password=password,
> org.wso2.xkms2.service.crypto.default.expriy.interval=365,
> org.wso2.xkms2.service.crypto.default.private.key.password=password,
> org.wso2.xkms2.service.crypto.server.cert.aliase=bob,
> org.wso2.xkms2.service.crypto.issuer.cert.aliase=alice,
> org.wso2.xkms2.service.crypto.issuer.key.password=password,
> java.naming.provider.url=tcp://localhost:61616,
> org.wso2.xkms2.service.crypto.authen.code=secret}
> at
> org.apache.axis2.transport.jms.ServiceTaskManager.handleException(ServiceTaskManager.java:1000)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager.access$700(ServiceTaskManager.java:50)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:835)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
> at
> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.jms.JMSException: Could not connect to broker URL:
> tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:280)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
> at
> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
> ... 7 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:529)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
> at
> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
> ... 11 more
>
>
>
> On Mon, Aug 19, 2013 at 5:38 AM, Ishara Premadasa <ish...@wso2.com> wrote:
>
>> Hi Kasun,
>>
>> We will be looking into this today and update.
>> Thanks!
>>
>>
>> On Sun, Aug 18, 2013 at 12:14 PM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>
>>> Hi,
>>>
>>> It seems that AS 5.2.0 is not patched with the issue with JMS transport
>>> where the services with JMS enabled repeatedly try to establish a
>>> connection with the message broker. This results in GBs of logs in a matter
>>> of minutes filling up the disk.
>>>
>>> As I got to know from Shammi, the solution is to exponentially increase
>>> the waiting time for the connection. Something like, trying in 1sec, 2sec,
>>> 4sec, 8, 16, 32 etc. This fix is already impelemented but it seems that it
>>> didn't get in to AS packs that is about to be released. I tried a setup
>>> with ActiveMQ, and I can see that it tries to establish a connection with
>>> the broker every second or so.
>>>
>>> Can the MB team look in to this ASAP please? Carbon 4.2.0 is due on
>>> coming Thursday.
>>>
>>> Thanks,
>>> KasunG
>>>
>>> --
>>> *Kasun Gajasinghe*
>>> Software Engineer;
>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>
>>>
>>> ,
>>> *email: **kasung AT spamfree wso2.com
>>>
>>>
>>> ** cell: **+94 (77) 678-0813*
>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>
>>>
>>> *
>>> *
>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>
>>>
>>> *
>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>
>>>
>>> *
>>> *
>>>
>>
>>
>>
>> --
>> Ishara Premasada
>> Software Engineer,
>> WSO2 Inc. http://wso2.com/
>> *Blog : http://isharapremadasa.blogspot.com/
>> Twitter : https://twitter.com/ishadil
>> Mobile : +94 714445832*
>>
>>
>>
>
>
> --
> *Kasun Gajasinghe*
> Software Engineer;
> Development Technologies Team, WSO2 Inc.; http://wso2.com
>
>
> ,
> *email: **kasung AT spamfree wso2.com
>
>
> ** cell: **+94 (77) 678-0813*
> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>
>
> *
> *
> *blog: **http://kasunbg.org* <http://kasunbg.org>
>
>
> *
> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>
>
> *
> *
>
--
Ishara Premasada
Software Engineer,
WSO2 Inc. http://wso2.com/
*Blog : http://isharapremadasa.blogspot.com/
Twitter : https://twitter.com/ishadil
Mobile : +94 714445832*
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
