Hi all,
in my project we are widely adopting a "customized" version of GT4.0.X (we
call it *gHN*). Since the beginning, we have been experiencing an issue we
were never able to manage and now we would like to understand if this is a
well-known problem or is due to our customization.
The problem is the following: we have an infrastructure composed by tens of
nodes equipped with the gHN and it is supposed to stay up and running even
for months without any human intervention. Almost all works well, but the
most contacted containers (2/3 in the whole infrastructure) after some days
(or weeks, it depends on their exploitation) do stop to manage incoming SOAP
calls. On the client side, we observe a time out exception on the connection
attempt (that I report below), but that's it. We have deeply investigated
the hosting machines, but all looks fine there (I mean, the memory is
available, the maximum number of file descriptors is not reached and so on).
We also went through the Java WS Core (we are using the 4.0.4 version)
source code looking for eventually-left-open Streams, but nothing wrong was
discovered there.
Did anyone else experience this behaviour? Or can anyone suggest where to
look for?
Thanks in advance for any hint.
manuele
----------------------------------------------------------------------------------------------------------
AxisFault
faultCode:
{http://schemas.xmlsoap.org/soap/envelope/}Server.userException<http://schemas.xmlsoap.org/soap/envelope/%7DServer.userException>
faultSubcode:
faultString: java.net.ConnectException: Connection timed out
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:java.net.ConnectException<http://xml.apache.org/axis/%7DstackTrace:java.net.ConnectException>:
Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:520)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.components.net.DefaultSocketFactory.create(DefaultSocketFactory.java:153)
at
org.apache.axis.components.net.DefaultSocketFactory.create(DefaultSocketFactory.java:120)
at
org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:181)
at
org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:397)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:135)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
at org.apache.axis.client.Call.invoke(Call.java:2710)
at org.apache.axis.client.Call.invoke(Call.java:2386)
at org.apache.axis.client.Call.invoke(Call.java:2309)
at org.apache.axis.client.Call.invoke(Call.java:1766)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub.updateResource(RegistryFactoryPortTypeSOAPBindingStub.java:1191)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub$$FastClassByCGLIB$$23292bb5.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at
org.gcube.common.core.contexts.GCUBERemotePortTypeContext$GCUBEMethodInterceptor.intercept(GCUBERemotePortTypeContext.java:382)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub$$EnhancerByCGLIB$$4b03b469.updateResource(<generated>)
at
org.gcube.common.informationsystem.publisher.impl.tasks.ISRegistryServiceUpdaterHandler.interact(Unknown
Source)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler._interact(GCUBEServiceHandler.java:164)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler.tryInstances(GCUBEServiceHandler.java:194)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler.run(GCUBEServiceHandler.java:112)
at
org.gcube.common.informationsystem.publisher.impl.GCUBEProfileManager.update(Unknown
Source)
at
org.gcube.common.informationsystem.publisher.impl.GCUBEPublisher.updateGCUBEResource(Unknown
Source)
at
org.gcube.common.vremanagement.ghnmanager.impl.contexts.ServiceContext$1.onGHNUpdated(ServiceContext.java:194)
at
org.gcube.common.core.contexts.ghn.GHNConsumer.onEvent(GHNConsumer.java:21)
at
org.gcube.common.core.utils.events.GCUBEProducer$1.run(GCUBEProducer.java:172)
{http://xml.apache.org/axis/}hostname:nb-manuele.isti.cnr.it<http://xml.apache.org/axis/%7Dhostname:nb-manuele.isti.cnr.it>
java.net.ConnectException: Connection timed out
at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
at org.apache.axis.client.Call.invoke(Call.java:2710)
at org.apache.axis.client.Call.invoke(Call.java:2386)
at org.apache.axis.client.Call.invoke(Call.java:2309)
at org.apache.axis.client.Call.invoke(Call.java:1766)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub.updateResource(RegistryFactoryPortTypeSOAPBindingStub.java:1191)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub$$FastClassByCGLIB$$23292bb5.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at
org.gcube.common.core.contexts.GCUBERemotePortTypeContext$GCUBEMethodInterceptor.intercept(GCUBERemotePortTypeContext.java:382)
at
org.gcube.informationsystem.registry.stubs.bindings.RegistryFactoryPortTypeSOAPBindingStub$$EnhancerByCGLIB$$4b03b469.updateResource(<generated>)
at
org.gcube.common.informationsystem.publisher.impl.tasks.ISRegistryServiceUpdaterHandler.interact(Unknown
Source)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler._interact(GCUBEServiceHandler.java:164)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler.tryInstances(GCUBEServiceHandler.java:194)
at
org.gcube.common.core.utils.handlers.GCUBEServiceHandler.run(GCUBEServiceHandler.java:112)
at
org.gcube.common.informationsystem.publisher.impl.GCUBEProfileManager.update(Unknown
Source)
at
org.gcube.common.informationsystem.publisher.impl.GCUBEPublisher.updateGCUBEResource(Unknown
Source)
at
org.gcube.common.vremanagement.ghnmanager.impl.contexts.ServiceContext$1.onGHNUpdated(ServiceContext.java:194)
at
org.gcube.common.core.contexts.ghn.GHNConsumer.onEvent(GHNConsumer.java:21)
at
org.gcube.common.core.utils.events.GCUBEProducer$1.run(GCUBEProducer.java:172)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:520)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.components.net.DefaultSocketFactory.create(DefaultSocketFactory.java:153)
at
org.apache.axis.components.net.DefaultSocketFactory.create(DefaultSocketFactory.java:120)
at
org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:181)
at
org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:397)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:135)
