Lin,
How do you submit the jobs, i.e. what client do you use and
what's the job description?
Please also send the complete container logfile.
Martin
Lin Wan wrote:
Dear all,
When I sumit one job to one GT 4 (4.0.5) node, I found such
exception messages in container's debug info:
2008-01-03 14:43:28,502 DEBUG
handlers.MessageLoggingHandler [ServiceThread-12,logMessage:53]
SOAPEnvelope: <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
xmlns:xsd="http://www.w3.org/2001/XMLSchema";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";><soapenv:Body><soapenv:Fault><faultcode>soapenv:Server.userException</faultcode><faultstring>java.rmi.RemoteException:
Failed to acquire resource; nested exception is:
org.globus.wsrf.InvalidResourceKeyException: Argument key is
null</faultstring><detail><ns1:stackTrace
xmlns:ns1="http://xml.apache.org/axis/";>java.rmi.RemoteException:
Failed to acquire resource; nested exception is:
org.globus.wsrf.InvalidResourceKeyException: Argument key is null
at
org.globus.wsrf.impl.properties.QueryResourcePropertiesProvider.queryResourceProperties(QueryResourcePropertiesProvider.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
at
org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:107)
at
org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:42)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:55)
at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:90)
at
org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:97)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
at
org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
at
org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:147)
at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
Caused by: org.globus.wsrf.InvalidResourceKeyException: Argument key is null
at
org.globus.mds.aggregator.impl.AggregatorServiceGroupEntryHome.find(AggregatorServiceGroupEntryHome.java:88)
at
org.globus.wsrf.impl.ResourceContextImpl.getResource(ResourceContextImpl.java:175)
at
org.globus.wsrf.impl.properties.QueryResourcePropertiesProvider.queryResourceProperties(QueryResourcePropertiesProvider.java:76)
... 23 more
</ns1:stackTrace><ns2:hostname
xmlns:ns2="http://xml.apache.org/axis/";>g3.grid.cn</ns2:hostname></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>
......
I'm more confused about such things:
When I submit the job to one Grid node on the same machine, GT
container output debug messages listed above, but the job finished
successfully. However, when I submit the job to another node on the
first machine, GT container output more exception messages besides
above listed :
2008-01-03 14:46:11,137 DEBUG handlers.MessageLoggingHandler
[RunQueueThread_3,logMessage:53] SOAPEnvelope: <?xml version="1.0"
encoding="UTF-8"?>
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
xmlns:xsd="http://www.w3.org/2001/XMLSchema";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";><soapenv:Body><soapenv:Fault><faultcode
xmlns:ns1="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";>ns1:FailedCheck</faultcode><faultstring>The
signature or decryption was invalid</faultstring><detail><stackTrace
xmlns="http://xml.apache.org/axis/";>org.globus.wsrf.impl.security.authentication.wssec.WSSecurityException:
The signature or decryption was invalid
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.verifyXMLSignature(WSSecurityEngine.java:277)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.verifyXMLSignature(WSSecurityRequestEngine.java:97)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.handleSignatureElement(WSSecurityEngine.java:116)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:516)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:482)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:397)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.processSecurityHeader(WSSecurityRequestEngine.java:61)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityBasicHandler.handleMessage(WSSecurityBasicHandler.java:43)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityHandler.handleRequest(WSSecurityHandler.java:21)
at
org.apache.axis.handlers.HandlerChainImpl.handleRequest(HandlerChainImpl.java:105)
at org.apache.axis.handlers.JAXRPCHandler.invoke(JAXRPCHandler.java:52)
at org.globus.wsrf.handlers.JAXRPCHandler.invoke(JAXRPCHandler.java:26)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:248)
at
org.apache.axis.transport.local.LocalSender.invoke(LocalSender.java:141)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
at org.apache.axis.client.Call.invoke(Call.java:2710)
at org.apache.axis.client.Call.invoke(Call.java:2386)
at org.apache.axis.client.Call.invoke(Call.java:2309)
at org.apache.axis.client.Call.invoke(Call.java:1766)
at
org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
at
org.globus.exec.service.exec.utils.WSStagingHelper.submitStagingRequest(WSStagingHelper.java:122)
at
org.globus.exec.service.exec.StateMachine.processStageOutState(StateMachine.java:1853)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
</stackTrace><ns2:hostname
xmlns:ns2="http://xml.apache.org/axis/";>g4.grid.cn</ns2:hostname></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>
...
Can anyone tell me what caused such error? Thanks in advance.
--------
Lin Wan
