Hi All,
Our Grid, which has been functioning perfectly well, seems to have broken
now all of a sudden in the sense that all GRAM job submissions from host A
to host B, C, D, etc (where B C D are independent remote sites) are failing
with the following error:
2007-11-05 16:54:38,950 INFO
exec.StateMachine[RunQueueThread_0,logJobAccepted:3193] Job
b347a4d0-8be9-11dc-a3ac-ba9ad65cba4d accepted for local user 'globus'
2007-11-05 16:54:39,457 ERROR
exec.StateMachine[RunQueueThread_0,fileCleanUp:2730] A secondary fault
occured while trying
to gracefully fail.
AxisFault
faultCode: {
http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}General
faultSubcode:
faultString: [Caused by: Conversion failed [Caused by: java.io.IOException:
Not a UTC_TIME]]
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:. Caused by Conversion
failed. Caused by COM.claymoresystems.cert.CertificateDecodeException:
java.io.IOException: Not a UTC_TIME
at COM.claymoresystems.cert.X509Cert.<init>(
X509Cert.java:209)
at org.globus.gsi.ptls.PureTLSUtil.certificateChainToVector(
PureTLSUtil.java:226)
at org.globus.gsi.proxy.ProxyPathValidator.validate(
ProxyPathValidator.java:216)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.verifyXMLSignature
(WSSecurityEngine.java:299)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.verifyXMLSignature
(WSSecurityRequestEngine.java:97)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.handleSignatureElement
(WSSecurityEngine.java:116)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader
(WSSecurityEngine.java:516)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader
(WSSecurityEngine.java:482)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader
(WSSecurityEngine.java:397)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.processSecurityHeader
(WSSecurityRequestEngine.java:61)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityBasicHandler.handleMessage
(WSSecurityBasicHandler.java:43)
at
org.globus.wsrf.impl.security.authentication.wssec.WSSecurityHandler.handleRequest
(WSSecurityHandler.java:21)
at org.apache.axis.handlers.HandlerChainImpl.handleRequest(
HandlerChainImpl.java:105)
at org.apache.axis.handlers.JAXRPCHandler.invoke(JAXRPCHandler.java
:52)
at org.globus.wsrf.handlers.JAXRPCHandler.invoke(JAXRPCHandler.java
:26)
at org.apache.axis.strategies.InvocationStrategy.visit(
InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:248)
at org.apache.axis.transport.local.LocalSender.invoke(
LocalSender.java:141)
at org.apache.axis.strategies.InvocationStrategy.visit(
InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
at org.apache.axis.client.Call.invoke(Call.java:2710)
at org.apache.axis.client.Call.invoke(Call.java:2386)
at org.apache.axis.client.Call.invoke(Call.java:2309)
at org.apache.axis.client.Call.invoke(Call.java:1766)
at
org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer
(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
at
org.globus.exec.service.exec.utils.StagingHelper.submitStagingRequest(
StagingHelper.java:168)
at org.globus.exec.service.exec.StateMachine.fileCleanUp(
StateMachine.java:2716)
at
org.globus.exec.service.exec.StateMachine.processFailureFileCleanUpState(
StateMachine.java:2091)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:302)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
{http://xml.apache.org/axis/}hostname:seil.umd.edu
[Caused by: Conversion failed [Caused by: java.io.IOException: Not a
UTC_TIME]]
at org.apache.axis.message.SOAPFaultBuilder.createFault(
SOAPFaultBuilder.java:221)
at org.apache.axis.message.SOAPFaultBuilder.endElement(
SOAPFaultBuilder.java:128)
at org.apache.axis.encoding.DeserializationContext.endElement(
DeserializationContext.java:1087)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown
Source)
at
org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at org.apache.axis.encoding.DeserializationContext.parse(
DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
at
org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse
(AddressingHandler.java:305)
at
org.apache.axis.message.addressing.handler.AddressingHandler.invoke(
AddressingHandler.java:110)
at org.apache.axis.strategies.InvocationStrategy.visit(
InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
at org.apache.axis.client.Call.invoke(Call.java:2710)
at org.apache.axis.client.Call.invoke(Call.java:2386)
at org.apache.axis.client.Call.invoke(Call.java:2309)
at org.apache.axis.client.Call.invoke(Call.java:1766)
at
org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer
(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
at
org.globus.exec.service.exec.utils.StagingHelper.submitStagingRequest(
StagingHelper.java:168)
at org.globus.exec.service.exec.StateMachine.fileCleanUp(
StateMachine.java:2716)
at
org.globus.exec.service.exec.StateMachine.processFailureFileCleanUpState(
StateMachine.java:2091)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:302)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
2007-11-05 16:54:40,046 INFO
exec.StateMachine[RunQueueThread_0,logJobFailed:3212] Job
b347a4d0-8be9-11dc-a3ac-ba9ad65cba4d failed
What is this "Not a UTC_TIME" stuff? I've checked and re-checked the certs,
they are all valid and I can't think of anything that has changed, except
daylight savings! If anyone has ideas, please let me know ASAP!
Thanks,
Adam
