Hi Adam,
i can't say much about it right now, but at first glance it looks
to me that the application causes the problem. I have to add that
i don't know BOINC and probably i didn't understand all details.
Can you describe the role of hmmpfam a bit more:
* this is not the main executable, right?
* it is called by the main executable somehow under certain
conditions?
* if so: you said that hmmpfam actually should not be used at all.
in what situation could the executable call hmmpfam (i love
that word ... :-) ).
Martin
Hi,
We are experiencing a strange problem that is causing jobs to fail,
albeit
somewhat randomly and infrequently. At any given time, we may have
100
active GRAM jobs on a given resource. All of these jobs submit fine
and
there are usually no immediate failures. However, every so often, one
will
fail, and this can be days after it was submitted, with the following
error:
[EMAIL PROTECTED]:/export/grid_files/260600020.09477316738932795>
globusrun-ws -status -j jobEPR.txt
Current job state: Failed
globusrun-ws: Job failed: Invalid executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".
ProcessDied
Now, the resource we are submitting to is unique in that we are not
actually
transferring in the hmmpfam executable; that is just a dummy path,
and
our
custom BOINC job manager does not attempt to make use of it, as BOINC
executables live elsewhere. So until recently, the executable
specified
on
that path never existed, and so the error *kinda* made sense; what
didn't
make sense is why it happened randomly. In an attempt to make this
problem
go away, I now have the BOINC job manager create a dummy executable
on that
path when the job is submitted, but it doesn't look like that has
helped
because the error is still popping up. *Now* the error message
certainly
doesn't make sense if taken at face value, because that has been a
"valid
path", technically speaking, for the lifetime of the job in
question -- yet
the job still failed. If it helps, I'll attach debug output below,
though
I
wasn't able to glean any additional information from it. Does anyone
have
a
guess as to why this would happen so randomly and infrequently, or
happen
in
the first place?
This one is costing us big time because when a job fails, Globus
deletes
all
the output collected thus far, and these are large batches of work.
Thanks!
Adam
[EMAIL PROTECTED]:/export/grid_files/260600020.09477316738932795>
globusrun-ws -debug -status -j jobEPR.txt
=== REQUEST MESSAGE (length 816) (time 1206627634.506184000) ===
<ns00:Envelope
xmlns:ns00="http://schemas.xmlsoap.org/soap/
envelope/"><ns00:Header></
ns00:Header><ns00:Body><ns01:GetMultipleResourceProperties
xmlns:ns01="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd
"><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/
types">ns02:state</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/
types">ns02:holding</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns03="http://www.globus.org/namespaces/2004/10/gram/job/
faults">ns03:fault</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/types
">ns02:exitCode</ns01:ResourceProperty></
ns01:GetMultipleResourceProperties></ns00:Body></ns00:Envelope>
----------------------------------------------
=== RESPONSE MESSAGE (length 6399) (time 1206627634.546965000) ===
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
xmlns:xsd="http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"; xmlns:wsa="
http://schemas.xmlsoap.org/ws/2004/03/
addressing"><soapenv:Header><wsa:MessageID
soapenv:mustUnderstand="0">uuid:f6d14500-fc08-11dc-
a90a-8a2384ad991f</wsa:MessageID><wsa:To
soapenv:mustUnderstand="0">
http://schemas.xmlsoap.org/ws/2004/03/addressing/role/anonymous</
wsa:To><wsa:Action
soapenv:mustUnderstand="0">
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties/GetMultipleResourcePropertiesResponse
</wsa:Action><wsa:From
soapenv:mustUnderstand="0" xmlns:ns4="
http://www.globus.org/namespaces/2004/10/gram/job";><wsa:Address>
https://128.8.120.35:8443/wsrf/services/ManagedExecutableJobService</
wsa:Address><wsa:ReferenceProperties><ns4:ResourceID
xmlns:ns4="http://www.globus.org/namespaces/2004/10/gram/
job">a1253c20-fa72-11dc-a908-8a2384ad991f</ns4:ResourceID></
wsa:ReferenceProperties></wsa:From><wsa:RelatesTo
RelationshipType="wsa:Reply"
soapenv:mustUnderstand="0">uuid:f6d04560-fc08-11dc-
b962-000f1f66888a</wsa:RelatesTo></
soapenv:Header><soapenv:Body><GetMultipleResourcePropertiesResponse
xmlns="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd
"><ns1:state
xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job/
types">Failed</ns1:state><ns2:holding
xmlns:ns2="http://www.globus.org/namespaces/2004/10/gram/job/
types">false</ns2:holding><ns3:fault
xmlns:ns3="http://www.globus.org/namespaces/2004/10/gram/job/
faults"><ns3:invalidPathFault><ns4:Timestamp
xmlns:ns4="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
">2008-03-27T06:09:37.203Z</ns4:Timestamp><ns5:Originator xmlns:ns5="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><wsa:Address>
https://128.8.120.35:8443/wsrf/services/ManagedJobFactoryService</
wsa:Address><wsa:ReferenceProperties><ns6:ResourceID
xmlns:ns6="http://www.globus.org/namespaces/2004/10/gram/
job">a1253c20-fa72-11dc-a908-8a2384ad991f</ns6:ResourceID></
wsa:ReferenceProperties><wsa:ReferenceParameters/></
ns5:Originator><ns7:Description
xmlns:ns7="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
">Invalid
executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".</ns7:Description><ns8:FaultCause
xmlns:ns8="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><ns8:Timestamp>2008-03-27T06:09:37.203Z</
ns8:Timestamp><ns8:ErrorCode
dialect="http://www.globus.org/fault/stacktrace";>
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at java.lang.Class.newInstance0(Class.java:350)
at java.lang.Class.newInstance(Class.java:303)
at
org.globus.exec.utils.FaultUtils.makeFault(FaultUtils.java:485)
at org.globus.exec.utils.FaultUtils.createInvalidPathFault(
FaultUtils.java:129)
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3184)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:
85)
</ns8:ErrorCode><ns8:Description>
org.globus.exec.generated.InvalidPathFaultType</ns8:Description></
ns8:FaultCause><ns9:FaultCause
xmlns:ns9="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><ns9:Timestamp>2008-03-27T06:09:37.203Z
</ns9:Timestamp><ns9:Description>ProcessDied</
ns9:Description><ns9:FaultCause><ns9:Timestamp>2008-03-27T06:09:
37.207Z</ns9:Timestamp><ns9:ErrorCode dialect="
http://www.globus.org/fault/stacktrace";>java.lang.Exception:
ProcessDied
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3127)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:
85)
</ns9:ErrorCode><ns9:Description>java.lang.Exception
</ns9:Description></
ns9:FaultCause><ns9:FaultCause><ns9:Timestamp>2008-03-27T06:09:
37.207Z</ns9:Timestamp><ns9:ErrorCode dialect="
http://www.globus.org/fault/stacktrace";>
at
org.globus.wsrf.utils.FaultHelper.toBaseFault(FaultHelper.java
:282)
at
org.globus.exec.utils.FaultUtils.makeFault(FaultUtils.java:505)
at org.globus.exec.utils.FaultUtils.createInvalidPathFault(
FaultUtils.java:129)
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3184)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:
85)
</
ns9:ErrorCode><ns9:Description>org.oasis.wsrf.faults.BaseFaultType</
ns9:Description></ns9:FaultCause></
ns9:FaultCause><ns3:stateWhenFailureOccurred>Active</
ns3:stateWhenFailureOccurred><ns3:command>submit</
ns3:command><ns3:gt2ErrorCode>5</
ns3:gt2ErrorCode><ns3:attribute>executable</
ns3:attribute><ns3:path>/export/scratch/applications/
a5671f0138bc65dc700001aa80a3f378/hmmpfam</ns3:path></
ns3:invalidPathFault></ns3:fault><ns10:exitCode
xmlns:ns10="http://www.globus.org/namespaces/2004/10/gram/job/types
">5</ns10:exitCode></GetMultipleResourcePropertiesResponse></
soapenv:Body></soapenv:Envelope>
----------------------------------------------
Current job state: Failed
globusrun-ws: Job failed: Invalid executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".
ProcessDied
