Dear Globus Team,
I've written about this issue before, but I've encountered another thorny
case I need help with. So, all the containers in question are 4.2.x
containers. The mechanism we're using for job submission is the GramJob
Java class. In this case, the host we're submitting from is
lysine.umiacs.umd.edu (128.8.141.68), and the host we're submitting to is
ness.coppin.edu (131.118.128.31). To jump straight to the punch line for a
second, here is the error:
2009-09-24T15:18:59.029-04:00 ERROR impl.GARLIService
[ServiceThread-68,runService:265] Could not create GSBL job
java.lang.Exception: ; nested exception is:
org.globus.gsi.gssapi.auth.AuthorizationException: Mutual
authentication failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
I've got the entire error at the bottom of the email. We already have
several resources as part of our Grid (all GT 4.2.x containers), and they're
all configured similarly. We are using HostAuthorization with the GramJob
class. I realize this probably has something to with DNS, so here's how
things are currently resolving:
@lysine:> host 131.118.128.31
31.128.118.131.in-addr.arpa domain name pointer ness.coppin.edu.
31.128.118.131.in-addr.arpa domain name pointer classic.coppin.edu.
@lysine:> host ness.coppin.edu
ness.coppin.edu has address 131.118.128.31
@lysine:> host classic.coppin.edu
Host classic.coppin.edu not found: 3(NXDOMAIN) <-- not sure if this is
causing a problem or not
------------
@ness:$ host lysine.umiacs.umd.edu
lysine.umiacs.umd.edu has address 128.8.141.68
@ness:$ host 128.8.141.68
68.141.8.128.in-addr.arpa is an alias for
68.141.8.128.in-addr.umiacs.umd.edu.
68.141.8.128.in-addr.umiacs.umd.edu domain name pointer
lysine.umiacs.umd.edu.
@ness:$ host ness
ness.coppin.edu has address 131.118.128.31
@ness:$ host ness.coppin.edu
ness.coppin.edu has address 131.118.128.31
@ness:$ host 131.118.128.31
31.128.118.131.in-addr.arpa domain name pointer ness.coppin.edu.
31.128.118.131.in-addr.arpa domain name pointer classic.coppin.edu.
Also, in the script that starts the Globus container on ness,
GLOBUS_HOSTNAME=131.118.128.31.
In $GL/etc/globus_wsrf_core/server_config.wsdd, we have disableDNS=true and
logicalHost=131.118.128.31. This general configuration works on all of our
other hosts, but not this one.
I take that back, and this is the weirdest part - initially the config
worked, and then one day this error started popping up... since then,
sometimes it works, sometimes it doesn't!
What kind of transient thing (about DNS, about the network, etc.) could be
changing that would cause this problem?
Thanks so much for your ideas.
Adam
P.S. I've tried messing with all different combinations of logicalHost,
disableDNS, publishHostName, GLOBUS_HOSTNAME, the handle of the service
submitting to (whether by FQDN or IP), and nothing... no effect on the
error. Have double-checked the host certificate to make sure it's valid,
have made sure the clocks between the two hosts are in sync, etc...
2009-09-24T15:18:58.734-04:00 DEBUG GSBL.GSBLJob [ServiceThread-68,<init>:?]
GSBLJob using working dir of
'/export/grid_files/303260000.25842498147968995/'.
2009-09-24T15:18:58.734-04:00 DEBUG GSBL.GSBLJob [ServiceThread-68,<init>:?]
GSBLJob has been initialized.
2009-09-24T15:18:59.025-04:00 ERROR GSBL.GSBLJobManager
[ServiceThread-68,submit:?] Unable to submit GSBLJob:
org.globus.gsi.gssapi.auth.AuthorizationException: Mutual authentication
failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: org.globus.gsi.gssapi.auth.AuthorizationException: Mutual
authentication failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
faultActor:
faultNode:
faultDetail:
{
http://xml.apache.org/axis/}stackTrace:org.globus.gsi.gssapi.auth.AuthorizationException:
Mutual authentication failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
at
org.globus.gsi.gssapi.auth.Authorization.generateAuthorizationException(Authorization.java:54)
at
org.globus.gsi.gssapi.auth.HostAuthorization.authorize(HostAuthorization.java:97)
at
org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:156)
at
org.globus.gsi.gssapi.net.GssSocket.getInputStream(GssSocket.java:177)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:744)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at
org.apache.axis.transport.http.CommonsHTTPSender.invoke(CommonsHTTPSender.java:224)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.client.AxisClient.invokeTransport(AxisClient.java:150)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:289)
at org.apache.axis.client.Call.invokeEngine(Call.java:2838)
at org.apache.axis.client.Call.invoke(Call.java:2824)
at org.apache.axis.client.Call.invoke(Call.java:2501)
at org.apache.axis.client.Call.invoke(Call.java:2424)
at org.apache.axis.client.Call.invoke(Call.java:1835)
at
org.globus.exec.generated.bindings.ManagedJobFactoryPortTypeSOAPBindingStub.getMultipleResourceProperties(ManagedJobFactoryPortTypeSOAPBindingStub.java:1794)
at
org.globus.exec.client.GramJob.fetchDelegationFactoryEndpoints(GramJob.java:322)
at
org.globus.exec.client.GramJob.populateJobDescriptionEndpoints(GramJob.java:1298)
at org.globus.exec.client.GramJob.submit(GramJob.java:489)
at edu.umd.umiacs.cummings.GSBL.GSBLJobManager.submit(Unknown
Source)
at
edu.umd.grid.bio.garli.impl.GARLIService.runService(GARLIService.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:410)
at
org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:112)
at
org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:47)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:60)
at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:100)
at
org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:102)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:332)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
at
org.apache.axis.server.AxisServer.invokeService(AxisServer.java:234)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:375)
at
org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:930)
at
org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:667)
at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:163)
at
org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:468)
org.globus.gsi.gssapi.auth.AuthorizationException: Mutual authentication
failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
at org.apache.axis.AxisFault.makeFault(AxisFault.java:104)
at
org.apache.axis.transport.http.CommonsHTTPSender.invoke(CommonsHTTPSender.java:337)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.client.AxisClient.invokeTransport(AxisClient.java:150)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:289)
at org.apache.axis.client.Call.invokeEngine(Call.java:2838)
at org.apache.axis.client.Call.invoke(Call.java:2824)
at org.apache.axis.client.Call.invoke(Call.java:2501)
at org.apache.axis.client.Call.invoke(Call.java:2424)
at org.apache.axis.client.Call.invoke(Call.java:1835)
at
org.globus.exec.generated.bindings.ManagedJobFactoryPortTypeSOAPBindingStub.getMultipleResourceProperties(ManagedJobFactoryPortTypeSOAPBindingStub.java:1794)
at
org.globus.exec.client.GramJob.fetchDelegationFactoryEndpoints(GramJob.java:322)
at
org.globus.exec.client.GramJob.populateJobDescriptionEndpoints(GramJob.java:1298)
at org.globus.exec.client.GramJob.submit(GramJob.java:489)
at edu.umd.umiacs.cummings.GSBL.GSBLJobManager.submit(Unknown
Source)
at
edu.umd.grid.bio.garli.impl.GARLIService.runService(GARLIService.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:410)
at
org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:112)
at
org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:47)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:60)
at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:100)
at
org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:102)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:332)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
at
org.apache.axis.server.AxisServer.invokeService(AxisServer.java:234)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:375)
at
org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:930)
at
org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:667)
at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:163)
at
org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:468)
Caused by: org.globus.gsi.gssapi.auth.AuthorizationException: Mutual
authentication failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
at
org.globus.gsi.gssapi.auth.Authorization.generateAuthorizationException(Authorization.java:54)
at
org.globus.gsi.gssapi.auth.HostAuthorization.authorize(HostAuthorization.java:97)
at
org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:156)
at
org.globus.gsi.gssapi.net.GssSocket.getInputStream(GssSocket.java:177)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:744)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at
org.apache.axis.transport.http.CommonsHTTPSender.invoke(CommonsHTTPSender.java:224)
... 40 more
2009-09-24T15:18:59.029-04:00 ERROR impl.GARLIService
[ServiceThread-68,runService:265] Could not create GSBL job
java.lang.Exception: ; nested exception is:
org.globus.gsi.gssapi.auth.AuthorizationException: Mutual
authentication failed
Expected target subject name="/CN=host/131.118.128.31"
Target returned subject name="/O=Grid/OU=GlobusTest/OU=
simpleCA-ness.coppin.edu/CN=host/ness.coppin.edu"
