In case anyone is interested: I solved this problem in two steps. One, I installed an older version of Condor that writes to globus-condor.log in a format the SEG will accept. That did not solve the stageout/RFT problem, however. Two, I noticed that the clock on the machine was approximately two minutes off. After correcting that, the strange RFT database error went away.
On Thu, Sep 13, 2012 at 5:06 PM, Ravi K Madduri <[email protected]> wrote: > Adam > There is post deploy script that automatically creates the derby database > for RFT. If you want to recreate it, you have first remove the directory > rftDatabase. I don't remember the name of the script that creates the > database for you of the top off my head. It is really strange that RFT does > not give a database error in step 1 below but does in step 4. As far as i > know there is no link between condor.log and RFT database setup. > On Sep 12, 2012, at 3:30 PM, Adam Bazinet wrote: > > Hi Ravi, > > I cleaned out the installation and copied over an rftDatabase folder from > a pristine 4.2.1 installation (something I've done to reset Derby/RFT in > the past). Is there another way "manually create the derby database", as > you say? > > Anyway, I tried another test and got the same results. I'll outline what I > did, though. > > 1) submitted two very short jobs to condor ... the files get staged in and > the jobs run just fine (see container.log.submission) > 2) stopped the container > 3) for one of the jobs, I went into var/globus-condor.log and rearranged > lines in the JobTerminatedEvent for one of the jobs so that upon container > restart, the SEG will detect the job is finished > 4) restarted the container ... the job I mucked around with is detected to > be 'Done', but I get the same RFT error upon stageout > (container.log.after_changing_condor_log) > > So, any idea what can be done here? If it is a straight RFT error, how can > I get more information about it? > > thanks, > Adam > > > > On Wed, Sep 12, 2012 at 10:11 AM, Ravi K Madduri <[email protected]>wrote: > >> Adam >> I can't think of any reason why messing with condor log file will affect >> RFT. Can you, may be, manually create the derby database and restart the >> container? Or alternatively, you can create a mySQL database (instructions >> are here: >> http://www.globus.org/toolkit/docs/4.2/4.2.0/data/rft/admin/rft-admin-mysql.html >> ) >> >> On Sep 11, 2012, at 5:23 PM, Adam Bazinet wrote: >> >> Hi Ravi, >> >> Just the default Derby database, which has worked fine on all of our >> other installations. Again, I'm leaning towards something weird about what >> I was doing messing with the Condor log file, and maybe something gets out >> of sync, I don't know; the other possibility, of course, is that something >> with RFT isn't configured properly and the other problem just happened to >> expose it. If you know of anything I can do to debug, please let me know. >> >> thanks, >> Adam >> >> >> >> On Tue, Sep 11, 2012 at 6:19 PM, Ravi K Madduri <[email protected]>wrote: >> >>> Adam >>> What database do you use for RFT? Is it Postgresql or derby? >>> >>> On Sep 11, 2012, at 5:08 PM, Adam Bazinet wrote: >>> >>> Sorry, forgot to attach the log: >>> >>> ---------------------------------------- >>> PROCESSING INTERNAL STATE: -- StageOut -- >>> ---------------------------------------- >>> 2012-09-11T17:49:48.478-04:00 DEBUG processing.StateMachine >>> [pool-1-thread-1,processInternalState:115] Processing resource >>> d69fb7c0-fc45-11e1-93da-f931d11c3d83 in internal state StageOut >>> 2012-09-11T17:49:48.478-04:00 DEBUG handler.InternalStateHandler >>> [pool-1-thread-1,processInternalState:44] >>> [resourceKey:d69fb7c0-fc45-11e1-93da-f931d11c3d83] Start processing >>> internal state StageOut >>> 2012-09-11T17:49:48.484-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:846] receiving request for state change to >>> StageOut >>> 2012-09-11T17:49:48.484-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:525] Setting new state RP value StageOut >>> 2012-09-11T17:49:48.484-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:854] State of job { >>> http://www.globus.org/namespaces/2008/03/gram/job}ResourceID=d69fb7c0-fc45-11e1-93da-f931d11c3d83changed >>> to: StageOut >>> 2012-09-11T17:49:48.484-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:862] Holding: false >>> 2012-09-11T17:49:48.484-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:896] exitCode is null >>> 2012-09-11T17:49:48.485-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,setState:905] Notifying of job state change to topic >>> listeners >>> 2012-09-11T17:49:48.549-04:00 DEBUG utils.LocalStagingHelper >>> [pool-1-thread-1,submitStagingRequest:72] >>> [resourceKey:d69fb7c0-fc45-11e1-93da-f931d11c3d83] Entering >>> sumbitStagingRequest() >>> 2012-09-11T17:49:48.552-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,getStagingCredential:434] getStagingCredential() called >>> 2012-09-11T17:49:48.552-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,getResourceDatum:209] getting resource datum gridMapFile >>> 2012-09-11T17:49:48.554-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,getStagingCredential:453] staging credential endpoint: >>> Address: https://128.8.91.70:8443/wsrf/services/DelegationService >>> Reference property[0]: >>> <ns1:DelegationKey xmlns:ns1=" >>> http://www.globus.org/08/2004/delegationService >>> ">d61f6430-fc45-11e1-93da-f931d11c3d83</ns1:DelegationKey> >>> >>> 2012-09-11T17:49:48.554-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getDelegatedCredential:98] Entering >>> getDelegatedCredential() >>> 2012-09-11T17:49:48.554-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getDelegatedCredential:112] checking for existing >>> credential listener >>> 2012-09-11T17:49:48.555-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getDelegationKey:605] Pulled out DelegationKey: >>> d61f6430-fc45-11e1-93da-f931d11c3d83 >>> 2012-09-11T17:49:48.555-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getDelegatedCredential:124] reusing DelegatedCredential >>> object >>> 2012-09-11T17:49:48.555-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,getStagingCredential:476] Delegated Staging Credential >>> Object: org.globus.exec.service.utils.DelegatedCredential@26b98a06 >>> 2012-09-11T17:49:48.555-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getCredential:333] waiting to receive initial credential... >>> 2012-09-11T17:49:48.556-04:00 DEBUG utils.DelegatedCredential >>> [pool-1-thread-1,getCredential:341] done waiting... >>> 2012-09-11T17:49:48.559-04:00 DEBUG >>> PersistentManagedExecutableJobResource.d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> [pool-1-thread-1,getStagingCredential:479] Using proxy with subject >>> /O=Grid/OU=GlobusTest/OU= >>> simpleCA-leucine.umiacs.umd.edu/OU=umiacs.umd.edu/CN=GT4 Adminto >>> contact the staging service >>> 2012-09-11T17:49:48.580-04:00 DEBUG utils.LocalStagingHelper >>> [pool-1-thread-1,submitStagingRequest:91] creating RFT resource >>> 2012-09-11T17:49:48.711-04:00 DEBUG >>> factory.ReliableFileTransferFactoryService [pool-1-thread-1,oldLog:168] >>> Registration to MDS enabled >>> 2012-09-11T17:49:50.981-04:00 DEBUG utils.LocalInvocationHelper >>> [pool-1-thread-1,getServiceObject:403] >>> className=org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService >>> handlerClass=org.globus.axis.providers.RPCProvider >>> 2012-09-11T17:49:50.981-04:00 DEBUG utils.LocalInvocationHelper >>> [pool-1-thread-1,getServiceObject:416] getting the service object >>> 2012-09-11T17:49:50.982-04:00 DEBUG utils.LocalInvocationHelper >>> [pool-1-thread-1,getServiceObject:422] caching service object for service >>> ReliableFileTransferFactoryService >>> 2012-09-11T17:49:50.983-04:00 DEBUG utils.LocalInvocationHelper >>> [pool-1-thread-1,getMethod:457] caching method >>> ReliableFileTransferFactoryService.createReliableFileTransfer >>> 2012-09-11T17:49:50.984-04:00 DEBUG >>> factory.ReliableFileTransferFactoryService [pool-1-thread-1,oldLog:168] >>> PerformanceLog createReliableFileTransfer() enter >>> 2012-09-11T17:49:50.984-04:00 DEBUG >>> factory.ReliableFileTransferFactoryService [pool-1-thread-1,oldLog:168] Got >>> a transfer request >>> 2012-09-11T17:49:51.049-04:00 DEBUG service.ReliableFileTransferHome >>> [pool-1-thread-1,oldLog:168] Loading the RFT home's list of resource keys >>> 2012-09-11T17:49:51.051-04:00 DEBUG database.RFTDatabaseSetup >>> [pool-1-thread-1,oldLog:168] getDBConnection() enter >>> 2012-09-11T17:49:51.051-04:00 WARN service.ReliableFileTransferHome >>> [pool-1-thread-1,oldLog:190] All RFT requests will fail and all GRAM jobs >>> that require file staging will fail.Database driver is not initialized, >>> Need to setup database >>> 2012-09-11T17:49:51.052-04:00 DEBUG service.ReliableFileTransferHome >>> [pool-1-thread-1,oldLog:164] All RFT requests will fail and all GRAM jobs >>> that require file staging will fail.Database driver is not initialized, >>> Need to setup database >>> org.globus.transfer.reliable.service.database.RftDBException: Database >>> driver is not initialized, Need to setup database >>> at >>> org.globus.transfer.reliable.service.database.RFTDatabaseSetup.getDBConnection(RFTDatabaseSetup.java:249) >>> at >>> org.globus.transfer.reliable.service.database.ReliableFileTransferDbAdapter.getActiveRequestIds(ReliableFileTransferDbAdapter.java:917) >>> at >>> org.globus.transfer.reliable.service.ReliableFileTransferResource.getStoredResourceKeyValues(ReliableFileTransferResource.java:544) >>> at >>> org.globus.transfer.reliable.service.ReliableFileTransferHome.initialize(ReliableFileTransferHome.java:69) >>> at >>> org.globus.wsrf.jndi.BasicBeanFactory.getObjectInstance(BasicBeanFactory.java:54) >>> at >>> org.globus.wsrf.jndi.BeanFactory.getInstance(BeanFactory.java:118) >>> at >>> org.globus.wsrf.jndi.BeanFactory.access$100(BeanFactory.java:40) >>> at >>> org.globus.wsrf.jndi.BeanFactory$GetInstanceAction.run(BeanFactory.java:142) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:416) >>> at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:60) >>> at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:100) >>> at >>> org.globus.wsrf.jndi.BeanFactory.getObjectInstance(BeanFactory.java:90) >>> at >>> org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:135) >>> at >>> javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:827) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:155) >>> at >>> org.apache.naming.SynchronizedContext.lookup(SynchronizedContext.java:69) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:815) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:155) >>> at >>> org.apache.naming.SynchronizedContext.lookup(SynchronizedContext.java:69) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:815) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:155) >>> at >>> org.apache.naming.SynchronizedContext.lookup(SynchronizedContext.java:69) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:815) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:155) >>> at >>> org.apache.naming.SynchronizedContext.lookup(SynchronizedContext.java:69) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:815) >>> at org.apache.naming.NamingContext.lookup(NamingContext.java:168) >>> at >>> org.apache.naming.SynchronizedContext.lookup(SynchronizedContext.java:81) >>> at javax.naming.InitialContext.lookup(InitialContext.java:409) >>> at >>> org.apache.naming.SelectorContext.lookup(SelectorContext.java:145) >>> at javax.naming.InitialContext.lookup(InitialContext.java:409) >>> at >>> org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:247) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:616) >>> at >>> org.globus.exec.service.exec.utils.LocalInvocationHelper.callService(LocalInvocationHelper.java:140) >>> at >>> org.globus.exec.service.exec.utils.LocalStagingHelper.submitStagingRequest(LocalStagingHelper.java:95) >>> at >>> org.globus.exec.service.exec.processing.handler.StageOutStateHandler.process(StageOutStateHandler.java:79) >>> at >>> org.globus.exec.service.exec.processing.handler.InternalStateHandler.processInternalState(InternalStateHandler.java:49) >>> at >>> org.globus.exec.service.exec.processing.StateMachine.processInternalState(StateMachine.java:121) >>> at >>> org.globus.exec.service.exec.processing.StateProcessingTask.run(StateProcessingTask.java:82) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:679) >>> 2012-09-11T17:49:51.062-04:00 DEBUG database.RFTDatabaseSetup >>> [pool-1-thread-1,oldLog:168] getDBConnection() enter >>> 2012-09-11T17:49:51.062-04:00 ERROR >>> factory.ReliableFileTransferFactoryService [pool-1-thread-1,oldLog:175] >>> Unable to create RFT resource >>> org.globus.transfer.reliable.service.database.RftDBException: Database >>> driver is not initialized, Need to setup database >>> at >>> org.globus.transfer.reliable.service.database.RFTDatabaseSetup.getDBConnection(RFTDatabaseSetup.java:249) >>> at >>> org.globus.transfer.reliable.service.database.ReliableFileTransferDbAdapter.storeTransferRequest(ReliableFileTransferDbAdapter.java:204) >>> at >>> org.globus.transfer.reliable.service.ReliableFileTransferResource.<init>(ReliableFileTransferResource.java:191) >>> at >>> org.globus.transfer.reliable.service.ReliableFileTransferHome.create(ReliableFileTransferHome.java:126) >>> at >>> org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:249) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:616) >>> at >>> org.globus.exec.service.exec.utils.LocalInvocationHelper.callService(LocalInvocationHelper.java:140) >>> at >>> org.globus.exec.service.exec.utils.LocalStagingHelper.submitStagingRequest(LocalStagingHelper.java:95) >>> at >>> org.globus.exec.service.exec.processing.handler.StageOutStateHandler.process(StageOutStateHandler.java:79) >>> at >>> org.globus.exec.service.exec.processing.handler.InternalStateHandler.processInternalState(InternalStateHandler.java:49) >>> at >>> org.globus.exec.service.exec.processing.StateMachine.processInternalState(StateMachine.java:121) >>> at >>> org.globus.exec.service.exec.processing.StateProcessingTask.run(StateProcessingTask.java:82) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:679) >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:541] Fault Class: class >>> org.globus.exec.generated.StagingFaultType >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:542] Resource Key: { >>> http://www.globus.org/namespaces/2008/03/gram/job}ResourceID=d69fb7c0-fc45-11e1-93da-f931d11c3d83 >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:543] Description: Staging error for RSL element >>> fileStageOut. >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:545] Cause: java.rmi.RemoteException >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:549] State when failure occurred: StageOut >>> 2012-09-11T17:49:51.070-04:00 DEBUG utils.FaultUtils >>> [pool-1-thread-1,makeFault:551] Script Command: StageOut >>> >>> >>> On Tue, Sep 11, 2012 at 6:05 PM, Adam Bazinet >>> <[email protected]>wrote: >>> >>>> Dear list and particularly GT 4.x and Condor GRAM users, >>>> >>>> To those of you who may be interested, I have confirmed that the Condor >>>> SEG is sensitive to the order and formatting of the elements in >>>> globus-condor.log. If I shift them around manually so that they conform to >>>> what a <c> block looks like with an older version of Condor, Globus >>>> properly picks up the fact that my jobs are complete. However, as it goes >>>> to stage out the files, I get an RFT failure saying the "Database driver is >>>> not initialized, Need to setup database". My question is: do I have >>>> something wrong with RFT in this Globus installation? This seems unlikely >>>> since I have several others configured the same way without any issue, and >>>> there are no problems staging files *in*. Does it have to do with my >>>> manually mucking around with the globus-condor.log file? A relevant bit of >>>> the container.log is below. I'd appreciate your input because regarding >>>> this and also a possible workaround for the SEG parsing problem (Condor >>>> version downgrade, modifications to SEG parsing code, or otherwise). >>>> >>>> thanks, >>>> Adam >>>> >>>> >>>> >>>> On Thu, Aug 30, 2012 at 4:09 PM, Adam Bazinet >>>> <[email protected]>wrote: >>>> >>>>> Hi all, >>>>> >>>>> I just installed a new GT 4.2.1 on a Condor resource, and while jobs >>>>> run and complete in Condor fine, the finished status is not being picked >>>>> up >>>>> by Globus appropriately. >>>>> >>>>> Is the GT 4.2.1 sensitive to the order/formatting of the elements in >>>>> globus-condor.log? >>>>> >>>>> For example, here is a JobTerminatedEvent from globus-condor.log from >>>>> an older Condor/Globus installation that works properly: >>>>> >>>>> <c> >>>>> <a n="MyType"><s>JobTerminatedEvent</s></a> >>>>> <a n="EventTypeNumber"><i>5</i></a> >>>>> <a n="MyType"><s>JobTerminatedEvent</s></a> >>>>> <a n="EventTime"><s>2012-08-29T00:32:04</s></a> >>>>> <a n="Cluster"><i>18720</i></a> >>>>> <a n="Proc"><i>36</i></a> >>>>> <a n="Subproc"><i>0</i></a> >>>>> <a n="TerminatedNormally"><b v="t"/></a> >>>>> <a n="ReturnValue"><i>0</i></a> >>>>> <a n="RunLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="RunRemoteUsage"><s>Usr 0 07:01:33, Sys 0 00:00:13</s></a> >>>>> <a n="TotalLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="TotalRemoteUsage"><s>Usr 0 07:01:33, Sys 0 00:00:13</s></a> >>>>> <a n="SentBytes"><r>3.244000000000000E+04</r></a> >>>>> <a n="ReceivedBytes"><r>4.281464000000000E+06</r></a> >>>>> <a n="TotalSentBytes"><r>3.244000000000000E+04</r></a> >>>>> <a n="TotalReceivedBytes"><r>5.565903200000000E+07</r></a> >>>>> </c> >>>>> >>>>> and here is the one from the new resource that is NOT working properly: >>>>> >>>>> <c> >>>>> <a n="MyType"><s>JobTerminatedEvent</s></a> >>>>> <a n="TotalLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="Proc"><i>0</i></a> >>>>> <a n="EventTime"><s>2012-08-28T16:07:17</s></a> >>>>> <a n="TotalRemoteUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="TotalReceivedBytes"><r>2.231476000000000E+06</r></a> >>>>> <a n="ReturnValue"><i>0</i></a> >>>>> <a n="RunRemoteUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="RunLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a> >>>>> <a n="SentBytes"><r>3.979500000000000E+04</r></a> >>>>> <a n="Cluster"><i>20</i></a> >>>>> <a n="TotalSentBytes"><r>3.979500000000000E+04</r></a> >>>>> <a n="Subproc"><i>0</i></a> >>>>> <a n="CurrentTime"><e>time()</e></a> >>>>> <a n="EventTypeNumber"><i>5</i></a> >>>>> <a n="ReceivedBytes"><r>2.231476000000000E+06</r></a> >>>>> <a n="TerminatedNormally"><b v="t"/></a> >>>>> </c> >>>>> >>>>> Basically, the only variable that's different about this resource is >>>>> that it's running a newer version of Condor. My hunch is that broke >>>>> compatibility somewhere along the way. Can someone confirm this, or >>>>> provide >>>>> another mechanism to debug? I'm attaching the container.log from the >>>>> resource in question, which has GRAM debugging enabled as some jobs came >>>>> in. It didn't really show me anything, though. >>>>> >>>>> thanks, >>>>> Adam >>>>> >>>>> >>>> >>> >>> --Ravi >>> >>> >> >> --Ravi >> >> > <container.log.submission><container.log.after_changing_condor_log> > > > --Ravi > >
