Stian Soiland-Reyes wrote: > On Fri, Mar 26, 2010 at 09:09, Stian Soiland-Reyes > <[email protected]> wrote: > > >> I would have to look at the code, this is a while ago for me! >> > > The method which removes completed jobs from DefaultQueueMonitor: > > private void removeCompletedJobs() { > Queue queue=daoFactory.getQueueDAO().defaultQueue(); > List<Job> completedJobs = > findQueuedJobsByStatus(Status.COMPLETE); > completedJobs.addAll(findQueuedJobsByStatus(Status.CANCELLED)); > completedJobs.addAll(findQueuedJobsByStatus(Status.FAILED)); > for (Job job : completedJobs) { > QueueEntry entry=queue.removeJob(job); > daoFactory.getQueueEntryDAO().delete(entry); > daoFactory.getJobDAO().update(job); > } > daoFactory.getQueueDAO().update(queue); > daoFactory.commit(); > } > > And Queue.removeJob(): > > public QueueEntry removeJob(Job job) { > QueueEntry removeEntry = job.getQueueEntry(); > if (removeEntry == null) { > throw new IllegalArgumentException("Unknown job " + > job); > } > job.setQueueEntry(null); > entries.remove(removeEntry); > setLastModified(); > return removeEntry; > } > > > Meaning that if the queueentry is removed through this method, it > should also have job.queueEntry set to null. > > I can see that if you use a database without transactional support, > like MySQL MyISAM, you could have two different connections running > intro trouble between the two lines: > > daoFactory.getQueueEntryDAO().delete(entry); > daoFactory.getJobDAO().update(job); > > (Note however that there is only one thread doing this database > cleanup - but there could be a second thread from the worker > requesting what jobs are on the queue) > > >
> But if you are right that you redid the tests with InnoDB (which is > transactional) and had a blank database, then something else must be > wrong.. > I dropped all tables from the DB and restarted the Remote Execution Service (RES). RES creates new tables. Before going further, I "altered" the tables. I then created a new RES admin and user account. > The ConsoleReaderThread should only be reading the job it has been > asked to execute, which should not be ready to be deleted yet. > > I also checked the creation of the jobs, where the job's queue entry > is set *before* it is added to the list of queue entries, so it should > not be visible to the DefaultQueueMonitor before that has completed > either.. > > > Sounds like a little mystery to me, but obviously there are some race > conditions involved here. > > So you submit 6 jobs at once, all from the same Taverna client? Are > these workflows the type that takes short or long time to execute? > > I had 6 jobs all launched to the RES from my taverna client on my laptop. All 6 were submitted from the same client. The jobs were identical and take about 15-30 mins to complete. I currently have the following connection info in .my.cnf. [client] max_allowed_packet = 1G connect_timeout=432000 But this has been the setting for 12-16 months. I've been hesitating to do this, but here is a more complete exception listing. Please note: it was for a slightly different test ( I was ensuring reproducibility) but resulted in the same effect. Only the specific ID in question (=2) is different. > a) catalina.out: > > Exception while instantiating the target resource. > java.lang.reflect.InvocationTargetException > at > sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:494) > at org.restlet.Finder.createResource(Finder.java:160) > at org.restlet.Finder.findTarget(Finder.java:136) > at org.restlet.Finder.handle(Finder.java:268) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Guard.accept(Guard.java:83) > at org.restlet.Guard.doHandle(Guard.java:197) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:87) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at > com.noelios.restlet.component.ComponentHelper.handle(ComponentHelper.java:118) > at org.restlet.Component.handle(Component.java:228) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:87) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at > com.noelios.restlet.application.ApplicationHelper.handle(ApplicationHelper.java:96) > at org.restlet.Application.handle(Application.java:293) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Router.handle(Router.java:444) > at org.restlet.Filter.doHandle(Filter.java:105) > at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:87) > at org.restlet.Filter.handle(Filter.java:134) > at org.restlet.Filter.doHandle(Filter.java:105) > at org.restlet.Filter.handle(Filter.java:134) > at > com.noelios.restlet.component.ComponentHelper.handle(ComponentHelper.java:118) > at org.restlet.Component.handle(Component.java:228) > at org.restlet.Server.handle(Server.java:281) > at com.noelios.restlet.ServerHelper.handle(ServerHelper.java:99) > at > com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:94) > at > com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:397) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:595) > Caused by: javax.persistence.EntityNotFoundException: Unable to find > net.sf.taverna.service.datastore.bean.QueueEntry with id 2 > at > org.hibernate.ejb.Ejb3Configuration$Ejb3EntityNotFoundDelegate.handleEntityNotFound(Ejb3Configuration.java:107) > at > org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:145) > at > org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195) > at > org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103) > at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878) > at > org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846) > at > org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557) > at org.hibernate.type.EntityType.resolve(EntityType.java:379) > at > org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:116) > at > org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842) > at org.hibernate.loader.Loader.doQuery(Loader.java:717) > at > org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224) > at org.hibernate.loader.Loader.loadEntity(Loader.java:1851) > at > org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48) > at > org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42) > at > org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3038) > at > org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395) > at > org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375) > at > org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139) > at > org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195) > at > org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103) > at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878) > at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815) > at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808) > at > org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:174) > at > net.sf.taverna.service.datastore.dao.jpa.GenericDaoImpl.read(GenericDaoImpl.java:52) > at > net.sf.taverna.service.rest.resources.AbstractJobResource.<init>(AbstractJobResource.java:19) > at > net.sf.taverna.service.rest.resources.JobReportResource.<init>(JobReportResource.java:21) > ... 74 more > Unhandled exception or error intercepted > javax.persistence.EntityNotFoundException: Unable to find > net.sf.taverna.service.datastore.bean.QueueEntry with id 2 > at > org.hibernate.ejb.Ejb3Configuration$Ejb3EntityNotFoundDelegate.handleEntityNotFound(Ejb3Configuration.java:107) > at > org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:145) > at > org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195) > at > org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103) > at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878) > at > org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846) > at > org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557) > at org.hibernate.type.EntityType.resolve(EntityType.java:379) > at > org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:116) > at > org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842) > at org.hibernate.loader.Loader.doQuery(Loader.java:717) > at > org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224) > at org.hibernate.loader.Loader.loadCollection(Loader.java:1985) > at > org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:36) > at > org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:565) > at > org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60) > > (etc) I have the full sql logs, but I need to heavily process them before sending. The sql updates are pretty large and uninteresting I'll try to do this today. --jeff -- ****************************************** Jeffrey L. Tilson, Ph.D. Senior Research Scientist Renaissance Computing Institute (RENCI) 100 Europa Drive University of North Carolina Chapel Hill, NC 27599-3455 919.445.9657 (o) 919.962.1380 (f) ***************************************** ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ taverna-hackers mailing list [email protected] Web site: http://www.taverna.org.uk Mailing lists: http://www.taverna.org.uk/taverna-mailing-lists/ Developers Guide: http://www.mygrid.org.uk/tools/developer-information
