Thanks Chris. You provided a good clue. We have a NAS appliance providing shared NFS storage and so I'm trying to figure out what has gone wrong. Given my past history troubleshooting NFS problems, this will take awhile. So thanks for the interpretation.
On Fri, Mar 9, 2012 at 8:47 AM, Christopher Brooks <[email protected]> wrote: > On quick glance it looks to me that the working file repository is > trying to save a file but can't because jetty is timing out. Maybe the > file is particularly large? > > Maybe someone on list can advise on how to change the timeout size for > jetty? > > Chris > > On Fri, 09 Mar 2012 08:40:52 -0700 Jack > Vant <[email protected]> wrote: > >> We've been running version 1.2 since early January with no problems >> until late last week when captures suddenly started failing with this: >> >> 2012-03-01 13:00:09 ERROR (AbstractFaultChainInitiatorObserver:101) - >> Error occurred during error handling, gi >> ve up! >> >> I'll paste the rest of the message below. Just wondering what this >> means. I have since stopped matterhorn, cleaned out the cache >> directory and restarted the admin and worker servers, so the latest >> capture now fails with a text extraction error which is another issue. >> Here's the rest of the error message. If some java expert could >> 'splain AbstractFaultChainInitiatorObserver:101, I would sure >> appreciate it. >> >> Here's the rest of the error: >> >> org.apache.cxf.interceptor.Fault: org.mortbay.jetty.EofException: >> timeout at >> org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155) >> at >> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121) >> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:133) at >> org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:82) at >> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) at >> org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) >> at >> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:1 >> 06) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) >> at >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:110) >> at >> org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:98) >> at >> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:423) >> at >> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:139) >> at >> org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:142) >> at >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179) >> at >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159) >> at >> org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:91) >> at >> org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:77) >> at >> org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42) >> at >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j >> ava:49) at >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :343) >> at >> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityIn >> terceptor.java:109) at >> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurity >> Interceptor.java:83) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFil >> ter.java:97) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.j >> ava:100) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAut >> henticationFilter.java:78) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter( >> RememberMeAuthenticationFilter.java:119) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(Securi >> tyContextHolderAwareRequestFilter.java:54) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFil >> ter.java:35) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderPro >> cessingFilter.java:173) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthe >> nticationFilter.java:319) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(Defaul >> tLoginPageGeneratingFilter.java:90) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(Abs >> tractAuthenticationProcessingFilter.java:188) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextP >> ersistenceFilter.java:79) at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java >> :355) >> at >> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149) >> at >> org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:115) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) >> at >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j >> ava:47) at >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) >> at >> org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:94) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) >> at >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j >> ava:47) at >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) >> at >> org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:106) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) >> at >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) >> at >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j >> ava:47) at >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) >> at >> org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48) >> at >> org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39) >> at >> org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:55) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) >> at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) >> at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> at org.mortbay.jetty.Server.handle(Server.java:326) at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536) >> at >> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930) >> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:834) at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405) at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >> Caused by: java.lang.RuntimeException: >> org.mortbay.jetty.EofException: timeout at >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryImpl.putInCollection(WorkingFil >> eRepositoryImpl.java:444) at >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryRestEndpoint.restPutInCollectio >> n(WorkingFileRepositoryRestEndpoint.java:141) at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) at >> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) >> at >> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) >> ... >> 84 more Caused by: org.mortbay.jetty.EofException: timeout at >> org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:1164) >> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:1114) at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:97 >> 7) at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) >> at java.io.InputStream.read(InputStream.java:85) at >> org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025) at >> org.apache.commons.io.IOUtils.copy(IOUtils.java:999) at >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryImpl.putInCollection(WorkingFil >> eRepositoryImpl.java:442) ... 91 more 2012-03-01 13:00:24 INFO >> (SchedulerServiceSolrIndex:681) - Solr query: dc_spatial:mpcf201 >> 2012-03-01 13:00:40 INFO (SchedulerServiceSolrIndex:681) - Solr >> query: dc_spatial:mpcf106 2012-03-01 13:00:48 ERROR >> (WorkflowOperationWorker:156) - Workflow operation 'prepare-av' >> failed org.opencastproject.workflow.api.WorkflowOperationException: >> Muxing video track >> http://feather1.boisestate.edu >> :8080/files/mediapackage/5458/track-2/windows.mpg >> and audio track http://feather1.boisestate.edu:8080/files/me >> diapackage/5458/track-3/audio.mp2 failed at >> org.opencastproject.workflow.handler.PrepareAVWorkflowOperationHandler.mux(PrepareAVWorkflowOperati >> onHandler.java:263) at >> org.opencastproject.workflow.handler.PrepareAVWorkflowOperationHandler.start(PrepareAVWorkflowOpera >> tionHandler.java:130) at >> org.opencastproject.workflow.impl.WorkflowOperationWorker.start(WorkflowOperationWorker.java:216) >> at >> org.opencastproject.workflow.impl.WorkflowOperationWorker.execute(WorkflowOperationWorker.java:138) >> at >> org.opencastproject.workflow.impl.WorkflowServiceImpl.runWorkflowOperation(WorkflowServiceImpl.java >> :704) >> at >> org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1412) >> at >> org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:1717) >> at >> org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:1690) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) 2012-03-01 13:00:53 INFO >> (ZipWorkflowOperationHandler:158) - Archiving mediapackage >> 8c477517-df4b-4f8f-a9a5-3 53963bf4a8d in workflow Workflow {5458} >> >> >> > > > > -- > Christopher Brooks, BSc, MSc > ARIES Laboratory, University of Saskatchewan > > Web: http://www.cs.usask.ca/~cab938 > Phone: 1.306.966.1442 > Mail: Advanced Research in Intelligent Educational Systems Laboratory > Department of Computer Science > University of Saskatchewan > 176 Thorvaldson Building > 110 Science Place > Saskatoon, SK > S7N 5C9 > _______________________________________________ > Matterhorn-users mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn-users -- Jack Vant System Engineer - Unix Office of Information Technology Boise State University 208-426-4446 208-863-0031 _______________________________________________ Matterhorn-users mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
