How beefy is the NAS? Matterhorn is a pig with disk access. Chris
On Mon, 12 Mar 2012 10:24:20 -0600 Jack Vant <[email protected]> wrote: > Thanks Chris. You provided a good clue. We have a NAS appliance > providing shared NFS storage and so I'm trying to figure out what has > gone wrong. Given my past history troubleshooting NFS problems, this > will take awhile. So thanks for the interpretation. > > On Fri, Mar 9, 2012 at 8:47 AM, Christopher Brooks > <[email protected]> wrote: > > On quick glance it looks to me that the working file repository is > > trying to save a file but can't because jetty is timing out. Maybe > > the file is particularly large? > > > > Maybe someone on list can advise on how to change the timeout size > > for jetty? > > > > Chris > > > > On Fri, 09 Mar 2012 08:40:52 -0700 Jack > > Vant <[email protected]> wrote: > > > >> We've been running version 1.2 since early January with no problems > >> until late last week when captures suddenly started failing with > >> this: > >> > >> 2012-03-01 13:00:09 ERROR > >> (AbstractFaultChainInitiatorObserver:101) - Error occurred during > >> error handling, gi ve up! > >> > >> I'll paste the rest of the message below. Just wondering what this > >> means. I have since stopped matterhorn, cleaned out the cache > >> directory and restarted the admin and worker servers, so the latest > >> capture now fails with a text extraction error which is another > >> issue. Here's the rest of the error message. If some java expert > >> could 'splain AbstractFaultChainInitiatorObserver:101, I would sure > >> appreciate it. > >> > >> Here's the rest of the error: > >> > >> org.apache.cxf.interceptor.Fault: org.mortbay.jetty.EofException: > >> timeout at > >> org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155) > >> at > >> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121) > >> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:133) > >> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:82) > >> at > >> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) at > >> org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) > >> at > >> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:1 > >> 06) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) > >> at > >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:110) > >> at > >> org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:98) > >> at > >> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:423) > >> at > >> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:139) > >> at > >> org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:142) > >> at > >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179) > >> at > >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103) > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at > >> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159) > >> at > >> org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:91) > >> at > >> org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:77) > >> at > >> org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42) > >> at > >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j > >> ava:49) at > >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > >> at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :343) > >> at > >> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityIn > >> terceptor.java:109) at > >> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurity > >> Interceptor.java:83) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFil > >> ter.java:97) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.j > >> ava:100) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAut > >> henticationFilter.java:78) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter( > >> RememberMeAuthenticationFilter.java:119) > >> at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(Securi > >> tyContextHolderAwareRequestFilter.java:54) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFil > >> ter.java:35) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderPro > >> cessingFilter.java:173) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthe > >> nticationFilter.java:319) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(Defaul > >> tLoginPageGeneratingFilter.java:90) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(Abs > >> tractAuthenticationProcessingFilter.java:188) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) > >> at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextP > >> ersistenceFilter.java:79) at > >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java > >> :355) > >> at > >> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149) > >> at > >> org.opencastproject.kernel.security.SecurityFilter.doFilter(SecurityFilter.java:115) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) > >> at > >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j > >> ava:47) at > >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > >> at > >> org.opencastproject.kernel.security.OrganizationFilter.doFilter(OrganizationFilter.java:94) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) > >> at > >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j > >> ava:47) at > >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > >> at > >> org.opencastproject.kernel.rest.JsonpFilter.doFilter(JsonpFilter.java:106) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:80) > >> at > >> org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:68) > >> at > >> org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.j > >> ava:47) at > >> org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > >> at > >> org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48) > >> at > >> org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39) > >> at > >> org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:55) > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at > >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > >> at > >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) > >> at > >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > >> at > >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > >> at > >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > >> at org.mortbay.jetty.Server.handle(Server.java:326) at > >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536) > >> at > >> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930) > >> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:834) at > >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > >> at > >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405) > >> at > >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) > >> at > >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > >> Caused by: java.lang.RuntimeException: > >> org.mortbay.jetty.EofException: timeout at > >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryImpl.putInCollection(WorkingFil > >> eRepositoryImpl.java:444) at > >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryRestEndpoint.restPutInCollectio > >> n(WorkingFileRepositoryRestEndpoint.java:141) at > >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> at > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> at java.lang.reflect.Method.invoke(Method.java:597) at > >> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) > >> at > >> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) > >> ... > >> 84 more Caused by: org.mortbay.jetty.EofException: timeout at > >> org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:1164) > >> at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:1114) > >> at > >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:97 > >> 7) at > >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) > >> at java.io.InputStream.read(InputStream.java:85) at > >> org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025) at > >> org.apache.commons.io.IOUtils.copy(IOUtils.java:999) at > >> org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryImpl.putInCollection(WorkingFil > >> eRepositoryImpl.java:442) ... 91 more 2012-03-01 13:00:24 INFO > >> (SchedulerServiceSolrIndex:681) - Solr query: dc_spatial:mpcf201 > >> 2012-03-01 13:00:40 INFO (SchedulerServiceSolrIndex:681) - Solr > >> query: dc_spatial:mpcf106 2012-03-01 13:00:48 ERROR > >> (WorkflowOperationWorker:156) - Workflow operation 'prepare-av' > >> failed > >> org.opencastproject.workflow.api.WorkflowOperationException: > >> Muxing video track > >> http://feather1.boisestate.edu > >> :8080/files/mediapackage/5458/track-2/windows.mpg > >> and audio track http://feather1.boisestate.edu:8080/files/me > >> diapackage/5458/track-3/audio.mp2 failed at > >> org.opencastproject.workflow.handler.PrepareAVWorkflowOperationHandler.mux(PrepareAVWorkflowOperati > >> onHandler.java:263) at > >> org.opencastproject.workflow.handler.PrepareAVWorkflowOperationHandler.start(PrepareAVWorkflowOpera > >> tionHandler.java:130) at > >> org.opencastproject.workflow.impl.WorkflowOperationWorker.start(WorkflowOperationWorker.java:216) > >> at > >> org.opencastproject.workflow.impl.WorkflowOperationWorker.execute(WorkflowOperationWorker.java:138) > >> at > >> org.opencastproject.workflow.impl.WorkflowServiceImpl.runWorkflowOperation(WorkflowServiceImpl.java > >> :704) > >> at > >> org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1412) > >> at > >> org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:1717) > >> at > >> org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:1690) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) at > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:662) 2012-03-01 13:00:53 INFO > >> (ZipWorkflowOperationHandler:158) - Archiving mediapackage > >> 8c477517-df4b-4f8f-a9a5-3 53963bf4a8d in workflow Workflow {5458} > >> > >> > >> > > > > > > > > -- > > Christopher Brooks, BSc, MSc > > ARIES Laboratory, University of Saskatchewan > > > > Web: http://www.cs.usask.ca/~cab938 > > Phone: 1.306.966.1442 > > Mail: Advanced Research in Intelligent Educational Systems > > Laboratory Department of Computer Science > > University of Saskatchewan > > 176 Thorvaldson Building > > 110 Science Place > > Saskatoon, SK > > S7N 5C9 > > _______________________________________________ > > Matterhorn-users mailing list > > [email protected] > > http://lists.opencastproject.org/mailman/listinfo/matterhorn-users > > > _______________________________________________ Matterhorn-users mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
