Matterhorn runs all distribution jobs for a single recording/workflow in parallel. With 250+ objects to distribute, I could see this being a problem without a sufficiently large cluster of servers capable of running distribution jobs.
If there is no configuration option available to limit the number of concurrent distribution jobs, there probably should be. At least there should be a sensible default limit. Apparently, infinite is not a sensible default :) I'd file this in Jira. Josh On Oct 11, 2011, at 12:47 PM, [email protected] wrote: > currently we watch the distribution of a processed video (standard full > workflow) with playtime 2-3 hours and > 250 player-slides in it fail with an > extremly high JVM MemoryPool peak - it suddenly rises in the moment of > distribution with a steep slope, and usually fails the processing with an > OutofMemory error. > > Is this the mentioned bug with Rest-calls not getting garbage collected? I'm > running out of ideas what to do. > > LG, Andreas > > Walter Schwarz schrieb am Mon, 10 Oct 2011 betreff "Re: > [Matterhorn-users]...": >> I've seen that behavior a few times, in my case the error I see reported >> is something like "destination XXXXX already exists". I've found the fix >> to be a manual cleanup of the previous failure. This means deleting the >> failed media package from /files/mediapackage and /workspace/mediapackage. >> With these two items cleaned up I've found reingesting a media package >> doesn't have these failures. >> >> It seems like a better way for matterhorn to handle this situation is to >> simply overwrite the old file, I've put this in as a bug but it is a low >> priority: http://opencast.jira.com/browse/MH-8007 >> >>> [email protected] >>> >>> Hi, >>> >>> since our lectures (w2-3 hours with like > 200 slides) fail in the >>> last step of the of the "Encode, Analyze, Distribute" workflow, I need >>> to understand: what are the steps required to perform this last step >>> manually, ie. copy files/call REST-points to distribute the correctly >>> created media? >>> >>> Any input highly appreciated, Andreas >>> >>> >>> [email protected] schrieb am Thu, 6 Oct 2011 betreff >> "Re:...": >>>> since we are seem to run into this issue, and crashes the processing >> of the >>>> recordings (after doing fine for 7 hours), I'd be interrested if this >> issue >>>> is resolved, or the causes detected or some setting/workaround >> available >>>> >>>> thx, Andreas >>>> >>>> Christopher Brooks schrieb am Fri, 30 Sep 2011 betreff "Re: >>>> [Matterhorn-use...": >>>>> Hi Walter, >>>>> >>>>> We're also getting a bit of a memory issue it seems: >>>>> >>>>> 2011-09-30 09:54:01 WARN (ServiceRegistryJpaImpl$JobDispatcher:1468) >> - >>>>> Error dispatching jobs >>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>> >>>>> I'm eyeballing calendaring right now, but seems we need a bug for >> this >>>>> issue specifically and try and do some profiling... >>>>> >>>>> Chris >>>>> >>>>> On Fri, 30 Sep 2011 >>>>> 08:20:03 -0500 Walter Schwarz <[email protected]> wrote: >>>>> >>>>>> We've got a different environment but the same symptom, our details >>>>>> might be useful. >>>>>> >>>>>> RHEL, kernel 2.6.18-238.19.1.el5.x86_64 >>>>>> java version "1.6.0_26" >>>>>> Matterhorn version 1.1.x from about a month ago with some updates >>>>>> since then. >>>>>> >>>>>> In our case there is a clear error: java.lang.OutOfMemoryError: Java >>>>>> heap space. >>>>>> We currently have 2048m configured for our java heap space, >>>>>> increasing or decreasing that limit lengthens or shortens the time >>>>>> the server can run without crashing. With 2048m it can last about >> 36 >>>>>> hours before a crash. >>>>>> >>>>>> My workaround has been a nightly restart of matterhorn on admin. I >>>>>> don't like that solution but it is the best option available for me >>>>>> right now. >>>>>> >>>>>>> Christopher Brooks <[email protected]> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We're having lots of troubles with our admin server not staying >>>>>>> responsive (1.2, 10 agents, maybe 500 entries in the calendars >>>>>>> total). Once it's been running for a couple of days java eats up >>>>>>> 100% cpu and the UI gets really sluggish. Let it go for a bit more >>>>>>> and calls to rest endpoints end up timing out (e.g. getting a >>>>>>> calendar times out). >>>>>>> >>>>>>> Anyone else experiencing this? >>>>>>> >>>>>>> Right now we are just restarting the admin machine every once and >>>>>>> awhile, but this is not awesome, and might be the source of a >>>>>>> couple of our other problems. >>>>>>> >>>>>>> RHEL, kernel 2.6.32-131.12.1.el6.x86_64 >>>>>>> java version "1.6.0_26" >>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_26-b03) >>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) >>>>>>> >>>>>>> Chris >>>>>>> -- >>>>> >>>>> >>>>> >>>>> -- >>>>> Christopher Brooks, BSc, MSc >>>>> ARIES Laboratory, University of Saskatchewan >>>>> >>>>> Web: http://www.cs.usask.ca/~cab938 >>>>> Phone: 1.306.966.1442 >>>>> Mail: Advanced Research in Intelligent Educational Systems Laboratory >>>>> Department of Computer Science >>>>> University of Saskatchewan >>>>> 176 Thorvaldson Building >>>>> 110 Science Place >>>>> Saskatoon, SK >>>>> S7N 5C9 >>>>> _______________________________________________ >>>>> Matterhorn-users mailing list >>>>> [email protected] >>>>> http://lists.opencastproject.org/mailman/listinfo/matterhorn-users >>>>> >>>> >>>> ----------------------- >>>> [email protected] >>>> 01/58801 DW 41523 >>>> mobil: 0664/60 588 4523 >>>> TU Wien >>>> DVR-Nummer: 0005886 >>>> ----------------------- >>>> >>> >>> ----------------------- >>> [email protected] >>> 01/58801 DW 41523 >>> mobil: 0664/60 588 4523 >>> TU Wien >>> DVR-Nummer: 0005886 >>> ----------------------- >>> _______________________________________________ >>> Matterhorn-users mailing list >>> [email protected] >>> http://lists.opencastproject.org/mailman/listinfo/matterhorn-users >> > > ----------------------- > [email protected] > 01/58801 DW 41523 > mobil: 0664/60 588 4523 > TU Wien > DVR-Nummer: 0005886 > ----------------------- > _______________________________________________ > Matterhorn-users mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn-users _______________________________________________ Matterhorn-users mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
