Josh and Ruben, Thank you for your replies. Regarding ./files and ./workspace , yes, they are on the same NFS mounted volume. From what I've read, hard links do not work on directories, so I'm not sure how this is accomplished, but this is what the Matterhorn startup log says on the Worker server:
2010-12-31 13:47:21 INFO (RestPublisher:90) - Registered REST endpoint at /files 2010-12-31 13:47:21 INFO (WorkspaceImpl:384) - Mapping workspace to working file repository using /mnt/zfs/matterhorn/files 2010-12-31 13:47:21 INFO (WorkspaceImpl:103) - CONFIG org.opencastproject.workspace.rootdir: /mnt/zfs/matterhorn/workspace 2010-12-31 13:47:21 INFO (WorkspaceImpl:148) - *Hard links between the working file repository and the workspace enabled* Regarding ffmpeg, I thought I would try placing '-threads 0' on all instances of ffmpeg on files in the encoding directory. I then reloaded matterhorn and watched as the log showed the encoding profiles register, activate and install. However, when the Worker was activated, I received the following in the log: 2010-12-31 13:36:46 INFO (ComposerServiceImpl$1:286) - Muxing audio track track-3 and video track track-1 2010-12-31 13:36:46 INFO (AbstractCmdlineEncoderEngine:164) - Executing encoding command: /usr/local/bin/ffmpeg -strict inofficial -threads 0 -i /mnt/zfs/matterhorn/workspace/http_feather1.boisestate.edu_8080/files/mediapackage/4e91c0cf-934c-4b0c-b7cc-b8eb0ca695a6/track-3/Audio.mp2 -i /mnt/zfs/matterhorn/workspace/http_feather1.boisestate.edu_8080/files/mediapackage/4e91c0cf-934c-4b0c-b7cc-b8eb0ca695a6/track-1/Camera.mpg -sameq -shortest /mnt/zfs/matterhorn/workspace/http_feather1.boisestate.edu_8080/files/mediapackage/4e91c0cf-934c-4b0c-b7cc-b8eb0ca695a6/track-1/Camera-work.mpg 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - *Warning: not compiled with thread support, using thread emulation* 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [mp3 @ 0x77313f0]max_analyze_duration reached 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [mp3 @ 0x77313f0]Estimating duration from bitrate, this may be inaccurate 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [mpegts @ 0x77572b0]invalid dts/pts combination 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [mpegts @ 0x77572b0]max_analyze_duration reached 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [NULL @ 0x7756130]start time is not set in av_estimate_timings_from_pts 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) -* Seems stream 0 codec frame rate differs from container frame rate: 50.00 (50/1) -> 25.00 (25/1)* 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - [mpeg1video @ 0x7756780]*automatic thread number detection not supported by codec, patch welcome* 2010-12-31 13:36:46 INFO (FFmpegEncoderEngine:144) - Error while opening encoder for output stream #0.0 - maybe incorrect parameters such as bit_rate, rate, width or height 2010-12-31 13:36:46 WARN (AbstractCmdlineEncoderEngine:195) - Error while encoding audio track Audio.mp2 and video track Camera.mpg using 'mux-av.work': Encoder exited abnormally with status 1 Ruben, it looks like the codec might be problematic in regards to using multiple threads, but I am interested to know how you were able to successfully use multiple threads. Much thanks and Happy New Year, Brian On Fri, Dec 31, 2010 at 1:13 AM, Josh Holtzman <[email protected]>wrote: > On Fri, Dec 31, 2010 at 5:54 AM, Brian Bolt <[email protected]>wrote: > >> I have a distributed install running 1.0.1. The Admin and Engage servers >> each have 16 cores (Intel). The Worker server has 24 cores (AMD). My first >> one minute capture took about five minutes to process. My second capture is >> an hour long, and at 1 hour into processing it is currently encoding the >> presentation (screen) for preview. >> >> I expected that the Worker server would endure the greatest processing >> load, and 'top' shows that ffmpeg is using 95 - 110% cpu. However, when I >> run mpstat -P ALL, all processors are greater than 95% idle. Based on what >> I saw in my VM test environment, I anticipated that a 1 hour capture would >> take well under 15 minutes to process. >> > > Unless you tell ffmpeg to use mutltiple threads, it won't. I've seen > references to a "-threads [n]" option, but from what I understand, it's > codec dependent. All of those cores you've got in your worker will help > you process more videos concurrently, but they won't help you process *one* > video any faster than a single core machine. > > We're actively investigating gstreamer as a replacement for ffmpeg, which > should enable multithreaded encoding. You can follow the discussion on the > matterhorn (dev) list. > > >> On another note, the amount of space consumed is very large, 345 MB for >> the one minute capture an 15.6 GB for the one hour capture. Note that I >> don't yet have the workflow yet configured for streaming, so I anticipate >> that these numbers would be even larger if streaming were in the mix. >> >> [r...@server matterhorn]# du -h --max-depth=1 >> 25M ./streams >> 8.8G ./files >> 6.5G ./workspace >> 68K ./server1 >> 2.5K ./inbox >> 78K ./searchindex >> 22M ./downloads >> 16G . >> > > Are ./workspace and ./files on the same volume? If so, the 6.5G in > workspace should be hard links, and therefore these won't actually consume > space on the drive. Of the 8.8G in ./files, do you have any idea how large > the source files are? If you're capturing large source files, there's not > much you can do to save space here. If the problem is that there are > multiple copies of the same file, you can tweak the workflow (e.g. remove > unneeded encoding formats such as AVI) to reduce the storage requirements. > > Josh > > >> >> Thoughts and ideas are greatly appreciated. >> >> Thanks, >> Brian >> >> > > _______________________________________________ > Matterhorn-users mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn-users > >
_______________________________________________ Matterhorn-users mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
