On 10/3/2012 4:13 PM, Tobias Wunden wrote: > Hi Greg, > > the main problem with this is that now the files referenced in the > mediapackage will contain urls to an arbitrary number of working file > repositories. In addition, your setup leads to lots of unnecessary downloads > from the working file repositories to the node's loca workspace.
Yep. This was mainly a cautionary tale, and perhaps a prod for better instructions on how to setup shared storage. My 'solution', if you can call it that, it to delete the workspace when it fills up. It's not a good solution, but I know I don't need to reprocess things later, and I have the originals archived elsewhere. > With the latest 1.4 and its archive implementation, this problem will be gone > because the working file repository will no longer be a relevant as a > long-term storage for mediapackage elements. Instead, it will become the > short term working storage that it has been designed for. However in 1.3, I > don't see how you would go about changing the mediapackage element's > references that are being stored in the workflows, the episode service and > last but not least search. You could be mapping all DNS names to one single > host, but that would not help you get rid of the downloading between > workspace and working file repository. Maybe someone else has an idea? This was with a trunk core, so technically it is a 1.4 install :) Another solution I explored was using a program to find identical files and then replace them with hardlinks. This would have recovered about two thirds of the disk space that was wasted. G > Tobias > > On 29.09.2012, at 01:04, Greg Logan <[email protected]> wrote: > >> Hi folks, >> >> I have a cluster here where I failed to follow the instructions when >> setting it up. Basically, I left org.opencastproject.file.repo.url key >> commented out, which resulted in each node of the cluster creating its >> own set of files on the NFS share. Does anyone have any experience >> deduplicating this type of situation? >> >> I copied the configuration of the cluster on a different machine, and >> from my experiments a properly configured cluster will only consume ~93 >> MB for a single demo capture, whereas a misconfigured system consumes >> ~763 MB! >> >> Does anyone have experience with this type of issue? How did you solve it? >> >> G >> >> _______________________________________________ >> Matterhorn-users mailing list >> [email protected] >> http://lists.opencastproject.org/mailman/listinfo/matterhorn-users > > _______________________________________________ > Matterhorn-users mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn-users > _______________________________________________ Matterhorn-users mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
