We keep large objects as E datastreams (on the file system), and in BagIt. We've been just sucking it up while the checksums are validated for the E datastreams, which is agonizingly slow on SAMFS. Asger's suggestion would be really useful in those contexts. Perhaps even better would be a hook for file system data that understands BagIt...
On Tue, Oct 25, 2011 at 4:55 PM, Chris Wilper <cwil...@duraspace.org> wrote: > Hi Adam, > > I suspect you were running up against this problem, reported by Asger > a while ago: > > https://jira.duraspace.org/browse/FCREPO-752 > > Your approach sounds pretty reasonable to me. Using "E" datastreams is > often done for repositories dealing with very large files for a > variety of reasons. And storing such datastreams in a BagIt structure > seems like a good idea on its own. Perhaps at some point when > FCREPO-752 is fixed, it will be more practical to *also* store fixity > info inside the Fedora objects. > > I'm also interested in other people's opinions and experiences with > this kind of approach. > > - Chris > > On Tue, Oct 25, 2011 at 10:56 AM, Adam Wead <amsterda...@gmail.com> wrote: >> Hi all, >> >> I'd like to throw out my latest strategy for storing external video files in >> Fedora and have you all pick at it, see if I'm doing something dumb. >> >> My first attempt used external datastreams (type E) with checksums. This >> worked fine in test scenarios but I ran into problems with large video >> files. I would get timeouts from the ruby code that I was using. I'm >> developing a hydra head to ingest the video. I was going to start digging >> in and figure out what was going on, but a weekend off made me think it >> might be better (and easier) to decouple these two things a bit more. >> >> The next idea I have is to use external datastreams but ditch Fedora's >> checksums. I didn't have any problems when I tried this last week with >> large files, so I know it will work. Obviously, I need the checksums >> somewhere, so I'm considering adopting a BagIt structure on the filesystem >> to store all the video files, or groups of video files, and then point >> fedora to the location of each file. >> >> Has anyone tried such a scenario with or without success? I could always >> put a checksum file right alongside the video file and forgo the BagIt >> option, but I like the idea of having two independent but functional >> systems. On one side Fedora which tracks the locations of the files and >> their metadata, and BagIt which tracks the binary data. >> >> As a matter of policy, I would use Fedora's checksums for data stored within >> Fedora, which we don't have at the moment, but I anticipate using for images >> and other smaller sized files. >> >> Any thoughts are appreciated. >> >> thanks, >> >> ...adam >> >> ____________________________________________ >> Adam Wead >> Systems and Digital Collections Librarian >> Rock and Roll Hall of Fame and Museum >> 216.515.1960 (t) >> 215.515.1964 (f) >> >> ------------------------------------------------------------------------------ >> The demand for IT networking professionals continues to grow, and the >> demand for specialized networking skills is growing even more rapidly. >> Take a complimentary Learning@Cisco Self-Assessment and learn >> about Cisco certifications, training, and career opportunities. >> http://p.sf.net/sfu/cisco-dev2dev >> _______________________________________________ >> Fedora-commons-users mailing list >> Fedora-commons-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >> >> > > ------------------------------------------------------------------------------ > The demand for IT networking professionals continues to grow, and the > demand for specialized networking skills is growing even more rapidly. > Take a complimentary Learning@Cisco Self-Assessment and learn > about Cisco certifications, training, and career opportunities. > http://p.sf.net/sfu/cisco-dev2dev > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > ------------------------------------------------------------------------------ The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
