Michael,
We're also investigating this very thing at the Rock Hall. We'll be using
Fedora, along with Hydra, to manage a large video collection of about 200
TB. Fedora will be storing binary video content externally and metadata
datastreams as managed content.
The uncompressed video files are very large, and I'm using a hierarchical
storage manager with IBM's Tivoli Storage Manager. Access video files
always remain on the SAN, while the uncompressed video are kept on tape and
spooled to disk when needed. Fedora accesses the files via http.
Initial tests look promising. The only catch thus far is that Fedora will
hang while an uncompressed video file is retrieved. However, this will not
happen very often.
I chose externally referenced files because of the HSM issue and also
because the video files are available via HTTP making it easier to stream
them with different software packages and not have to involve Fedora. Your
system will probably have different requirements, and it might make more
sense to keep all the data internal to Fedora. In any case, I'd be
interested to hear what you all decide.
...adam
____________________________________________
Adam Wead
Systems and Digital Collections Librarian
Rock and Roll Hall of Fame and Museum
216.515.1960 (t)
215.515.1964 (f)
On Thu, Jul 21, 2011 at 4:54 AM, Richard Green <r.gr...@hull.ac.uk> wrote:
> Michael
>
> In Hull we are treading a similar path to Tufts.
>
> Since we started with Fedora we have used externally referenced content.
> The main reason was that we did not want to be limited by the disk
> capacity of our server and, at that time, had no way of making external
> storage (the University SAN) look like a drive. This has worked very
> well for us but meant that we had to build ingest utilities that also
> moved content to the SAN as well as just building FOXML objects.
>
> We are going live with a new Hydra repository in September to replace
> the existing Muradora-based one. We can now map the SAN as a server
> drive and so we are going back to managed content so that Fedora can
> just be left to get on with things. We are doing this only because we
> can map the SAN, we wouldn't have done it if we had to be dependent on
> local hard drives. Potentially using Fedora's managed content will give
> us easier interaction with new Fedora storage technologies, too.
>
> I'm a little puzzled by Deborah's "we are looking at Hydra, which wants
> us to have managed content". I'm not aware that in designing Hydra we
> have ever taken a managed versus external view but I'll take that up
> with her :-) The only recommendation I recall making is that non-core
> XML datastreams should be managed to keep the FOXML small.
>
> Best
>
> Richard
>
> ___________________________________________________________________
>
> Richard Green
> Consultant to Library & Learning Innovation, University of Hull
> managing the Hydrangea in Hull and Hydra (Hull) Projects
>
> http://edocs.hull.ac.uk
> http://hydrangeainhull.wordpress.com
> http://projecthydra.org
>
>
>
> -----Original Message-----
> From: Michael Della Bitta [mailto:michaeldellabi...@nypl.org]
> Sent: 20 July 2011 7:57 PM
> To: fedora-commons-develop...@lists.sourceforge.net;
> fedora-commons-users@lists.sourceforge.net
> Subject: [fcrepo-dev] External Datastream Storage Questions
>
> Hello everyone,
>
> NYPL is currently assessing the architecture of our Fedora effort. As
> we are unable to accurately scope the long term ingest throughput
> requirements of our repository, a major question at hand is the
> viability of recording all datastreams as externally-referenced
> content and managing datastream storage using a RESTful storage
> manager of some kind.
>
> I would greatly appreciate hearing briefly from any institution who is
> using Fedora alongside a separate storage subsystem. What's the
> overall size of your Fedora rollout, what are you using to manage
> storage, what are you doing to keep your datastream storage and Fedora
> in sync, and what pitfalls have you run into?
>
> Thank you very much for any information you are able to provide,
>
> Michael Della Bitta
>
> Senior Applications Developer
> Office of Strategy
> The New York Public Library
> 188 Madison Avenue, 4th Floor
> New York, NY 10016
> (212) 592-7178
>
> ------------------------------------------------------------------------
> ------
> 10 Tips for Better Web Security
> Learn 10 ways to better secure your business today. Topics covered
> include:
> Web security, SSL, hacker attacks & Denial of Service (DoS), private
> keys,
> security Microsoft Exchange, secure Instant Messaging, and much more.
> http://www.accelacomm.com/jaw/sfnl/114/51426210/
> _______________________________________________
> Fedora-commons-developers mailing list
> fedora-commons-develop...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
>
> *****************************************************************************************
> To view the terms under which this email is distributed, please go to
> http://www.hull.ac.uk/legal/email_disclaimer.html
>
> *****************************************************************************************
>
> ------------------------------------------------------------------------------
> 5 Ways to Improve & Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
------------------------------------------------------------------------------
5 Ways to Improve & Secure Unified Communications
Unified Communications promises greater efficiencies for business. UC can
improve internal communications as well as offer faster, more efficient ways
to interact with customers and streamline customer service. Learn more!
http://www.accelacomm.com/jaw/sfnl/114/51426253/
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
