Adam--
To your Akubra question: We've been working on an Akubra architecture utilizing
multiple filesystems multiplexed together in a binary tree developed from a
custom URI hash calculation, which we're putting into production this month.
The relevance is that we're very interested in bringing Akubra's Quiescing
Storage design into play for HSM (our IT folks currently support a Quantum
StorNext instance which we use for workflow but without strong integration with
the repository architecture), and would be interested in discussing partnership
with another institution to do just that.
An alternative would be to employ a rules-based storage architecture like
iRODS. An iRODS-Fedora integration is available now, although I believe it is
under redesign for Akubra.
I'm thinking that in any naive implementation of your first two designs,
applications that request datastreams from Fedora will have to be aware that
responses from an HTTP request to a Fedora Web service may take as long to
appear as the HSM takes to retrieve material. If your HSM is like ours, that is
often several minutes and must involve a checksum operation for surety. This
implies some sophisticated behavior on the part of requesting applications that
must present some unusual workflow to an end user. ("Click here and wait ten
minutes for download to begin"?) Of course, your HSM may be better than ours
and be able to respond more immediately with more immediately useful
information. If it is and can, those designs make more sense. (And I envy you
your HSM!)
---
A. Soroka
Online Library Environment
the University of Virginia Library
On Mar 21, 2011, at 4:56 PM, Adam Wead wrote:
> Dear all,
>
> I'm currently mulling options as to how to implement a storage solution for
> our fedora repository. For backup and disaster recovery, I'm using Tivoli
> Storage Manager, with a SAN and LTO tape environment. I also have a Tivoli
> add-on called Space Management which acts as a hierarchical storage manager
> and moves files from disk to tape while leaving a portion of the file on the
> disk. We'll be holding a lot of uncompressed video files which won't change
> (ever) and won't be accessed very often. Possible strategies include:
>
> 1. Place Fedora's objects directory atop an HSM-managed filesystem, exported
> to the fedora server via NFS. Tweak HSM policies so that large files are
> moved out to tape, but Fedora can still "see" that they're there. Smaller
> derivative files remain on disk for faster access. Advantages: Fedora just
> gets a filesystem. Everything else is pushed back to the storage layer.
> Drawbacks/unknowns: complexity, performance via NFS and HSM, unknowns, others?
>
> 2. Keep the large files away from Fedora as externally-referenced objects
> sitting on an HSM filesystem; they're pulled from tape when needed.
> Derivative and datastream data stays in the repository. Advantages: simpler
> implementation. Drawbacks: more complicated ingest putting derivative files
> in one place and preservation files in another, maintaining link between the
> object reference and its actual file.
>
> 3. Akubra? The above two examples assume the default low-level storage
> method in versions prior to 3.4. I haven't used Akubra and don't know what
> it's able to do in this regard, or if it's a better option to use with #1 or
> #2 instead of the default objects directory.
>
> From what I've seen on the wiki, HSM directly with Fedora is on the wishlist,
> but not ready. Is anyone out there using Fedora with HSM in some way,
> doesn't have to be IBM specifically, and could offer suggestions, advice,
> caveats?
>
> thanks in advance,
>
> ...adam
> ____________________________________________
> Adam Wead
> Systems and Digital Collections Librarian
> Rock and Roll Hall of Fame and Museum
> 216.515.1960 (t)
> 215.515.1964 (f)
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today!
> http://p.sf.net/sfu/intel-dev2devmar_______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
