Thanks, Adam.
Throwing iRODS into the mix could work, but is also increasing complexity.
Unfortunately, I need to implement something fairly soon so I'm trying to
keep things simple, as much as that's possible. However, after I get that
going, it would be good to talk future work.
As far as our HSM goes, I don't think it's any faster than yours. It takes
at least 60 seconds to grab a tape and put it in a drive. That time will
increase as the library increases in size and the robot as to traverse more
tape slots.
Given that ingesting large files directly into Fedora would mean lots of
data getting thrown back and forth, I'm probably going to go with the
external option. This especially makes more sense since users really won't
need to get at those large files, only the smaller, managed derivative files
in the repository.
...adam
____________________________________________
Adam Wead
Systems and Digital Collections Librarian
Rock and Roll Hall of Fame and Museum
216.515.1960 (t)
215.515.1964 (f)
On Mon, Mar 21, 2011 at 10:05 PM, <aj...@virginia.edu> wrote:
> Adam--
>
> To your Akubra question: We've been working on an Akubra architecture
> utilizing multiple filesystems multiplexed together in a binary tree
> developed from a custom URI hash calculation, which we're putting into
> production this month. The relevance is that we're very interested in
> bringing Akubra's Quiescing Storage design into play for HSM (our IT folks
> currently support a Quantum StorNext instance which we use for workflow but
> without strong integration with the repository architecture), and would be
> interested in discussing partnership with another institution to do just
> that.
>
> An alternative would be to employ a rules-based storage architecture like
> iRODS. An iRODS-Fedora integration is available now, although I believe it
> is under redesign for Akubra.
>
> I'm thinking that in any naive implementation of your first two designs,
> applications that request datastreams from Fedora will have to be aware that
> responses from an HTTP request to a Fedora Web service may take as long to
> appear as the HSM takes to retrieve material. If your HSM is like ours, that
> is often several minutes and must involve a checksum operation for surety.
> This implies some sophisticated behavior on the part of requesting
> applications that must present some unusual workflow to an end user. ("Click
> here and wait ten minutes for download to begin"?) Of course, your HSM may
> be better than ours and be able to respond more immediately with more
> immediately useful information. If it is and can, those designs make more
> sense. (And I envy you your HSM!)
>
> ---
> A. Soroka
> Online Library Environment
> the University of Virginia Library
>
>
>
>
> On Mar 21, 2011, at 4:56 PM, Adam Wead wrote:
>
> > Dear all,
> >
> > I'm currently mulling options as to how to implement a storage solution
> for our fedora repository. For backup and disaster recovery, I'm using
> Tivoli Storage Manager, with a SAN and LTO tape environment. I also have a
> Tivoli add-on called Space Management which acts as a hierarchical storage
> manager and moves files from disk to tape while leaving a portion of the
> file on the disk. We'll be holding a lot of uncompressed video files which
> won't change (ever) and won't be accessed very often. Possible strategies
> include:
> >
> > 1. Place Fedora's objects directory atop an HSM-managed filesystem,
> exported to the fedora server via NFS. Tweak HSM policies so that large
> files are moved out to tape, but Fedora can still "see" that they're there.
> Smaller derivative files remain on disk for faster access. Advantages:
> Fedora just gets a filesystem. Everything else is pushed back to the
> storage layer. Drawbacks/unknowns: complexity, performance via NFS and HSM,
> unknowns, others?
> >
> > 2. Keep the large files away from Fedora as externally-referenced objects
> sitting on an HSM filesystem; they're pulled from tape when needed.
> Derivative and datastream data stays in the repository. Advantages:
> simpler implementation. Drawbacks: more complicated ingest putting
> derivative files in one place and preservation files in another, maintaining
> link between the object reference and its actual file.
> >
> > 3. Akubra? The above two examples assume the default low-level storage
> method in versions prior to 3.4. I haven't used Akubra and don't know what
> it's able to do in this regard, or if it's a better option to use with #1 or
> #2 instead of the default objects directory.
> >
> > From what I've seen on the wiki, HSM directly with Fedora is on the
> wishlist, but not ready. Is anyone out there using Fedora with HSM in some
> way, doesn't have to be IBM specifically, and could offer suggestions,
> advice, caveats?
> >
> > thanks in advance,
> >
> > ...adam
> > ____________________________________________
> > Adam Wead
> > Systems and Digital Collections Librarian
> > Rock and Roll Hall of Fame and Museum
> > 216.515.1960 (t)
> > 215.515.1964 (f)
> >
> ------------------------------------------------------------------------------
> > Enable your software for Intel(R) Active Management Technology to meet
> the
> > growing manageability and security demands of your customers. Businesses
> > are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> > be a part of the solution? Download the Intel(R) Manageability Checker
> > today!
> http://p.sf.net/sfu/intel-dev2devmar_______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
>
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
