There was another discussion on this list (I believe) about E vs M storage
regarding large files like preservation masters. It made me wonder how many
people are actually managing preservation masters in Fedora Commons (vs
simply managing derivative dissemination files). The files I am looking at
(for example) are 800MB TIFF images (e.g. full color newspaper scans,
maps). We are still developing our solution, but it involves internally
managing the serving files and externally managing/referencing the
preservation masters.
I would love to hear more use cases for this type of application.
Specifically, how people are managing preservation masters--which are
really kind of our bread and butter as digital preservation-minded people.
Maybe I am overlooking something?
-Aaron
On Wed, Mar 6, 2013 at 9:56 AM, aj...@virginia.edu <aj...@virginia.edu>wrote:
> The limits on upload depend radically on your network design. Uploading
> from a single node to itself, the limits may be in the several GBs.
> Uploading across a limited network, maybe not so much. Fedora has no
> inherent limitations on the upload process, but many other pieces of the
> system probably will. Fedora will require a little more storage than the
> actual size of binary content. That additional storage is rarely an problem
> or even interesting to budget, unless you are creating a huge number of
> tiny objects.
>
> 500k objects is not a large amount for a Fedora repository. What will
> matter more is how large each is. How large are they, on average?
>
> I want to emphasize what Justin said: "external datastreams are a...
> workaround for when you have no other choice." To my mind that's exactly
> right.
>
> ---
> A. Soroka
> The University of Virginia Library
>
> On Mar 6, 2013, at 9:40 AM, James, Eric wrote:
>
> > Have any benchmarks been done regarding file size thresholds for managed
> datastreams? I.E. how many MB/GB would break the upload process or just be
> too slow to be practical? And are the other issues involved (network,
> storage, etc).
> >
> > I'm dealing with 500k images on disk, and am considering either to leave
> them where they are and use an external datastream to point to them, or
> transfer them into a managed datastreams. There is also another collection
> of large AV files in the GBs range which at this point seem like external
> is the way to go.
> >
> > Thanks,
> > Eric
> > From: Justin Coyne [jus...@curationexperts.com]
> > Sent: Monday, March 04, 2013 9:44 PM
> > To: Support and info exchange list for Fedora users.
> > Subject: Re: [fcrepo-user] Managed vs. External Storage
> >
> > I find that it's much easier to put all your objects within the Fedora
> repository if you have the option. If you're storing externally, you loose
> the ability to do automatic checksum validation and versioning.
> Furthermore, you have to make certain that you maintain integrity between
> your external store and the reference within Fedora. To me external
> datastreams are a great workaround for when you have no other choice.
> >
> > Best Regards,
> > Justin Coyne
> > Data Curation Experts
> >
> > On Mon, Mar 4, 2013 at 4:30 PM, Schmidt, Lisa (lschmidt) <
> lschm...@msu.edu> wrote:
> > At the Michigan State University Archives, we are wrestling with the
> question of where to store digital objects/AIPs: within our Fedora
> repository (managed), or externally.
> >
> > What are the issues associated with each approach?
> >
> > We have external storage available on an IX Systems storage device, and
> have been planning to use it for archival storage of AIPs with pointers in
> the Fedora repository; it would be synched to a second IX storage device
> that would function as our dark archive. We want to do our due diligence,
> however, to ensure that this is the right approach.
> >
> > Thank you,
> > Lisa
> >
> > ____________________________________________________
> >
> > Lisa M. Schmidt
> > Electronic Records Archivist
> > University Archives & Historical Collections
> > 888 Wilson Road
> > Room 101 , Conrad Hall
> > Michigan State University
> > East Lansing, MI 48824
> >
> > lschm...@msu.edu
> > 1-517-884-6441
> >
> >
> >
> ------------------------------------------------------------------------------
> > Everyone hates slow websites. So do we.
> > Make your web apps faster with AppDynamics
> > Download AppDynamics Lite for free today:
> > http://p.sf.net/sfu/appdyn_d2d_feb
> > _______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> >
> >
> ------------------------------------------------------------------------------
> > Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
> > Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
> > endpoint security space. For insight on selecting the right partner to
> > tackle endpoint security challenges, access the full report.
> >
> http://p.sf.net/sfu/symantec-dev2dev_______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
>
> ------------------------------------------------------------------------------
> Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
> Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
> endpoint security space. For insight on selecting the right partner to
> tackle endpoint security challenges, access the full report.
> http://p.sf.net/sfu/symantec-dev2dev
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
endpoint security space. For insight on selecting the right partner to
tackle endpoint security challenges, access the full report.
http://p.sf.net/sfu/symantec-dev2dev
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
