At Rutgers we are managing very large (ofter more than one GB) preservation
masters within Fedora as "M" datastreams.
Jeffery
On Wed, Mar 6, 2013 at 11:25 AM, Aaron Collie <w.a.col...@gmail.com> wrote:
> There was another discussion on this list (I believe) about E vs M storage
> regarding large files like preservation masters. It made me wonder how many
> people are actually managing preservation masters in Fedora Commons (vs
> simply managing derivative dissemination files). The files I am looking at
> (for example) are 800MB TIFF images (e.g. full color newspaper scans,
> maps). We are still developing our solution, but it involves internally
> managing the serving files and externally managing/referencing the
> preservation masters.
>
> I would love to hear more use cases for this type of application.
> Specifically, how people are managing preservation masters--which are
> really kind of our bread and butter as digital preservation-minded people.
> Maybe I am overlooking something?
>
> -Aaron
>
>
> On Wed, Mar 6, 2013 at 9:56 AM, aj...@virginia.edu <aj...@virginia.edu>wrote:
>
>> The limits on upload depend radically on your network design. Uploading
>> from a single node to itself, the limits may be in the several GBs.
>> Uploading across a limited network, maybe not so much. Fedora has no
>> inherent limitations on the upload process, but many other pieces of the
>> system probably will. Fedora will require a little more storage than the
>> actual size of binary content. That additional storage is rarely an problem
>> or even interesting to budget, unless you are creating a huge number of
>> tiny objects.
>>
>> 500k objects is not a large amount for a Fedora repository. What will
>> matter more is how large each is. How large are they, on average?
>>
>> I want to emphasize what Justin said: "external datastreams are a...
>> workaround for when you have no other choice." To my mind that's exactly
>> right.
>>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>> On Mar 6, 2013, at 9:40 AM, James, Eric wrote:
>>
>> > Have any benchmarks been done regarding file size thresholds for
>> managed datastreams? I.E. how many MB/GB would break the upload process or
>> just be too slow to be practical? And are the other issues involved
>> (network, storage, etc).
>> >
>> > I'm dealing with 500k images on disk, and am considering either to
>> leave them where they are and use an external datastream to point to them,
>> or transfer them into a managed datastreams. There is also another
>> collection of large AV files in the GBs range which at this point seem like
>> external is the way to go.
>> >
>> > Thanks,
>> > Eric
>> > From: Justin Coyne [jus...@curationexperts.com]
>> > Sent: Monday, March 04, 2013 9:44 PM
>> > To: Support and info exchange list for Fedora users.
>> > Subject: Re: [fcrepo-user] Managed vs. External Storage
>> >
>> > I find that it's much easier to put all your objects within the Fedora
>> repository if you have the option. If you're storing externally, you loose
>> the ability to do automatic checksum validation and versioning.
>> Furthermore, you have to make certain that you maintain integrity between
>> your external store and the reference within Fedora. To me external
>> datastreams are a great workaround for when you have no other choice.
>> >
>> > Best Regards,
>> > Justin Coyne
>> > Data Curation Experts
>> >
>> > On Mon, Mar 4, 2013 at 4:30 PM, Schmidt, Lisa (lschmidt) <
>> lschm...@msu.edu> wrote:
>> > At the Michigan State University Archives, we are wrestling with the
>> question of where to store digital objects/AIPs: within our Fedora
>> repository (managed), or externally.
>> >
>> > What are the issues associated with each approach?
>> >
>> > We have external storage available on an IX Systems storage device, and
>> have been planning to use it for archival storage of AIPs with pointers in
>> the Fedora repository; it would be synched to a second IX storage device
>> that would function as our dark archive. We want to do our due diligence,
>> however, to ensure that this is the right approach.
>> >
>> > Thank you,
>> > Lisa
>> >
>> > ____________________________________________________
>> >
>> > Lisa M. Schmidt
>> > Electronic Records Archivist
>> > University Archives & Historical Collections
>> > 888 Wilson Road
>> > Room 101 , Conrad Hall
>> > Michigan State University
>> > East Lansing, MI 48824
>> >
>> > lschm...@msu.edu
>> > 1-517-884-6441
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Everyone hates slow websites. So do we.
>> > Make your web apps faster with AppDynamics
>> > Download AppDynamics Lite for free today:
>> > http://p.sf.net/sfu/appdyn_d2d_feb
>> > _______________________________________________
>> > Fedora-commons-users mailing list
>> > Fedora-commons-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
>> > Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
>> > endpoint security space. For insight on selecting the right partner to
>> > tackle endpoint security challenges, access the full report.
>> >
>> http://p.sf.net/sfu/symantec-dev2dev_______________________________________________
>> > Fedora-commons-users mailing list
>> > Fedora-commons-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
>> Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
>> endpoint security space. For insight on selecting the right partner to
>> tackle endpoint security challenges, access the full report.
>> http://p.sf.net/sfu/symantec-dev2dev
>> _______________________________________________
>> Fedora-commons-users mailing list
>> Fedora-commons-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>>
>
>
>
> ------------------------------------------------------------------------------
> Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
> Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
> endpoint security space. For insight on selecting the right partner to
> tackle endpoint security challenges, access the full report.
> http://p.sf.net/sfu/symantec-dev2dev
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
------------------------------------------------------------------------------
Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
endpoint security space. For insight on selecting the right partner to
tackle endpoint security challenges, access the full report.
http://p.sf.net/sfu/symantec-dev2dev
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
