If you partition across full repository instances as described, you have set
yourself one set of scaling problems. If you instead partition your storage
layers (Low Level Store, RI, SQL DB, etc.) and provide one Fedora web
application (one suite of web services) over them, you can set yourself a
different set of scaling problems. We chose the latter for local reasons, but
it's worth noting that you can make that choice (and several others along the
same lines).
If I'm not mistaken, NSDL was at one time almost the "canonical" example of a
scaled-out repository infrastructure, although I do not know if that is still
the case.
---
A. Soroka
Online Library Environment
the University of Virginia Library
On Sep 8, 2011, at 12:12 PM, Chris Wilper wrote:
> Hi Jiri,
>
> Fedora can certainly handle tens of millions of objects. But I don't
> know about deployed instances that do that. Part of the main push
> behind 3.6 is to get our testing house in order so we can say more
> than just "it can handle lots of objects", but "with this many objects
> in this type of environment, it can handle this many concurrent
> reads".
>
> Regarding the multi-node partitioning you mentioned, it's certainly
> been considered a few times (pid-based, or hash-of-pid-based, for more
> even distribution among nodes). But I don't know of anyone who's
> successfully done it.
>
> I think the work around High Level Storage (roughly slated for 4.0)
> will help us answer some of these questions better. For example, if we
> can show that Fedora can persist all object state in a store like
> HBase or Cassandra, we will be that much closer to achieving true
> horizontal scaling. These storage solutions, if we employ them,
> already have partitioning strategies built in.
>
> - Chris
>
> On Tue, Sep 6, 2011 at 10:47 AM, Jiri Kremser <jiri.krem...@gmail.com> wrote:
>> Hi Chris,
>> I would like to ask you if you know about some deployed instance of
>> Fedora Repository, where a lot of digital objects is stored. By "a lot
>> of" I mean about ~tens of millions. Because this is our final goal and
>> we would like to know if it is plausible. Duracloud seems to be a nice
>> alternative, but there are some law restrictions which make impossible
>> storing the data outside the country.
>> To me it seems that a simple solution with a few Fedora instances in
>> cluster could work. Something like, if an pid starts with {0-2}
>> store/read it to/from fedora instance number 1 ({3-5} -> fedora
>> instance 2, etc.) Are you familliar with this solution, could it work
>> with fedora? The only problem can be the RI search, but the MPStore
>> can run only on top of one shared DB, imho.
>>
>>
>> JK
>>
>> ------------------------------------------------------------------------------
>> Special Offer -- Download ArcSight Logger for FREE!
>> Finally, a world-class log management solution at an even better
>> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
>> download Logger. Secure your free ArcSight Logger TODAY!
>> http://p.sf.net/sfu/arcsisghtdev2dev
>> _______________________________________________
>> Fedora-commons-developers mailing list
>> Fedora-commons-developers@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
>>
>
> ------------------------------------------------------------------------------
> Doing More with Less: The Next Generation Virtual Desktop
> What are the key obstacles that have prevented many mid-market businesses
> from deploying virtual desktops? How do next-generation virtual desktops
> provide companies an easier-to-deploy, easier-to-manage and more affordable
> virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
> _______________________________________________
> Fedora-commons-developers mailing list
> Fedora-commons-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops? How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
Fedora-commons-developers mailing list
Fedora-commons-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
