Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-20 Thread Conal Tuohy
On 17/06/11 05:59, Chris Wilper wrote: On 16/06/2011, at 23.26, Benjamin Armintor wrote: I think moving FieldSearch, ResourceIndex, and GSearch (among unknown others) out of core and into a queue of indexing services is an admirable development goal. Since GSearch is notified via JMS of

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-20 Thread Conal Tuohy
On 19/06/11 23:27, Conal Tuohy wrote: In a project I'm working on we have built a mechanism to keep external systems up to date with the state of the Fedora repository; using a similar technique to GSearch (i.e. a JMS listener). It seems rather apropos of this discussion. The difference

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-19 Thread Aaron Birkland
One possible direction forward here would be to start by offering a configurable option to let Field Search sit over a GSearch instance for a given repository (or pool of repositories) If FieldSearch is to remain part of Fedora's external APIs, I think this would be the most logical way

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-17 Thread Gert Schmeltz Pedersen
If you move FieldSearch out of core, I am curious to know, what FieldSearch can do, that GSearch cannot do? The answer may influence the release plan for GSearch that I presented at OR11. By the way, I think that GSearch is already prepared to be part of a distributed solution, through JMS,

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-17 Thread Chris Wilper
Hi Gert, On Fri, Jun 17, 2011 at 4:39 AM, Gert Schmeltz Pedersen g...@dtic.dtu.dk wrote: If you move FieldSearch out of core, I am curious to know, what FieldSearch can do, that GSearch cannot do? I don't know of anything that FieldSearch does that GSearch (or more accurately, any of its

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-17 Thread ajs6f
Even before considering a specific technology like JMS, I'd like to spend a little list-traffic with the question of the pattern to be deployed for this purpose, without thinking too much about the protocols involved. Is a suite of event queues the right choice? Might we do better with

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-17 Thread ajs6f
One possible direction forward here would be to start by offering a configurable option to let Field Search sit over a GSearch instance for a given repository (or pool of repositories). The API that Field Search offers could be translated to calls against GSearch's API, if institutions that

[fcrepo-dev] HighLevelStorage / DOManager

2011-06-16 Thread Asseg, Frank
Hola Guys! We are currently trying to combine fedora with HBase/HDFS as a Data/Metadata store and after implementing a low level storage proof of concept for HDFS (https://github.com/smeg4brains/akubra-hdfs) the next step would be to have fedora write the metadata information that is

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-16 Thread Chris Wilper
Hi Frank! Without a fairly significant rewrite of key bits of Fedora, attempting to implement the persistence on HBase+HDFS currently would be pretty difficult. Supporting that kind of big change is what the High Level Storage effort is all about. So far, in our development discussions, we have

Re: [fcrepo-dev] HighLevelStorage / DOManager

2011-06-16 Thread Aaron Birkland
Hi Frank, I have some unpublished code that served as a proof of concept while initially proposing HighLevelStorage. Let me see if I can put it up on github so you can take a look. It an experiment in using HBase, and in investigating what would be required for Fedora to be used in a