Re: [Dspace-tech] Federating a number of DSpace instances
Hello *, Sorry for taking this thread up again. As we at U.PORTO are also very interested in the Federated Repository issue, due to our Organization distribution. Could some one tell if the proposal of Lerry Stone in the Dspace wiki: http://wiki.dspace.org/index.php/ObjectUri Is being considered in the future versions of Dspace ? On the other hand Scott, does this patch you developed fulfill Larry requirements of ObjectUri ? if so, can you share it with the community ? My ideia is to integrate this new type of objects in our Batch Import System, even if it takes Database changes. Best regards, Rui On Thu, 2009-02-12 at 08:50 -0600, Scott Phillips wrote: John, Down here in Texas we also face this challenge to create a federated repository among all institutional repositories across the state. Up until now we've been doing a manual federation process which includes dspace imports and exports plus some scripts that work on those exports. This has been a huge pain to work with, but it's bought us time to develop something better. We have been working on a patch which we expect to make public sometime next week. Right now we're working out a few library dependency issues with maven. The patch requires database changes so I don't expect it for 1.5.2, but hopefully soon their after. This patch will add: Part A: The ability for a DSpace collection to be harvested from another repository via OAI-PMH. When you look at the collection editing screen in either Manakin or the JSPUI there will be another tab for collection source that allows a curator to set up harvesting for a collection. It can be done either as a one-time thing just to move a collection or you can have it periodically harvest the collection. Further more you'll be able to select from three types of harvests 1) just metadata (plain old OAI-PMH), 2) metadata + bitstreams (via ORE), 3) metadata + links to bitstreams in their source repository (i.e. we get the ORE document and store that in DSpace but don't grab the bitstreams individualy). Part B: The ability for DSpace to serve up ORE documents via OAI-PHM. Scott-- On Feb 10, 2009, at 10:53 AM, John Preston wrote: Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech signature.asc Description: This is a digitally signed message part -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
Sorry to take so long to reply but I got side tracked. One question. On Thu, Feb 12, 2009 at 9:50 AM, Scott Phillips scott.a.phill...@gmail.com wrote: This patch will add: Part A: The ability for a DSpace collection to be harvested from another repository via OAI-PMH. When you look at the collection editing screen in either Manakin or the JSPUI there will be another tab for collection source that allows a curator to set up harvesting for a collection. It can be done either as a one-time thing just to move a collection or you can have it periodically harvest the collection. Further more you'll be able to select from three types of harvests 1) just metadata (plain old OAI-PMH), 2) metadata + bitstreams (via ORE), 3) metadata + links to bitstreams in their source repository (i.e. we get the ORE document and store that in DSpace but don't grab the bitstreams individualy). Sounds great. How will you handle the synchronization issues across repositories when changes are made to a source repository. John -- Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
On Thu, Mar 5, 2009 at 10:59 AM, Scott Phillips scott.a.phill...@gmail.com wrote: The OAI protocol handles resending items that have been updated since the last harvest. Could you expand on this a little for me as I am not very familiar with the OAI protocol. If I import an item (meta data only) from a source repository via OAI but leave the bitstreams in the source repository, then when changes are made to the metadata and/or bitstreams in the source repository, how are these transferred to my repository. Also if I were to use ORE, would I have to handle the synchronization issues as the protocol does not. John -- Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
John, Down here in Texas we also face this challenge to create a federated repository among all institutional repositories across the state. Up until now we've been doing a manual federation process which includes dspace imports and exports plus some scripts that work on those exports. This has been a huge pain to work with, but it's bought us time to develop something better. We have been working on a patch which we expect to make public sometime next week. Right now we're working out a few library dependency issues with maven. The patch requires database changes so I don't expect it for 1.5.2, but hopefully soon their after. This patch will add: Part A: The ability for a DSpace collection to be harvested from another repository via OAI-PMH. When you look at the collection editing screen in either Manakin or the JSPUI there will be another tab for collection source that allows a curator to set up harvesting for a collection. It can be done either as a one-time thing just to move a collection or you can have it periodically harvest the collection. Further more you'll be able to select from three types of harvests 1) just metadata (plain old OAI-PMH), 2) metadata + bitstreams (via ORE), 3) metadata + links to bitstreams in their source repository (i.e. we get the ORE document and store that in DSpace but don't grab the bitstreams individualy). Part B: The ability for DSpace to serve up ORE documents via OAI-PHM. Scott-- On Feb 10, 2009, at 10:53 AM, John Preston wrote: Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
Hi John, you may use SRW/U to search. OAI-PMH is not made for searching, but for harvesting. Furthermore there is pf-DSpace for Federating DSpace see http://hdl.handle.net/2160/281 But I don't know what has become of it. Hope that helps Claudia Jürgen John Preston schrieb: The individuals departments will want to maintain their own instances in some cases, thus I need to be able to search across them in these cases. John On Tue, Feb 10, 2009 at 4:19 PM, Dorothea Salo ds...@library.wisc.edu wrote: On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
Hi Claudia, Interesting scheme, but it requires more of a relationship between the nodes than I require (or desire). The federation that I'm thinking of is for searching. It would be nice for any of the DSpace instances to auto magically discover who belongs to the federation and should be searched. One question. With SRW/U, is one searching the full text or just the dublin core metadata. John On Wed, Feb 11, 2009 at 3:13 AM, Claudia Jürgen claudia.juer...@ub.uni-dortmund.de wrote: Hi John, you may use SRW/U to search. OAI-PMH is not made for searching, but for harvesting. Furthermore there is pf-DSpace for Federating DSpace see http://hdl.handle.net/2160/281 But I don't know what has become of it. Hope that helps Claudia Jürgen John Preston schrieb: The individuals departments will want to maintain their own instances in some cases, thus I need to be able to search across them in these cases. John On Tue, Feb 10, 2009 at 4:19 PM, Dorothea Salo ds...@library.wisc.edu wrote: On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
Hi John, as SRW searches via the Lucene indexes, the fulltext (as keyword) will be searched if it's extraction and indexing is done with filter-media. Hope that helps Claudia Jürgen John Preston schrieb: Hi Claudia, Interesting scheme, but it requires more of a relationship between the nodes than I require (or desire). The federation that I'm thinking of is for searching. It would be nice for any of the DSpace instances to auto magically discover who belongs to the federation and should be searched. One question. With SRW/U, is one searching the full text or just the dublin core metadata. John On Wed, Feb 11, 2009 at 3:13 AM, Claudia Jürgen claudia.juer...@ub.uni-dortmund.de wrote: Hi John, you may use SRW/U to search. OAI-PMH is not made for searching, but for harvesting. Furthermore there is pf-DSpace for Federating DSpace see http://hdl.handle.net/2160/281 But I don't know what has become of it. Hope that helps Claudia Jürgen John Preston schrieb: The individuals departments will want to maintain their own instances in some cases, thus I need to be able to search across them in these cases. John On Tue, Feb 10, 2009 at 4:19 PM, Dorothea Salo ds...@library.wisc.edu wrote: On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
On Wed, Feb 11, 2009 at 07:19:06AM -0500, John Preston wrote: The federation that I'm thinking of is for searching. Note that this is a special case of the more general problem: my institution or consortium operates a number of different digital document repositories of various types, and I need to make it possible to search across them all with (from the user's POV) one query. Sounds like what SRU was made for. I hope so, because that's what I've been considering to meet a similar requirement. There's not even a plan yet, though, let alone code. It would be nice for any of the DSpace instances to auto magically discover who belongs to the federation and should be searched. Have each register itself in some sort of directory service. The search front-end can then first search the directory to discover participating repositories, or just walk the membership of a group or the content of a single container if you want to have multiple repository 'cliques'. -- Mark H. Wood, Lead System Programmer mw...@iupui.edu Friends don't let friends publish revisable-form documents. pgpeAMIHF5KHm.pgp Description: PGP signature -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
I have an SRU database implementation that does federated searching. It is very simple. It exposes a single SRU database and a configuration file lists the other SRU databases to be searched. The results are returned in the order of the databases listed in the configuration file (as I said, very simple). I use the federated search engine in production here at OCLC for our Identities database (http://worldcat.org/identities/search/Identities). Under the covers, that database searches 4 other databases. The code is available as Open Source, but I haven't had time to get it up on my Google Code site (http://code.google.com/p/oclcsrw/). I believe I can find my documentation from my old CVS site and would be glad to share source and binary jars if anyone is interested in trying them. I'd be glad to work with someone on improving the code if they wanted to add something like ranking to it. Ralph -Original Message- From: Mark H. Wood [mailto:mw...@iupui.edu] Sent: Wednesday, February 11, 2009 8:29 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Federating a number of DSpace instances On Wed, Feb 11, 2009 at 07:19:06AM -0500, John Preston wrote: The federation that I'm thinking of is for searching. Note that this is a special case of the more general problem: my institution or consortium operates a number of different digital document repositories of various types, and I need to make it possible to search across them all with (from the user's POV) one query. Sounds like what SRU was made for. I hope so, because that's what I've been considering to meet a similar requirement. There's not even a plan yet, though, let alone code. It would be nice for any of the DSpace instances to auto magically discover who belongs to the federation and should be searched. Have each register itself in some sort of directory service. The search front-end can then first search the directory to discover participating repositories, or just walk the membership of a group or the content of a single container if you want to have multiple repository 'cliques'. -- Mark H. Wood, Lead System Programmer mw...@iupui.edu Friends don't let friends publish revisable-form documents. -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
It would be nice for any of the DSpace instances to auto magically discover who belongs to the federation and should be searched. Have each register itself in some sort of directory service. The search front-end can then first search the directory to discover participating repositories, or just walk the membership of a group or the content of a single container if you want to have multiple repository 'cliques'. Yes, thats what I was thinking. John -- Mark H. Wood, Lead System Programmer mw...@iupui.edu Friends don't let friends publish revisable-form documents. -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
We have that use case here, John -- we are implementing separate DSpace instances for our 23 campuses, but we also need a separate interface that can search them all. We're just going to harvest the data using OAI-PMH. I'm sure you've thought of that, too. --Dave == David Walker Library Web Services Manager California State University http://xerxes.calstate.edu From: John Preston [byhisde...@gmail.com] Sent: Tuesday, February 10, 2009 8:53 AM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] Federating a number of DSpace instances Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
We have a situation here in New Zealand where each University has a repository (mainly dspace) and the National Library maintains an OAI-powered search engine that runs across them. OAI doesn't include the full text, so searching is significantly limited. If you search for a relatively rare surname in both, you'd expect to find more in the cross-site search, but this isn't the case, because most personal names occur in the acknowledgements and bibliography, which don't make it into the OAI metadata. Compare http://researcharchive.vuw.ac.nz/search?query=yeatessubmit=Go http://nzresearch.org.nz/index.php/search/results?query=yeatesx=0y=0 The second URL is the poorer search over the larger body of documents. The nzresearch.org.nz has huge benefits, but exhaustive searching isn't really one of them. cheers stuart Walker, David wrote: We have that use case here, John -- we are implementing separate DSpace instances for our 23 campuses, but we also need a separate interface that can search them all. We're just going to harvest the data using OAI-PMH. I'm sure you've thought of that, too. --Dave == David Walker Library Web Services Manager California State University http://xerxes.calstate.edu From: John Preston [byhisde...@gmail.com] Sent: Tuesday, February 10, 2009 8:53 AM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] Federating a number of DSpace instances Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Stuart Yeates http://www.nzetc.org/ New Zealand Electronic Text Centre http://researcharchive.vuw.ac.nz/ Institutional Repository -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
Walker, David wrote: We have that use case here, John -- we are implementing separate DSpace instances for our 23 campuses, but we also need a separate interface that can search them all. NASA/Langley purchased Google to use as its internal search engine to search only its internal applications (with the option to search external sites also). Sue We're just going to harvest the data using OAI-PMH. I'm sure you've thought of that, too. --Dave == David Walker Library Web Services Manager California State University http://xerxes.calstate.edu From: John Preston [byhisde...@gmail.com] Sent: Tuesday, February 10, 2009 8:53 AM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] Federating a number of DSpace instances Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Stuart Yeates http://www.nzetc.org/ New Zealand Electronic Text Centre http://researcharchive.vuw.ac.nz/ Institutional Repository -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
What you need probably depends on what you actually mean by and want from federation. If common search is your primary reason, and your content is largely open access, you can set up a Google custom search engine that will only return results from your DSpaces. This would just be regular Google results and not Scholar-style results (with citation counts etc) but very easy and free, and you can brand it if branding is a concern. (If branding isn't a concern, I'd just say make sure your instances are indexed by Scholar -- most users want the most comprehensive possible search). Back when I was at HP Labs some students and I experimented with content mirroring using OAI-PMH, i.e. DSpaces that actually mirror each other's content. I'm not sure where that code is now, but even if content mirroring is not a primary concern, exposing METS (which links to fulltext) via OAI-PMH would allow better indexing than out-of-the-box Dublin Core (which as others have said is basically useless for effective indexing). However you'd have a custom solution to maintain ad infinitum which is something I see people leap into all too often, and should be considered as a last resort. Hope this helps, Rob On Tue, Feb 10, 2009 at 14:38, Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] susan.m.thorn...@nasa.gov wrote: Walker, David wrote: We have that use case here, John -- we are implementing separate DSpace instances for our 23 campuses, but we also need a separate interface that can search them all. NASA/Langley purchased Google to use as its internal search engine to search only its internal applications (with the option to search external sites also). Sue We're just going to harvest the data using OAI-PMH. I'm sure you've thought of that, too. --Dave == David Walker Library Web Services Manager California State University http://xerxes.calstate.edu From: John Preston [byhisde...@gmail.com] Sent: Tuesday, February 10, 2009 8:53 AM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] Federating a number of DSpace instances Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Stuart Yeates http://www.nzetc.org/ New Zealand Electronic Text Centre http://researcharchive.vuw.ac.nz/ Institutional Repository -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code
Re: [Dspace-tech] Federating a number of DSpace instances
My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John On Tue, Feb 10, 2009 at 3:41 PM, Robert Tansley roberttans...@google.com wrote: What you need probably depends on what you actually mean by and want from federation. If common search is your primary reason, and your content is largely open access, you can set up a Google custom search engine that will only return results from your DSpaces. This would just be regular Google results and not Scholar-style results (with citation counts etc) but very easy and free, and you can brand it if branding is a concern. (If branding isn't a concern, I'd just say make sure your instances are indexed by Scholar -- most users want the most comprehensive possible search). Back when I was at HP Labs some students and I experimented with content mirroring using OAI-PMH, i.e. DSpaces that actually mirror each other's content. I'm not sure where that code is now, but even if content mirroring is not a primary concern, exposing METS (which links to fulltext) via OAI-PMH would allow better indexing than out-of-the-box Dublin Core (which as others have said is basically useless for effective indexing). However you'd have a custom solution to maintain ad infinitum which is something I see people leap into all too often, and should be considered as a last resort. Hope this helps, Rob On Tue, Feb 10, 2009 at 14:38, Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] susan.m.thorn...@nasa.gov wrote: Walker, David wrote: We have that use case here, John -- we are implementing separate DSpace instances for our 23 campuses, but we also need a separate interface that can search them all. NASA/Langley purchased Google to use as its internal search engine to search only its internal applications (with the option to search external sites also). Sue We're just going to harvest the data using OAI-PMH. I'm sure you've thought of that, too. --Dave == David Walker Library Web Services Manager California State University http://xerxes.calstate.edu From: John Preston [byhisde...@gmail.com] Sent: Tuesday, February 10, 2009 8:53 AM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] Federating a number of DSpace instances Does anyone know if any work is going on regarding federating a number of DSpace instances so that the group could be considered as a single DSpace instance for searching say. My use case has a number of DSpace instances that are operated and maintained as individual instances. When a user wishes to search for some information, then the search is performed across all instances, and returns links to where the info was found on the individual instance. John -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Stuart Yeates http://www.nzetc.org/ New Zealand Electronic Text Centre http://researcharchive.vuw.ac.nz/ Institutional Repository -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download
Re: [Dspace-tech] Federating a number of DSpace instances
On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
The individuals departments will want to maintain their own instances in some cases, thus I need to be able to search across them in these cases. John On Tue, Feb 10, 2009 at 4:19 PM, Dorothea Salo ds...@library.wisc.edu wrote: On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Federating a number of DSpace instances
You can submit your sitemaps to yahoo and use their BOSS API to search across multiple domains using domain=dspace1.domain.tld,dspace2.domain.tld,dspace-n.domain.tld. I believe yahoo is also working towards indexing dc meta data for the semantic web, and hopefully they will integrate this into their search api in the near future. Alternatively you could use the SRW/U server and SRW DSpace plugin to search the DSpace database directly. Unfortunately you can only search one db at a time so you would have to combine your results before listing them on the screen. However, SRW/U does output results in dublin core xml so they are easy to format using an XSL transformation or some other type of XML manipulation. Regards Hayden John Preston wrote: The individuals departments will want to maintain their own instances in some cases, thus I need to be able to search across them in these cases. John On Tue, Feb 10, 2009 at 4:19 PM, Dorothea Salo ds...@library.wisc.edu wrote: On Tue, Feb 10, 2009 at 3:14 PM, John Preston byhisde...@gmail.com wrote: My use case involves running a number of DSpace instances in various departments, and collaborating entities that hold digital materials (text, image, video, spatial, etc). I need to have a central point for users to search across all these instances. Some of the instances may not be public. The Carrot2 software looks interesting from my perspective. Anyone think I'm going in the wrong direction? John Why not have one DSpace with Manakin theming and individual-collection policies on access? Seems to me that solves the problem, although it may not be feasible if individual departments insist on owning their entire DSpace. If you're running the whole shebang, though, it seems workable, and might even be less work than the alternatives. Dorothea -- Dorothea Salods...@library.wisc.edu Digital Repository Librarian AIM: mindsatuw University of Wisconsin Rm 218, Memorial Library (608) 262-5493 -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
