I'll admit, I'm not as familiar with this area of the code, but after doing a quick scan here, I can see what you mean.
As Christian mentions, we may need to consider moving the xoai dependency down a level to dspace-api. That seems to be essentially what was done for the RDFConsumer (which has a similar issue, in that the EventConsumer is for the "dspace-rdf" webapp). In the case of the RDFConsumer, all the relevant RDF API code (org.dspace.rdf.*) was moved from "dspace-rdf" to "dspace-api": RDF API: https://github.com/DSpace/DSpace/tree/master/dspace-api/src/main/java/org/dspace/rdf RDFConsumer: https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/rdf/RDFConsumer.java I suspect this whole issue may be why we never had an EventConsumer for OAI to begin with. In the longer term, there should be a better way to manage all these dependencies in a more dynamic fashion (so we won't need to stuff all these APIs/dependencies into dspace-api). But, for the short term, and to fix this specific issue, it sounds like the "best" option may be to move the xoai dependency to dspace-api for now. If others have thoughts on this, please chime in. I definitely realize this is not ideal. But, our DSpace dependency model is not currently set up well to manage this sort of situation. In the future, I think there may be some refactoring we could do to alleviate these issues (e.g. considering moving more of this EventConsumer stuff into being dynamically loaded via Spring or similar). So, to sum up. I'd still love to see a PR. We need to fix this issue soon, rather than waiting around for a larger refactoring, so PRs are more than welcome. - Tim On 5/28/2015 9:01 AM, Christian Scheible wrote: > Hi Ondřej, hi Tim, > > It would be great to see this as a pull request. I had a solution for > that problem (the OAIIndexEventConsumer was part of dspace-api) but to > make it good enough for a pull request it would be needed to move the > xoai dependency from dspace-oai to dspace-api and additionally add at > least the caching part, org.dspace.xoai.app.XOAI and > org.dspace.xoai.util.ItemUtils to dspace-api. > Or as Ondřej suggested add a dependency to dspace-oai to xmlui, jspui, > rest and wherever the event has to be triggered. > > Best Regards, > Christian > > Am 28.05.2015 um 14:08 schrieb Ondřej Košarko: >> Hey Tim, >> >> No reason for not submitting a pull request and I for sure can move >> the code around a bit. However, I don't see a way around adding >> dspace-oai classes as a dependency. At the end of the day the consumer >> just calls org.dspace.xoai.app.XOAI.index. >> >> xmlui webapp does not have dspace-oai jar/classes in it's WEB-INF dir, >> the same is probably true for jspui webapp. >> >> Perhaps there is another approach? >> >> Regards, >> OK >> >> 2015-05-27 16:23 GMT+02:00 Tim Donohue <tdono...@duraspace.org >> <mailto:tdono...@duraspace.org>>: >> >> Hi Ondřej, >> >> I wonder if you'd be willing to submit a Pull Request of these >> changes (the new OAIIndexEventConsumer, etc)? These seem like >> potentially useful changes overall, and would be something we'd >> consider adding to DSpace. >> >> My only (small) recommendation would be to simply put the >> OAIIndexEventConsumer into the "dspace-api", so that it can be >> easily enabled/disabled from the >> "event.dispatcher.default.consumers" setting. Plus, if it's in the >> dspace-api, that ensures it'd also be triggered by any updates >> from other interface (XMLUI, JSPUI, REST, etc) >> >> - Tim >> >> On 5/27/2015 5:08 AM, Ondřej Košarko wrote: >> >> >> ---------- Forwarded message ---------- >> From: *Ondrej Kosarko* <ko...@centrum.cz >> <mailto:ko...@centrum.cz> <mailto:ko...@centrum.cz >> <mailto:ko...@centrum.cz>>> >> Date: 2015-05-27 12:05 GMT+02:00 >> Subject: Re: [Dspace-tech] OAI-PMH data provider 2.0 not >> persistent? >> To: Christian Scheible <christian.schei...@uni-konstanz.de >> <mailto:christian.schei...@uni-konstanz.de> >> <mailto:christian.schei...@uni-konstanz.de >> <mailto:christian.schei...@uni-konstanz.de>>> >> Cc: "dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net> >> <mailto:dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net>>" >> <dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net> >> <mailto:dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net>>> >> >> >> Hi Christian, >> >> I'm not sure I remember all the setup, but here goes... >> >> in dspace.cfg there is >> >> event.dispatcher.default.consumers = versioning, discovery, >> eperson, >> harvester, eudatreplication, *xoai* >> >> >> and >> >> # consumer to maintain the oai index >> event.consumer.xoai.class = >> cz.cuni.mff.ufal.event.OAIIndexEventConsumer >> event.consumer.xoai.filters = >> >> Community|Collection|Item|Bundle|Bitstream+Add|Create|Modify|Modify_Metadata|Delete|Remove >> >> >> and you need to make sure the class is found by the event >> dispatcher, we >> solve it by adding dspace-oai(where the consumer lives) as a >> dependency >> to dspace-xmlui >> >> diff --git a/dspace-xmlui/pom.xml b/dspace-xmlui/pom.xml >> index 17d75c0..f19257f 100644 >> --- a/dspace-xmlui/pom.xml >> +++ b/dspace-xmlui/pom.xml >> @@ -93,6 +93,22 @@ >> </profiles> >> <dependencies> >> + <!-- because of event listener that updates oai --> >> + <dependency> >> + <groupId>org.dspace</groupId> >> + <artifactId>dspace-oai</artifactId> >> + <classifier>classes</classifier> >> + </dependency> >> >> I think that's about it... >> >> Regards, >> OK >> >> >> 2015-05-27 9:05 GMT+02:00 Christian Scheible >> <christian.schei...@uni-konstanz.de >> <mailto:christian.schei...@uni-konstanz.de> >> <mailto:christian.schei...@uni-konstanz.de >> <mailto:christian.schei...@uni-konstanz.de>>>: >> >> Hi Ondřej, >> >> I have seen that you are using your own >> OAIIndexEventConsumer. We >> did the same but I added It in the module additions (which >> leads to >> code duplication). >> How do you register your Consumer? Is it the same way as >> for other >> consumers? Because I couldn't solve that problem. >> >> Best Regards, >> Christian >> >> >> Am 27.05.2015 um 08:41 schrieb Ondřej Košarko: >> >> In short no... >> >> As I've suggested in the thread the indexAll should >> include also >> the withdrawn, that's what we do >> >> https://github.com/ufal/lindat-dspace/blob/lindat/dspace-oai/src/main/java/org/dspace/xoai/app/XOAI.java#L193 >> We use our Event consumer to trigger the update (it >> currently >> doesn't handle caching and changes in discoverable flag) >> >> https://github.com/ufal/lindat-dspace/blob/lindat/dspace-oai/src/main/java/cz/cuni/mff/ufal/event/OAIIndexEventConsumer.java >> And by the way dspace changed the deletion mode to >> transient in >> https://jira.duraspace.org/browse/DS-2491 so maybe the >> right way >> for dspace is updating the documentation... >> >> >> OK >> >> >> >> 2015-05-26 23:19 GMT+02:00 Jozef Misutka >> <misu...@ufal.mff.cuni.cz <mailto:misu...@ufal.mff.cuni.cz> >> <mailto:misu...@ufal.mff.cuni.cz >> <mailto:misu...@ufal.mff.cuni.cz>>>: >> >> Ondrej, is this happening to us as well? >> >> jm >> >> >> ------------------------------------------------------------------------ >> *From: *"Tim Donohue" <tdono...@duraspace.org >> <mailto:tdono...@duraspace.org> >> <mailto:tdono...@duraspace.org >> <mailto:tdono...@duraspace.org>>> >> *To: *dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net> >> <mailto:dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net>> >> *Sent: *Tuesday, 26 May, 2015 23:09:20 >> >> *Subject: *Re: [Dspace-tech] OAI-PMH data provider >> 2.0 not >> >> persistent? >> >> Hi All, >> >> I'm only just now "re-discovering" this thread, as >> I've found >> this same >> behavior in DSpace OAI-PMH servers. Here's how to >> see this >> behavior (and >> I've double checked it on http://demo.dspace.org, >> running 5.2): >> >> 1. Create an Item >> 2. Run the OAI-PMH import (./dspace oai import). >> 3. The item now appears in OAI-PMH >> 4. Withdraw that Item >> 5. Re-Run the OAI-PMH import (./dspace oai import) >> 6. The Item still appears in OAI-PMH and is NEVER >> flagged as >> withdrawn. >> >> So, you can still access all its metadata, etc. >> The ONLY way >> to trigger >> an update to the OAI-PMH record (of the withdrawn >> item) is to >> re-import >> EVERYTHING (./dspace oai import -c). However, as >> noted, since >> withdrawn >> items are not included in the XOAI "indexAll()" >> command, the >> item will >> now disappear from OAI-PMH entirely. >> >> Here's that indexAll() command: >> >> https://github.com/DSpace/DSpace/blob/master/dspace-oai/src/main/java/org/dspace/xoai/app/XOAI.java#L191 >> >> This behavior seems to be counter to what is >> documented at: >> >> https://wiki.duraspace.org/pages/viewpage.action?pageId=45548245#OAI-PMHDataProvider2.0%28Internals%29-Deletions >> >> The documentation specifically states that "DSpace >> keeps a >> permanent >> record of withdrawn items". It also states that a >> request for a >> withdrawn item "will yield the 'record deleted' >> header". >> >> This sounds like a bug to me. The documentation >> definitely >> does not >> match with the behavior. >> >> I've created a new bug ticket for this. It will need a >> volunteer to resolve. >> >> https://jira.duraspace.org/browse/DS-2593 >> >> - Tim >> >> On 3/6/2015 5:21 AM, helix84 wrote: >> > Yes, that makes sense. Still, it needs a brief >> verification >> because >> > sometimes our understanding of a concept and its >> implementation may >> > differ. >> > >> > >> > On Fri, Mar 6, 2015 at 10:12 AM, Kristian >> Roberto Salcedo >> > <k.r.salc...@ub.uio.no >> <mailto:k.r.salc...@ub.uio.no> >> <mailto:k.r.salc...@ub.uio.no >> <mailto:k.r.salc...@ub.uio.no>>> wrote: >> >> Hi Ivan, >> >> >> >> I might have overlooked something, but from >> your comments >> on https://jira.duraspace.org/browse/DS-2491 >> wouldnt it be >> meaningful >> >> to do both your declaration change and Ondřej's >> suggestion >> in order to match both what is declared and what the >> documentation says about OAI and deleted items? >> >> >> >> regards, >> >> Kristian >> >> >> >>> -----Original Message----- >> >>> From: ivan.ma...@gmail.com >> <mailto:ivan.ma...@gmail.com> <mailto:ivan.ma...@gmail.com >> <mailto:ivan.ma...@gmail.com>> >> [mailto:ivan.ma...@gmail.com >> <mailto:ivan.ma...@gmail.com> <mailto:ivan.ma...@gmail.com >> <mailto:ivan.ma...@gmail.com>>] On >> Behalf Of >> >>> helix84 >> >>> Sent: Thursday, March 05, 2015 3:46 PM >> >>> To: Kristian Roberto Salcedo >> >>> Cc: dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net> >> <mailto:dspace-tech@lists.sourceforge.net >> <mailto:dspace-tech@lists.sourceforge.net>>; João Melo >> >>> Subject: Re: [Dspace-tech] OAI-PMH data >> provider 2.0 not >> persistent? >> >>> >> >>> Hi Kristian, >> >>> >> >>> I think you're right that the declared >> >>> <deletedRecord>persistent</deletedRecord> >> doesn't match >> how DSpace >> >>> behaves. I filed a Jira issue and created a >> pull request >> to change the declared >> >>> status to >> <deletedRecord>transient</deletedRecord>. >> >>> >> >>> https://jira.duraspace.org/browse/DS-2491 >> >>> >> >>> >> >>> Regards, >> >>> ~~helix84 >> >>> >> >>> Compulsory reading: DSpace Mailing List Etiquette >> >>> >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >>> >> >>> >> >>> On Thu, Mar 5, 2015 at 2:52 PM, Kristian >> Roberto Salcedo >> >>> <k.r.salc...@ub.uio.no >> <mailto:k.r.salc...@ub.uio.no> >> <mailto:k.r.salc...@ub.uio.no >> <mailto:k.r.salc...@ub.uio.no>>> wrote: >> >>>> Hi all, >> >>>> >> >>>> In Dspace 4.2 we are currently seeing that >> our OAI feeds >> are not being >> >>> persistent as per these definitions when it >> comes to >> deleted (withdrawn) >> >>> items: >> >>>> >> >>>> >> >> http://www.openarchives.org/OAI/openarchivesprotocol.html#deletion >> >>>> >> >> https://wiki.duraspace.org/pages/viewpage.action?pageId=34640887#OAI- >> >>> P >> >>>> MHDataProvider2.0(Internals)-Deletions >> >>>> >> >>>> Withdrawn items do not get the <header >> status=deleted> as >> they should, >> >>> like in this example: >> >>>> >> >>>> >> >>> >> >> http://webservices.itcs.umich.edu/mediawiki/oaibp/index.php/Deleted_Re >> >>>> cord_Example_1 >> >>>> >> >>>> Withdrawing an item is not reflected in the >> OAI entry in >> any way. >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> This is our Identify page at the moment: >> >>>> >> >>>> https://www.duo.uio.no/oai/request?verb=Identify >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> I believe we're doing everything right >> regarding config >> parameters and >> >>> maintenance of the solr oai index. >> >>>> >> >>>> The only place I can find a >> "persistent"-parameter is in >> this config file: >> >>>> /www/var/data/dspace/config/oaicat.properties >> >>>> which I thought was deprecated, but we still >> set it just >> to be sure: >> >>>> Identify.deletedRecord=persistent >> >>>> >> >>>> All other relevant config parameters are set >> in these two >> files as far as I can >> >>> tell: >> >>>> >> >>>> /www/var/data/dspace/config/modules/oai.cfg >> >>>> >> /www/var/data/dspace/config/crosswalks/oai/description.xml >> >>>> >> >>>> We are running the oai import -o command nightly. >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> The only way we've found to remove withdrawn >> items from >> the OAI feed is >> >>> by running a complete re-indexing of the oai >> solr index: >> >>>> >> >>>> /www/var/data/dspace/bin/dspace oai import -c >> >>>> >> >>>> with a subsequent >> >>>> >> >>>> /www/var/data/dspace/bin/dspace oai clean-cache >> >>>> >> >>>> This removes a withdrawn item: >> >>>> >> >>>> https://www.duo.uio.no/handle/10852/42670 >> >>>> >> >>>> completely from the feed: >> >>>> >> >>>> >> >>> >> >> https://www.duo.uio.no/oai/request?verb=GetRecord&metadataPrefix=oai >> >>> _d >> >>>> c&identifier=oai:localhost:10852/42670 >> >>>> >> >>>> which is not what we want... >> >>>> >> >>>> ---------------------------------------------- >> >>>> >> >>>> >> >>>> Is anyone else experiencing the same problem? >> >>>> >> >>>> Maybe I've missed something - If this >> actually works >> differently than we >> >>> expect or has been addressed in OAI 2.1, >> please let me know. >> >>>> >> >>>> >> >>>> regards, >> >>>> Kristian Salcedo >> >>>> Universitetet of Oslo Library >> >>>> Department of digital services >> >>>> >> >>>> >> >>>> >> >> ---------------------------------------------------------------------- >> >>>> -------- Dive into the World of Parallel >> Programming The >> Go Parallel >> >>>> Website, sponsored by Intel and developed in >> partnership >> with Slashdot >> >>>> Media, is your hub for all things parallel >> software >> development, from >> >>>> weekly thought leadership blogs to news, >> videos, case >> studies, >> >>>> tutorials and more. Take a look and join the >> conversation >> now. >> >>>> http://goparallel.sourceforge.net/ >> >>>> _______________________________________________ >> >>>> DSpace-tech mailing list >> >>>> DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> <mailto:DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net>> >> >>>> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> >>>> List Etiquette: >> >>>> >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> > >> > >> >> ------------------------------------------------------------------------------ >> > Dive into the World of Parallel Programming The >> Go Parallel >> Website, sponsored >> > by Intel and developed in partnership with >> Slashdot Media, >> is your hub for all >> > things parallel software development, from >> weekly thought >> leadership blogs to >> > news, videos, case studies, tutorials and more. >> Take a look >> and join the >> > conversation now. http://goparallel.sourceforge.net/ >> > _______________________________________________ >> > DSpace-tech mailing list >> > DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> <mailto:DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net>> >> > >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> > List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> > >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> <mailto:DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net>> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> <mailto:DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net>> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List >> >> Etiquette:https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >> >> >> -- >> Christian Scheible >> Softwareentwickler / Abt. Content-basierte Dienste >> Kommunikations-, Informations- und Medienzentrum (KIM) >> Universität Konstanz >> 78457 Konstanz >> +49 (0)7531 / 88-2857 >> Raum B 703 >> >> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> <mailto:DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net>> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> <mailto:DSpace-tech@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List Etiquette: >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> List >> Etiquette:https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > > > -- > Christian Scheible > Softwareentwickler / Abt. Content-basierte Dienste > Kommunikations-, Informations- und Medienzentrum (KIM) > Universität Konstanz > 78457 Konstanz > +49 (0)7531 / 88-2857 > Raum B 703 > > > > ------------------------------------------------------------------------------ > > > > _______________________________________________ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > List Etiquette: > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > ------------------------------------------------------------------------------ _______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
