Hi John:
See below
Richard R
On Thu, 2008-06-19 at 11:13 -0500, John Preston wrote:
> I guess before making any firm recommendation, I'd need to
> know
> what your requirements are. Do you, e.g., require
> transactional closure
> over the content change that generates the first event and the
> additions
> you want your consumer to make based on it?
> No.
>
>
> Can you describe what you are trying to accomplish? As noted,
> both options 2 & 3 cause your code
> to fork from the distribution, creating maintenance issues in
> the future.
> Yeah I know.
>
> I have a number of different requirements for consumers, ranging from
> simple notification of item, bitstream or metadata changes,
> to item, bitstream or metadata additions. I'll try and summarize a few
> of the cases below.
>
> 1) When and item is added, I wish to include bitstreams in the item in
> particular bundles. These could be thumbnail images,
> or bitstreams derived from processing of already available bitstreams.
> For example, if I add spatial files to an item I wish to be able
> to be able to add various spatial indexes to specific bundles, as well
> as thumbnails.
So can I infer that in this use-case, the latency imposed by batching
these operations in media filter is intolerable/undesirable? (You could
run it quite frequently if needed). I'm generally cautious about
entangling the generation of derivatives and the addition of the primary
artifact. You want the latter always to succeed, since the former can
always be regenerated. There also can be issues of response-time to
worry about, depending on the derivative(s). These are a few of the
motivations behind the Media-Filter architecture, rather than having
synchronous creation of derivatives, which seems to be your objective
here.
> 2) When item, metadata or bitstreams are changed (modified or
> removed), I wish to update various metadata fields to ensure that
> they are consistent with the data held in the item.
Consistency constraints form a very interesting class of use-cases.
Are these changes coming though the admin UI, or elsewhere, or anywhere?
Do you want a consistency check to be performed on any change, or only
certain ones, etc?
> 3) When item, metadata or bitstream is added, I wish to update the
> configuration of other co-operating applications to include the
> changes made to the DSpace base data. For example, if I add a spatial
> file to an item, then I want to add this item to a WMS
> server so that it is also available to spatial clients.
This is in the sweet-spot of the event mech - you could have a very
simple consumer that does this notification following the creation
events that stem from use-case 1.
>
So I guess the short answer would be to write a filter for 1 & 2,
and a consumer for 3. This would keep you entirely in the codebase.
If you need a tighter coupling than the media filter can provide,
we have talked about providing an in-process asynchronous message queue
consumer (we already have a prototype enterprise JMS queue, if you would
want to look at that, but it might be over-kill). Let me know if you are
interested in this alternative.
As to your current code, I can see how this will work, and it's fine
in the interim, but I don't think in future releases we will want to
expose bare DB connections (getDBConnection()), since this breaks
some encapsulations we are striving for in our DAO and other work.
>
> As I mentioned earlier, I have now got my consumer working without the
> earlier errors usiing a new context in the following way:
>
> Context ctx = new Context();
> ctx.setCurrentUser(originalContext.getCurrentUser());
> ctx.setCurrentLocale(originalContext.getCurrentLocale());
> ....
> ....
> // If I have to create a new bundle
> tbundles = new Bundle[]{item.createBundle(THUMBNAIL_BUNDLE)};
> // Commit changes to datastore
> tbundles[0].update();
> item.update();
> // Add bundle and item objects to the original context
> originalContext.cache(tbundles[0], tbundles[0].getID());
> originalContext.cache(item, item.getID());
> ....
> ....
> ....
> // commit changes to datastore bypassing any event triggered during
> use
> ctx.getDBConnection().commit();
>
> // Clear temporary context so it will be garbage collected
> ctx.clearCache();
>
>
> By using the temporary context and NOT calling the
> ctx.getDBConnection().commit() method I save the changes
> to the database while bypassing any events triggered during its use. I
> found that I had to add any of the DSpace
> objects that I used with the temporary context to the origonal context
> so they would be found by my code
> later on. I also cleared the cache of the temporary so it will be
> properly garbage collected.
>
> I'll see how long this lasts.
>
> On question. Is there some early info at this stage of how the event
> system for 2.0 is shaping up to be.
Not that I know of.
>
> John
>
>
>
>
> If you don't need closure, why not just have your consumer add
> to a list
> of updates to be made? This is one of the intended uses of the
> event
> system when
> it's hooked up to a message queue, e.g.
>
> Thanks,
>
> Richard
>
>
> Quoting John Preston <[EMAIL PROTECTED]>:
>
> > Sorry to bother you again, but quick question. When does the
> end() method
> > get called. Is it after the events arraylist has completed
> iterations of all
> > events. if so then I could move all my processing of the
> bundles,
> > bitstreams and items to this method. The only other thing I
> would have to
> > guard against would be more events of the same type, subject
> and object
> > with the same transaction ID.
> >
> >
> > This may be a wise thing to try, yes it is done after
> "consume" has
> > completed on all the events, the dispatcher is done with the
> list of events
> > at that point. most of the consumers actually do complete
> their processing
> > at this point too. But be careful of what I stated above,
> you need to make
> > sure this list of Events is coming from the correct Context.
> >
> > On Wed, Jun 18, 2008 at 9:33 PM, John Preston
> <[EMAIL PROTECTED]> wrote:
> >
> >> Ok, I think I understand what is happening. When I create
> or remove
> >> bundles, or bitstreams or any of these objects
> >> that generate events, these add new events which cause a
> concurrent
> >> modification conflict with the events arraylist, that is
> being used
> >> to dispatch the events in the first place. So having anew
> context will not
> >> help me. I need to bypass adding new events for any
> modifications.
> >>
> >> I think I have three choices, and your thoughts would be
> welcome on them.
> >>
> >> 1) Keep my code as mediafiletsr run periodically, and wait
> for 2.0;
> >> 2) extend the code for Bundles, Items, Bitstreams, etc,
> anything that
> >> issues events when the update() method is called, and add
> another
> >> update(boolean issueEvent) merthod that performs the
> database update
> >> with or without issuing any events, depending on the
> >> parameter passed;
> >> 3) implement my own event system;
> >>
> >> I'm leaning towards 2 or 3, but I don't know how many
> DSpace objects issue
> >> events. I'm playimg with a GWT front end application to
> some
> >> RPC services that actually interact with the dspace objects
> via the
> >> dspace-api codebase. It has gone pretty quickly up till
> this point so I'm
> >> looking for
> >> the quickest long lasting method.
> >>
> >>
> > I'd avoid (3). I think (2) will be coming in the DSpace 2.0
> next year
> > sometime (we are still at the drawing board on this so its
> very early). If
> > you take this route and it isn't compatible with what gets
> done in our
> > development, it may be difficult to integrate. (1) Is your
> safest bet, but I
> > understand why you want to get away from it.
> >
> > If you would, please forward a summary of this to the
> dspace-tech list as
> > a reply so that we have a complete thread there.
> >
> > Cheers,
> > Mark
> >
> >
> >> John
> >>
> >>
> >> On Wed, Jun 18, 2008 at 7:26 PM, John Preston
> <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >>> We do not recommend using a Event Consumer to directly
> manipulate the Item
> >>>> in question by adding Bitstreams or altering its metadata
> because not only
> >>>> is that available Contexts transactional window is closed
> by the
> >>>> time you've
> >>>> reached processing the Event in the consumer, but its
> difficult to predict
> >>>> the behavior or risk of getting the Event System in the
> state of of an
> >>>> infinite event loop because your alterations will in
> turn, also alter the
> >>>> Item and generate new events in the process.
> >>>>
> >>> Yes, I saw that, that's why I started to use
> >>> context().getDBConnection.commit() to avoid infinite
> event loops.
> >>>
> >>>
> >>>> Unfortunately, this is the sort of task most folks were
> hoping the
> >>>> Event System would solve in DSpace 1.5. I feel we are
> attempting
> >>>> to address
> >>>> its shortcomings in the DSpace 2.0 roadmap to come up
> with a solution.
> >>>>
> >>> That would be great.
> >>>
> >>>
> >>>> If you do decide to attempt to use the Event System in
> such a manner as
> >>>> to alter the Object in question and update it again, I
> would
> >>>> recommend using
> >>>> a new dspace Context and make sure you can trap the
> events in some sort of
> >>>> conditional check to make sure they should be propagated
> to through your
> >>>> Consumer again.
> >>>>
> >>> I tried this but I still get the error. I did the
> following:
> >>>
> >>> Context c = new Context();
> >>> ctx.setCurrentUser(context.getCurrentUser());
> >>> ctx.setCurrentLocale(context.getCurrentLocale());
> >>>
> >>> I'll look again at the code in the morning to see if I can
> understand
> >>> where the problem lies.
> >>>
> >>> Thanks.
> >>>
> >>> John
> >>>
> >>>>
> >>>>
> >>>> Sincerely,
> >>>> Mark
> >>>>
> >>>> On Jun 18, 2008, at 4:31 PM, John Preston wrote:
> >>>>
> >>>> Can someone say whether it is advisable or not to use the
> new event
> >>>> system under 1.5 to add metadata or bitstreams to an
> item.
> >>>>
> >>>> I am moving some of my code to event consumers that would
> add metadata
> >>>> for various item types as well as add bitstreams based on
> >>>> bitstreams already
> >>>> ingested. However I'm getting an exception when I try to
> commit these
> >>>> additions within the consumer (either the consume, end or
> finish methods).
> >>>> The particular exception is:
> >>>>
> >>>> java.util.ConcurrentModificationException
> >>>> at
> >>>> java.util.AbstractList
> $Itr.checkForComodification(AbstractList.java:372)
> >>>> at java.util.AbstractList
> $Itr.next(AbstractList.java:343)
> >>>> at
> >>>>
> org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:122)
> >>>> at
> org.dspace.core.Context.commit(Context.java:286)
> >>>> at
> >>>>
>
> org.ejamaica.server.dspace.DSpaceInstance.addSaying(DSpaceInstance.java:1906)
> >>>> at
> >>>>
>
> org.ejamaica.server.dspace.DSpaceInstance.updateSaying(DSpaceInstance.java:1060)
> >>>> at
> >>>>
>
> org.ejamaica.server.MyLibraryServiceImpl.myMethod(MyLibraryServiceImpl.java:147)
> >>>>
> >>>> Just for info. the section in my code that triggers the
> initial
> >>>> context.commit() occurs in DSpaceInstance.java:1906.
> >>>>
> >>>> If I don't commit after I add some metadata and/or
> bitstreams then I
> >>>> don't get the error, but then the data and bitstreams
> don't appear. If I
> >>>> even try and use the context.getDBConnection().commit();
> I get the same
> >>>> exception.
> >>>>
> >>>> John
> >>>>
> >>>>
> >>>> ~~~~~~~~~~~~~
> >>>> Mark R. Diggory - DSpace Developer and Systems Manager
> >>>> MIT Libraries, Systems and Technology Services
> >>>> Massachusetts Institute of Technology
> >>>> Home Page: http://purl.org/net/mdiggory/homepage
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
>
>
>
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> DSpace-tech mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________ DSpace-tech mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
