Hi
We in Uppsala have benn using Fedora's message queues for more than a year
now for the following:
* the indexing of our meta-data in external Solr instances is triggered by
theme (we have three different Solr instance with different indexes for
different purposes)
* depending on the message content the sending of e-mails is triggered, for
example when an object has been created or updated
We're using the included ActiveMQ library. The batch ingest of about 30.000
objects - containing 3 datastreams at minimum - gave us no problem at all
with message queues. When we tried the same with topics we got out of memory
errors or the message clients stopped to recieve messages. We haven't tried
to use an external message broker.
For now, we're satisfied with the current solution.
Regards
Uwe Klosa
Electronic Publishing Centre
Uppsala University
2009/10/27 Scott Prater <pra...@wisc.edu>
> Hi, Bill --
>
> Thanks for your detailed description. You're quite a ways further down
> the workflow path than we are right now. I'd like to talk with you more
> offline about the details of your setup.
>
> As I understand it, so far, you've been using message queues just to
> trigger *Fedora* events, correct? Do you have any situations where a
> Fedora event is triggering outside actions?
>
> -- Scott
>
> Bill Parod wrote:
> > Hi Scott,
> >
> > I don't have numbers for you at this point, but we've been using
> > ActiveMQ as a queueing / message broker to loosely couple systems that
> > participate, along with Fedora, in various collection management
> > activity. We're planning to ingest a collection of 50k images and can
> > report on how that goes by the end of the year, probably in November.
> >
> > Here's a longer description of broader context and what we've done so
> > far in case of interest. I'd be very interested in comments or similar
> > experience you or others have.
> >
> > The general idea is to let various systems report significant events via
> > JMS with the event handling to be loosely coupled (some operations take
> > a while), extensible and flexibly/centrally managed externally to the
> > systems. The motivation is that we have various systems contributing
> > repository content (Drupal, Archon, and Voyager for starters), we need
> > to synchronize metadata updates (descriptive metadata and also authority
> > references) and want to minimize the system-specific customization
> > necessary for systems to communicate/participate.
> >
> > For example, we have started to use Drupal for creating repository
> > content, using a CCK content type to enter/edit media file paths and
> > repeating Dublin Core fields. We use a modified version of the MsgQueue
> > Drupal module to queue insert/update/delete node_api events to ActiveMQ
> > for Drupal configured CCK types of interest. Currently the message
> > includes the node and CCK data, as the initial content types are simple
> > and relatively small (Dublin Core). The JMS handler for these events
> > invokes ingest scripts on a conversion/ingest-preprocessing server that
> > obtain the media (images currently), JHove validate it, extract
> > technical metadata, convert to jp2, push the jp2 to image server, push
> > the source file to archive server, generate foxml with datastreams (DC,
> > MODS, SVG) and ingest into Fedora. In this scenario Drupal doesn't have
> > to know about Fedora or how to invoke the various ingest preprocessing.
> > It just reports events of interest. We plan to expand event handling to
> > include solr indexing and preservation metadata capture.
> >
> > We also use the Drupal Node Import module to trigger such ingests from a
> > spreadsheet. We're thinking about modifying this module to take fields
> > from an ODBC/JDBC source and from file systems structures (directories
> > of images for example). We're planning to test the ingest of a
> > collection of 50K images in this way. We can report on how that goes.
> >
> > We have a slightly different pattern for EAD objects created/edited in
> > Archon. Since finding aid EAD data is more complex than DC, in this case
> > Archon will just send a message to the JMS broker that an Archon
> > collection should be synchronized to Fedora. The JMS handler will turn
> > around and make a request to Archon for the collection as an EAD export
> > and ingest/update appropriately into Fedora. Haven't dine this yet, but
> > it's due by the end of this week.:) It's almost there.
> >
> > We've been talking about synchronizing authority data similarly, for
> > example when we get updates from OCLC for our local name authority file
> > in Voyager that should propagate to affected records in Archon, Drupal,
> > and Fedora. We're still talking this through, but it's part of the
> > motivation to handle synchronization processes in a JMS message broker.
> >
> > I'd be very interested in comments, anecdotes, benchmarks, suggestions
> > in this approach.
> >
> > Thanks,
> > Bill
> >
> >
> > On Oct 27, 2009, at 8:09 AM, Scott Prater wrote:
> >
> >> We'll soon be activating JMS in our Fedora installation, in order to
> >> update our Solr index and memory cache on purge/ingest/update events.
> >> Has anyone had any other experience using a third-party message broker?
> >> In particular, does anyone have any experience using a message broker
> >> in situations of high traffic or with large datasets or heavy batch
> >> operations? Any recommendations or gotchas to look out for?
> >>
> >> TIA,
> >>
> >> -- Scott
> >>
> >> aj...@virginia.edu wrote:
> >>> I may have missed part of the conversation where you describe this,
> >>> but are you using the JMS gear that comes with Fedora or a separate
> >>> JMS broker (e.g. ActiveMQ)?
> >>>
> >>> ---
> >>> A. Soroka
> >>> Digital Research and Scholarship R & D
> >>> the University of Virginia Library
> >>>
> >>>
> >>>
> >>> On Oct 26, 2009, at 6:56 PM, <carsten.friedr...@csiro.au> wrote:
> >>>
> >>>> · Using the message queue with fedoragsearch to update SOLR
> >>>> seems not to be a good idea for larger datasets. I suspect using the
> >>>> message queue for anything with a large datasets is not a good idea.
> >>>> It seems to consume a lot of resources when running, does not seem
> >>>> to shut down reliably (resulting in very long rebuild time next time
> >>>> you start tomcat) and seems to scale very badly (performance goes
> >>>> down significantly the more objects we add). There are a lot of
> >>>> “seems” in this statement because we didn’t actually measure this
> >>>> and it is only based on our subjective observation.
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>>
> >>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> >>> is the only developer event you need to attend this year. Jumpstart
> your
> >>> developing skills, take BlackBerry mobile applications to market and
> >>> stay
> >>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> >>> http://p.sf.net/sfu/devconference
> >>> _______________________________________________
> >>> Fedora-commons-users mailing list
> >>> Fedora-commons-users@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>
> >>
> >> --
> >> Scott Prater
> >> Library, Instructional, and Research Applications (LIRA)
> >> Division of Information Technology (DoIT)
> >> University of Wisconsin - Madison
> >> pra...@wisc.edu
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >>
> >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> >> is the only developer event you need to attend this year. Jumpstart your
> >> developing skills, take BlackBerry mobile applications to market and
> stay
> >> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> >> http://p.sf.net/sfu/devconference
> >> _______________________________________________
> >> Fedora-commons-users mailing list
> >> Fedora-commons-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>
> >
>
>
> --
> Scott Prater
> Library, Instructional, and Research Applications (LIRA)
> Division of Information Technology (DoIT)
> University of Wisconsin - Madison
> pra...@wisc.edu
>
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
