Hi Scott, I don't have numbers for you at this point, but we've been using ActiveMQ as a queueing / message broker to loosely couple systems that participate, along with Fedora, in various collection management activity. We're planning to ingest a collection of 50k images and can report on how that goes by the end of the year, probably in November.
Here's a longer description of broader context and what we've done so far in case of interest. I'd be very interested in comments or similar experience you or others have. The general idea is to let various systems report significant events via JMS with the event handling to be loosely coupled (some operations take a while), extensible and flexibly/centrally managed externally to the systems. The motivation is that we have various systems contributing repository content (Drupal, Archon, and Voyager for starters), we need to synchronize metadata updates (descriptive metadata and also authority references) and want to minimize the system-specific customization necessary for systems to communicate/ participate. For example, we have started to use Drupal for creating repository content, using a CCK content type to enter/edit media file paths and repeating Dublin Core fields. We use a modified version of the MsgQueue Drupal module to queue insert/update/delete node_api events to ActiveMQ for Drupal configured CCK types of interest. Currently the message includes the node and CCK data, as the initial content types are simple and relatively small (Dublin Core). The JMS handler for these events invokes ingest scripts on a conversion/ingest- preprocessing server that obtain the media (images currently), JHove validate it, extract technical metadata, convert to jp2, push the jp2 to image server, push the source file to archive server, generate foxml with datastreams (DC, MODS, SVG) and ingest into Fedora. In this scenario Drupal doesn't have to know about Fedora or how to invoke the various ingest preprocessing. It just reports events of interest. We plan to expand event handling to include solr indexing and preservation metadata capture. We also use the Drupal Node Import module to trigger such ingests from a spreadsheet. We're thinking about modifying this module to take fields from an ODBC/JDBC source and from file systems structures (directories of images for example). We're planning to test the ingest of a collection of 50K images in this way. We can report on how that goes. We have a slightly different pattern for EAD objects created/edited in Archon. Since finding aid EAD data is more complex than DC, in this case Archon will just send a message to the JMS broker that an Archon collection should be synchronized to Fedora. The JMS handler will turn around and make a request to Archon for the collection as an EAD export and ingest/update appropriately into Fedora. Haven't dine this yet, but it's due by the end of this week.:) It's almost there. We've been talking about synchronizing authority data similarly, for example when we get updates from OCLC for our local name authority file in Voyager that should propagate to affected records in Archon, Drupal, and Fedora. We're still talking this through, but it's part of the motivation to handle synchronization processes in a JMS message broker. I'd be very interested in comments, anecdotes, benchmarks, suggestions in this approach. Thanks, Bill On Oct 27, 2009, at 8:09 AM, Scott Prater wrote: > We'll soon be activating JMS in our Fedora installation, in order to > update our Solr index and memory cache on purge/ingest/update events. > Has anyone had any other experience using a third-party message > broker? > In particular, does anyone have any experience using a message broker > in situations of high traffic or with large datasets or heavy batch > operations? Any recommendations or gotchas to look out for? > > TIA, > > -- Scott > > aj...@virginia.edu wrote: >> I may have missed part of the conversation where you describe this, >> but are you using the JMS gear that comes with Fedora or a separate >> JMS broker (e.g. ActiveMQ)? >> >> --- >> A. Soroka >> Digital Research and Scholarship R & D >> the University of Virginia Library >> >> >> >> On Oct 26, 2009, at 6:56 PM, <carsten.friedr...@csiro.au> wrote: >> >>> · Using the message queue with fedoragsearch to update SOLR >>> seems not to be a good idea for larger datasets. I suspect using the >>> message queue for anything with a large datasets is not a good idea. >>> It seems to consume a lot of resources when running, does not seem >>> to shut down reliably (resulting in very long rebuild time next time >>> you start tomcat) and seems to scale very badly (performance goes >>> down significantly the more objects we add). There are a lot of >>> “seems” in this statement because we didn’t actually measure this >>> and it is only based on our subjective observation. >> >> >> ------------------------------------------------------------------------------ >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >> is the only developer event you need to attend this year. Jumpstart >> your >> developing skills, take BlackBerry mobile applications to market >> and stay >> ahead of the curve. Join us from November 9 - 12, 2009. Register now! >> http://p.sf.net/sfu/devconference >> _______________________________________________ >> Fedora-commons-users mailing list >> Fedora-commons-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > > -- > Scott Prater > Library, Instructional, and Research Applications (LIRA) > Division of Information Technology (DoIT) > University of Wisconsin - Madison > pra...@wisc.edu > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart > your > developing skills, take BlackBerry mobile applications to market and > stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
