I am working on updating and refactoring the messaging client so I hope we can keep this thread going. The goal of the current Fedora installer is to make it as easy as possible to get started and includes a basic messaging capability. However, as soon as you want to scale or take control of messaging to incorporate your own applications, for high availability or any of the myriad reasons for using messaging you will likely want install your own broker and possibly JNDI. The current messaging is pure JMS and should work with any product though ActiveMQ seems very solid and the price is right.
We are consolidating the workflow-like and messaging Wiki discussions to the Fedora Create space. Thorny Staples has asked me to help garden this area but the discussion needs to come from everybody with a stake in using middleware. We have barely tapped the capabilities of middleware products. In addition to this thread, can we collect or at least get pointers on your experiences? It would be great to nail down additional features and services. Since a lot of this is middleware and services can be delivered outside the Fedora Repo core it should be easier to share stuff. I believe quite a few use-cases can be accomplished using simple message queues (see Amazon SQS) and the same middleware base can be used to move gracefully to full workflow systems for more complex use-cases. It would also be nice to work on performance tests. We have been using The Grinder framework but it would be great to share tests from any source. I will attach some basic (non messaging) tests to the Wiki this week as a start. -- Daniel W. Davis Cornell University Fedora Commons Affiliate http://fedora-commons.org dda...@fedora-commons.org Bill Parod wrote: > Hi Scott, > > On Oct 27, 2009, at 10:34 AM, Scott Prater wrote: > > >> Hi, Bill -- >> >> Thanks for your detailed description. You're quite a ways further >> down the workflow path than we are right now. >> > > I don't know. We're just starting to look at the work flow aspect of > this, in terms of event analysis. I'd be interested in your work > there. It seems like a first pass event analysis is in noting when > control is passed between person and system or between systems. That's > a casual observation. I know there is a lot of work on event analysis > which I haven't consulted yet. We're just starting. > > >> I'd like to talk with you more offline about the details of your >> setup. >> >> > > Yes, I'd like to talk with you too. We're overdue. > > >> As I understand it, so far, you've been using message queues just to >> trigger *Fedora* events, correct? >> > > Yes, if the emphasis here is on "just to trigger". If your emphasis in > on "Fedora", that's not entirely true. The handling of an image ingest/ > update also results in image conversion (to jpeg2000) and transfer of > that delivery format to the delivery service (Aware JPEG2000 server on > a different box). The source image is similarly transferred to > archival storage, though not converted. None of that affects Fedora, > necessarily. We could stop there and have a jp2 image management > system. Though because we use Fedora as the object framework that ties > these pieces together around an identifier, datastream references to > those files (and inline extracts of their technical metadata) are > stored in and made available from Fedora. So beyond the image > handling, foxml is crated and ingested. The message broker is intended > to manage events that affect a variety of systems/components that > participate in collection management - not just Fedora, though Fedora > is a key component. > > >> Do you have any situations where a Fedora event is triggering >> outside actions? >> >> > > Not yet. We're just getting started and so far in most cases we're not > editing data directly in Fedora, so change events are mostly triggered > from other systems. But we may trigger object indexing from Fedora JMS > or as part of the central JMS broker. > > I'd love to catch up when you have the time. > > Bill > > > > >> -- Scott >> >> Bill Parod wrote: >> >>> Hi Scott, >>> I don't have numbers for you at this point, but we've been using >>> ActiveMQ as a queueing / message broker to loosely couple systems >>> that participate, along with Fedora, in various collection >>> management activity. We're planning to ingest a collection of 50k >>> images and can report on how that goes by the end of the year, >>> probably in November. >>> Here's a longer description of broader context and what we've done >>> so far in case of interest. I'd be very interested in comments or >>> similar experience you or others have. >>> The general idea is to let various systems report significant >>> events via JMS with the event handling to be loosely coupled (some >>> operations take a while), extensible and flexibly/centrally managed >>> externally to the systems. The motivation is that we have various >>> systems contributing repository content (Drupal, Archon, and >>> Voyager for starters), we need to synchronize metadata updates >>> (descriptive metadata and also authority references) and want to >>> minimize the system-specific customization necessary for systems to >>> communicate/participate. >>> For example, we have started to use Drupal for creating repository >>> content, using a CCK content type to enter/edit media file paths >>> and repeating Dublin Core fields. We use a modified version of the >>> MsgQueue Drupal module to queue insert/update/delete node_api >>> events to ActiveMQ for Drupal configured CCK types of interest. >>> Currently the message includes the node and CCK data, as the >>> initial content types are simple and relatively small (Dublin >>> Core). The JMS handler for these events invokes ingest scripts on a >>> conversion/ingest-preprocessing server that obtain the media >>> (images currently), JHove validate it, extract technical metadata, >>> convert to jp2, push the jp2 to image server, push the source file >>> to archive server, generate foxml with datastreams (DC, MODS, SVG) >>> and ingest into Fedora. In this scenario Drupal doesn't have to >>> know about Fedora or how to invoke the various ingest >>> preprocessing. It just reports events of interest. We plan to >>> expand event handling to include solr indexing and preservation >>> metadata capture. >>> We also use the Drupal Node Import module to trigger such ingests >>> from a spreadsheet. We're thinking about modifying this module to >>> take fields from an ODBC/JDBC source and from file systems >>> structures (directories of images for example). We're planning to >>> test the ingest of a collection of 50K images in this way. We can >>> report on how that goes. >>> We have a slightly different pattern for EAD objects created/edited >>> in Archon. Since finding aid EAD data is more complex than DC, in >>> this case Archon will just send a message to the JMS broker that an >>> Archon collection should be synchronized to Fedora. The JMS handler >>> will turn around and make a request to Archon for the collection as >>> an EAD export and ingest/update appropriately into Fedora. Haven't >>> dine this yet, but it's due by the end of this week.:) It's almost >>> there. >>> We've been talking about synchronizing authority data similarly, >>> for example when we get updates from OCLC for our local name >>> authority file in Voyager that should propagate to affected records >>> in Archon, Drupal, and Fedora. We're still talking this through, >>> but it's part of the motivation to handle synchronization processes >>> in a JMS message broker. >>> I'd be very interested in comments, anecdotes, benchmarks, >>> suggestions in this approach. >>> Thanks, >>> Bill >>> On Oct 27, 2009, at 8:09 AM, Scott Prater wrote: >>> >>>> We'll soon be activating JMS in our Fedora installation, in order to >>>> update our Solr index and memory cache on purge/ingest/update >>>> events. >>>> Has anyone had any other experience using a third-party message >>>> broker? >>>> In particular, does anyone have any experience using a message >>>> broker >>>> in situations of high traffic or with large datasets or heavy batch >>>> operations? Any recommendations or gotchas to look out for? >>>> >>>> TIA, >>>> >>>> -- Scott >>>> >>>> aj...@virginia.edu wrote: >>>> >>>>> I may have missed part of the conversation where you describe this, >>>>> but are you using the JMS gear that comes with Fedora or a separate >>>>> JMS broker (e.g. ActiveMQ)? >>>>> >>>>> --- >>>>> A. Soroka >>>>> Digital Research and Scholarship R & D >>>>> the University of Virginia Library >>>>> >>>>> >>>>> >>>>> On Oct 26, 2009, at 6:56 PM, <carsten.friedr...@csiro.au> wrote: >>>>> >>>>> >>>>>> · Using the message queue with fedoragsearch to update >>>>>> SOLR >>>>>> seems not to be a good idea for larger datasets. I suspect using >>>>>> the >>>>>> message queue for anything with a large datasets is not a good >>>>>> idea. >>>>>> It seems to consume a lot of resources when running, does not seem >>>>>> to shut down reliably (resulting in very long rebuild time next >>>>>> time >>>>>> you start tomcat) and seems to scale very badly (performance goes >>>>>> down significantly the more objects we add). There are a lot of >>>>>> “seems” in this statement because we didn’t actually measure this >>>>>> and it is only based on our subjective observation. >>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Come build with us! The BlackBerry(R) Developer Conference in SF, >>>>> CA >>>>> is the only developer event you need to attend this year. >>>>> Jumpstart your >>>>> developing skills, take BlackBerry mobile applications to market >>>>> and stay >>>>> ahead of the curve. Join us from November 9 - 12, 2009. Register >>>>> now! >>>>> http://p.sf.net/sfu/devconference >>>>> _______________________________________________ >>>>> Fedora-commons-users mailing list >>>>> Fedora-commons-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>>> >>>> -- >>>> Scott Prater >>>> Library, Instructional, and Research Applications (LIRA) >>>> Division of Information Technology (DoIT) >>>> University of Wisconsin - Madison >>>> pra...@wisc.edu >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA >>>> is the only developer event you need to attend this year. >>>> Jumpstart your >>>> developing skills, take BlackBerry mobile applications to market >>>> and stay >>>> ahead of the curve. Join us from November 9 - 12, 2009. Register >>>> now! >>>> http://p.sf.net/sfu/devconference >>>> _______________________________________________ >>>> Fedora-commons-users mailing list >>>> Fedora-commons-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>> >>>> >> -- >> Scott Prater >> Library, Instructional, and Research Applications (LIRA) >> Division of Information Technology (DoIT) >> University of Wisconsin - Madison >> pra...@wisc.edu >> >> > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
