I just finished working on a branch [1] that uses publish/subscribe to update series metadata in our local distribution channel, even if the series contains recordings that have been previously published.
Before merging this work with the trunk, I want to discuss what's in the branch and what used to be in the branch but was subsequently removed. My goal is to obtain a general consensus that this is an adequate approach to cluster-wide asynchronous eventing. This is a long message, so if you are not interested in Matterhorn's services design, feel free to move on. When I think of pub-sub messaging across a cluster, I think of JMS. So that was how I originally approached the problem. In order to make use of the simplified eventing model of OSGI's EventAdmin, I added a Sakai-inspired JMS to EventAdmin bridge to distribute OSGI events to other nodes in the cluster. This was: a) a bit heavy, as it introduced an embedded activemq broker [2] into each JVM b) required another piece of configuration, so the brokers could find each other over the network c) problematic, in that each node needed to decide whether to take action on an event. Since each service might be deployed on multiple nodes, it wasn't clear which node should act on the event. Fortunately, we already have cluster coordination via the service registry, so there should be a way around JMS style messaging. We should be able to respond to events centrally by using the endpoints that the service registry tells us are online and available. The matterhorn-conductor module, which provides our workflow operation handlers, has all dependencies we need to communicate with (possibly remote) Matterhorn services. By putting event handlers here, we can respond to any workflow or admin-tool related events by calling any of the services in the cluster. As long as the events we need to respond to come from the admin-related services (workflow, series, scheduling) this removes the need for JMS altogether. What we lose is the ability to listen for other kinds of events, such as users adding annotations, or jobs failing. We can deal with the latter via workflow. I don't know of any use cases related to the former, but I'm confident that they can be handled. Another annoyance inherent in this approach is the EventAdmin's tolerance for latency. If an EventHandler takes too long when handling an event, it will be blacklisted, and will not be called again. Long running operations are the rule in Matterhorn, not the exception, so we can either: a) disable timeouts on the event admin. I'd rather not do this, since it's a reasonable expectation that handlers do not block for long periods of time. b) wrap service calls in a Callable or Runnable, using a java.util.concurrent.ExecutorService to execute the event handling logic. See [3] for an example. This isn't hard, but it a little ugly. c) EventHandlers can be JobProducers, storing the details of the event in the database, to be dispatched to a service that can handle it. This is how we handle other kinds of long running jobs, but this would require rest endpoints to handle events. Seems like overkill to me. If you have comments on specific lines of code in the branch, feel free to join the review at [4]. Otherwise, let's discuss here. Thanks, Josh [1] https://opencast.jira.com/svn/MH/branches/MH-7938 [2] http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html [3] http://opencast.jira.com/svn/MH/branches/MH-7938/modules/matterhorn-conductor/src/main/java/org/opencastproject/event/handler/SeriesUpdatedEventHandler.java [4] http://opencast.jira.com/source/cru/CR-MH-324 _______________________________________________ Matterhorn mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn To unsubscribe please email [email protected] _______________________________________________
