I’d like to share (and get feedback on) an early pass of scalable Clojure-based forward-chaining rules over Storm, distributing its working memory across the cluster. Storm lays a great foundation, but some problems benefit from higher-level abstractions. I’ve found forward-chaining rules to be a good model for projects I’ve been working on and wanted to bring them to scalable systems like Storm.
The approach is based on Clara, a flexible Clojure-based rules engine. [1,2] Clara uses a pluggable working memory and transport for sharing data, so I wrote an implementation that uses Storm for both. The source code is at [3], and a detailed description of the rationale and approach are at [4]. This approach doesn’t really look like an idiomatic use of Storm, so I wanted to see if there were potential issues I’m overlooking. The post at [4] goes into depth, but this is essentially implemented as a single bolt type that is deployed with high parallelism, creating a number of “siblings”, each of which is responsible for part of the working memory. The siblings then send messages to each other as necessary to perform logical joins of related data. At this point it’s basically a proof-of-concept, but this could grow into a rich model for computation over Storm if the approach makes sense. I appreciate the thoughts! -Ryan [1] http://www.toomuchcode.org/2013/09/rules-as-control-structure.html [2] https://github.com/rbrush/clara-rules [3] https://github.com/rbrush/clara-storm [4] http://www.toomuchcode.org/2013/11/crossing-data-streams-scalable-realtime.html
