Hi, our first release used ActiveMQ and a plethora of frameworks.
We soon realize that it is was killing us. Our product runs on small clusters or should I say on clusters of small footprint nodes. For intra cluster stuff, external messaging softwares are to us an overkill so far. They come with an impressive set of functions but we need a few only. New release goes out by end of summer and uses zookeeper to implement shared queues, shared locking and configuration management. We use zookeeper-clj to interface with the servers. We now send Clojure expressions (units of work) instead of YAML messages. For now this layer is "alien" framework free. No more hippo traps until we can see if we will benefit from an external frameworks. Performance is now very good, previously we had several AMQ redundant monkey chains committing to disk the same message streams several times, now it's faster and much more straightforward to test/debug. It's also part of an effort to get rid of that diplodocus Java, the remaining Java components in the nearly ready release are using Clojure to connect to everything else. We got Spring out, Terracotta, AMQ, Hibernate is to get out in late fall, ... The whole elephant herd . In the next planned future release we will use Avout to implement shared logic between nodes. This logic is to increase parallelism. We have specific requirements hardly met by mammoths like AMQ or at the expense of more complexity. By next November, Java is out and a few other rhinos. The remaining software will be around 30k loc. Some of that is data. Then we start to work on innovative stuff... Luc > I'm curious as to what people are using for workers/queues. We're currently > using storm, but im interested in evaluating alternatives. > > Things I like about storm: > * Upload a jar and it automatically distributes the code to all worker > nodes. > * Easy to write/reuse topologies without horribly distorting my code. > * Easy to visualize data as it flows through multiple queues. > * Easy to do fail/retry. > > Things I don't like about storm: > * Can be somewhat flaky when performing tests, things fail/succeed in > unpredictable ways but work in production or vice versa. > * Can't distinguish between failure (exception) and timeout. > * Must dedicate resources full time from a limited worker pool even through > that queue is only used sporadicly. > * Can't link storm directly which means I need to > maintain separate projects. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to [email protected] > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- Softaddicts<[email protected]> sent by ibisMail from my ipad! -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en
