On Wed, Feb 16, 2011 at 3:04 PM, Pieter Hintjens <[email protected]> wrote:
> On Wed, Feb 16, 2011 at 10:41 PM, Matt Stump <[email protected]> wrote: > > > One of the questions that came up is who else is using 0mq and > > for what purpose? Do you have reference cases or success stories that we > > can read about? > > We did collect these but most 0MQ users tend to be discrete about the > technology they choose. > Allow me to be indiscrete then ;-) We're using it at Loggly to stream log events from our data collection app through a custom routing app and into multiple Solr instances for indexing. All written in Java, using 0MQ 2.0.7 for now. It works brilliantly for us, and the only problems we've had have been related to our bad code, where multiple threads were accessing and/or bouncing 0MQ connections. The lesson here is to be very careful to make sure the "one thread at a time" contract is honoured. Overall, I'd say the 0MQ is probably the least troublesome aspect of our overall architecture. It "Just Works" and we're very glad we chose it. I've been recommending it to everyone I can, because when it fits your problem, it also simplifies it. > > > What patterns are people using to ensure > > failover/reliability/proper error handling in case a server goes down. > > It depends on your requirements for reliability, your performance > goals, the scale of your network, and the type of work you are doing. > For reliable RPC, the usual approach is to detect server failure or > timeout, and either resend to another server or retry N times. 0MQ > does not do this directly but there are layers on top that do it, e.g. > RPC layer in the ZFL library[1]. > We're using PIPELINE to connect our apps to one another, and using the HWM stuff to avoid losing data if the receiver goes down. Given how fast data is pushed through 0MQ, we're not worried about losing data, although its theoretically possible. If one of our Solr boxes goes out, data is accumulated in the router, but because we use HWM (i.e. persist to disk), we can bring up a new Solr instance at our liesure. I should say that we can also dynamically reroute data to alternate Solr boxes if we need to, and thus avoid having to bring up a new box even longer. We're about to start using PUBSUB for a (to be released) real time streaming feature in our product. I played with this when we were initially looking at 0MQ for our primary data connections (where we're currently using PIPELINE) but because you must have your SUB running before you start pushing to PUB, we had to abandon it. For the live streaming stuff, this isn't an issue. > > > Which serializer do most people use? We were most likely going to use > > protobuf and optionally json. > > Again, depends significantly on your use case. Protobufs has the > advantage of performance over JSON, but there are half a dozen good > choices. In any case you should abstract the message encoding/decoding > from your apps so you can change you mind at any stage. > We're using a custom (but very simple) serialization, but are considering switching to something that would make it easier for non-Java clients to use the data. Its not a pressing need for us, so we're really just talking about it at the moment and (slowly) looking at the various alternatives. > > > What are common deployment strategies or best > > practices? Is there anything we should shy away from doing (rough > edges)? > > A common trap is to over-architect before you know the tool well. So a > good practice is to use 0MQ in some small but real case, and then a > somewhat larger one, and then only for your real (mission critical) > case. > Our system is 100% dependent on 0MQ, so I guess we're the opposite end of this spectrum :-) One thing we've been very cautious about is upgrading from version X to version Y. The last time we did it (from 2.0.5 to 2.0.7 if my memory serves me well) was completely painless, but did involve bouncing every app that used 0MQ. If you can get away with doing that, then upgrades are easy, provided you test ahead of time :-)
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
