Thanks for the offer to help Milind! We've covered most of the things that are still missing with Hedwig to make it more production ready. I mentioned having an engineering team mainly in the context of finding an internal team here at work to support and develop it. But we now have it open sourced so hopefully we can get enough interest in people using it to work on it. Here's a list of missing things in Hedwig.
1. Ops tools including monitoring and administration. 2. Cleaner deployment strategies (just have some hacky bash scripts to do this now). 3. Non-star topologies including logic on forwarding messages between regions, tools to redefine your topology on the fly (in case a datacenter region goes down for some reason), topology validation tool to make sure all regions are connected. 4. Policies on where we can send messages to. This could be topic based such as defining a topic containing inclusion/exclusion filters on which regions it should exist in. Additionally, perhaps a message level mechanism for this would be useful. e.g. expose this to clients so when they publish messages, they can state where they want the messages to go to. 5. More testing to find bugs, stress the system, performance tuning, etc. Erwin On Tue, 2010-11-09 at 12:47 -0800, Milind Parikh wrote: > What are the other features? How can I help? I would love to experiment with > Hedwig. So I could help, in my spare time , with the documentation. Why > would you need an engineering team to support it? You have this open > sourced. Couldn't you use the power of the community? > > As you can see, I am pretty eager to try hedwig. > > Regards > -- Milind > > > On Tue, Nov 9, 2010 at 12:00 PM, Erwin Tam <e...@yahoo-inc.com> wrote: > > > Hi Milind, Hedwig is still in prototype phase so it lacks several > > features to make it production ready. We are trying to find an > > engineering team to work on and support it. > > > > (a) Monitoring is one of the big missing pieces to make it production > > ready. We have some thoughts on it. The simplest solution would be to > > use JMX style bindings for monitoring and admin purposes. > > > > (b) We have designs on non-star topologies and the underlying pieces to > > support this are there (version vectors for messages published/consumed > > in all regions). This is something we've thought about since the > > beginning. However, due to lack of resources, we have not implemented > > this yet. There are a few enhancements that we know about and will > > shortly document them either via a wiki page or open jiras for Hedwig. > > > > (c) We are in the process of converting some internal documents we have > > for Hedwig for third party consumption. That's a priority since we > > definitely want people to know quickly what Hedwig is all about (large > > scale multi-colo guaranteed delivery pub/sub system) and the > > architecture of it (shared nothing, commodity boxes, ZooKeeper for > > coordination, Bookkeeper for persistence). We also want to emphasize > > that all pieces of Hedwig are modular and you can slot in your own > > implementation for the various parts. > > > > Erwin > > > > > > > From: Milind Parikh <milindpar...@gmail.com> > > > Reply-To: <zookeeper-user@hadoop.apache.org> > > > Date: Fri, 5 Nov 2010 20:08:18 -0700 > > > To: <zookeeper-user@hadoop.apache.org> > > > Subject: Key factors for production readiness of Hedwig > > > > > > What are the key factors for production readiness of Hedwig? I would > > > really > > > like to use Hedwig in some major projects; but having a production > > > readiness > > > stamp is important. For my perspective: > > > > > > (a) Monitoring > > > (b) Deployment Patterns (non star based) > > > (c) Documentation > > > > > > Regards > > > -- Milind > > > > > > > > >
