Thanks for the offer to help Milind! We've covered most of the things
that are still missing with Hedwig to make it more production ready. I
mentioned having an engineering team mainly in the context of finding an
internal team here at work to support and develop it. But we now have
it open sourced so hopefully we can get enough interest in people using
it to work on it. Here's a list of missing things in Hedwig.
1. Ops tools including monitoring and administration.
2. Cleaner deployment strategies (just have some hacky bash scripts to
do this now).
3. Non-star topologies including logic on forwarding messages between
regions, tools to redefine your topology on the fly (in case a
datacenter region goes down for some reason), topology validation tool
to make sure all regions are connected.
4. Policies on where we can send messages to. This could be topic based
such as defining a topic containing inclusion/exclusion filters on which
regions it should exist in. Additionally, perhaps a message level
mechanism for this would be useful. e.g. expose this to clients so when
they publish messages, they can state where they want the messages to go
to.
5. More testing to find bugs, stress the system, performance tuning,
etc.
Erwin
On Tue, 2010-11-09 at 12:47 -0800, Milind Parikh wrote:
What are the other features? How can I help? I would love to experiment with
Hedwig. So I could help, in my spare time , with the documentation. Why
would you need an engineering team to support it? You have this open
sourced. Couldn't you use the power of the community?
As you can see, I am pretty eager to try hedwig.
Regards
-- Milind
On Tue, Nov 9, 2010 at 12:00 PM, Erwin Tam e...@yahoo-inc.com wrote:
Hi Milind, Hedwig is still in prototype phase so it lacks several
features to make it production ready. We are trying to find an
engineering team to work on and support it.
(a) Monitoring is one of the big missing pieces to make it production
ready. We have some thoughts on it. The simplest solution would be to
use JMX style bindings for monitoring and admin purposes.
(b) We have designs on non-star topologies and the underlying pieces to
support this are there (version vectors for messages published/consumed
in all regions). This is something we've thought about since the
beginning. However, due to lack of resources, we have not implemented
this yet. There are a few enhancements that we know about and will
shortly document them either via a wiki page or open jiras for Hedwig.
(c) We are in the process of converting some internal documents we have
for Hedwig for third party consumption. That's a priority since we
definitely want people to know quickly what Hedwig is all about (large
scale multi-colo guaranteed delivery pub/sub system) and the
architecture of it (shared nothing, commodity boxes, ZooKeeper for
coordination, Bookkeeper for persistence). We also want to emphasize
that all pieces of Hedwig are modular and you can slot in your own
implementation for the various parts.
Erwin
From: Milind Parikh milindpar...@gmail.com
Reply-To: zookeeper-user@hadoop.apache.org
Date: Fri, 5 Nov 2010 20:08:18 -0700
To: zookeeper-user@hadoop.apache.org
Subject: Key factors for production readiness of Hedwig
What are the key factors for production readiness of Hedwig? I would
really
like to use Hedwig in some major projects; but having a production
readiness
stamp is important. For my perspective:
(a) Monitoring
(b) Deployment Patterns (non star based)
(c) Documentation
Regards
-- Milind
