Re: Persistent watch stream?

2010-11-12 Thread Ted Dunning
Persistent watches were omitted from ZK on purpose because of the perceived
danger of not have a load shedding mechanism.

Note that when you get a notification, the query you do to get the next
state typically sets the next watch.  This guarantees that you don't lose
anything, but it may mean that you effectively get notified of multiple
changes in one notification.

If you really, really want all notifications in order, then what you are
really looking for is a kind of a distributed transaction log.  For small
applications, you can implement this by writing logs into ZK znodes.  Your
client should remember where they were and as they are notified, they read
the current file to catch up.  This has the downside that to update a file
you have to completely rewrite it which makes it inconvenient to put a bunch
of stuff into a single chunk of the log.  You would also need a watcher on
the directory to notify you when new log files are created.  Aside from the
slightly quadratic update problem, this does what you need.

You can also check out Bookkeeper which is a more scalable distributed
transaction logging system.  It addresses the complications of (mis) using
ZK as a log system and attempts to give you a reliable and robust
transaction log.

On Fri, Nov 12, 2010 at 7:16 PM, Chang Song tru64...@me.com wrote:


 Hi.

 It's been a couple of weeks since we started integrating zookeeper into our
 app.
 Great piece of software. Cut down LOC by more than half.
 Thank you for open sourcing Zookeeper.

 I have a couple of questions, though.
 I was wondering you have any plan to support persistent watch stream
 feature?

 One-time watch is very inconvenient for us.
 We need to get a stream of notifications in order. We are talking thousands
 of clients.
 Since notification can happen in bulk, we need to set a watch first, and
 once we get a callback,
 we need to check periodically what happens to the children of the watch
 nodes.

 You can consider this as a automatic server-side watch registration
 feature: I'd call this sticky watch.
 I think it is easier to implement this as a sticky watch.

 Another question is if it is possible to coalesce predefined number of
 changes for watch callbacks.
 We have observed that if there are many changes in children node, clients
 gets different number of
 messages in bulks. This makes the architecture of our application from
 event-driven to polling for a while.
 Thus pretty much the same reasoning behind the first question.

 Any comments welcome.

 Thank you.

 Chang






Re: Persistent watch stream?

2010-11-12 Thread Chang Song

Thank you, Ted.

As far as getChildren goes, you are right.
Since you will get the latest children change and at the same time set a watch,
you will definitely not lose any children node change.

Thank you.

Chang


Nov 13, 2010, 12:51 PM, Ted Dunning 작성:

 Persistent watches were omitted from ZK on purpose because of the perceived
 danger of not have a load shedding mechanism.
 
 Note that when you get a notification, the query you do to get the next
 state typically sets the next watch.  This guarantees that you don't lose
 anything, but it may mean that you effectively get notified of multiple
 changes in one notification.
 
 If you really, really want all notifications in order, then what you are
 really looking for is a kind of a distributed transaction log.  For small
 applications, you can implement this by writing logs into ZK znodes.  Your
 client should remember where they were and as they are notified, they read
 the current file to catch up.  This has the downside that to update a file
 you have to completely rewrite it which makes it inconvenient to put a bunch
 of stuff into a single chunk of the log.  You would also need a watcher on
 the directory to notify you when new log files are created.  Aside from the
 slightly quadratic update problem, this does what you need.
 
 You can also check out Bookkeeper which is a more scalable distributed
 transaction logging system.  It addresses the complications of (mis) using
 ZK as a log system and attempts to give you a reliable and robust
 transaction log.
 
 On Fri, Nov 12, 2010 at 7:16 PM, Chang Song tru64...@me.com wrote:
 
 
 Hi.
 
 It's been a couple of weeks since we started integrating zookeeper into our
 app.
 Great piece of software. Cut down LOC by more than half.
 Thank you for open sourcing Zookeeper.
 
 I have a couple of questions, though.
 I was wondering you have any plan to support persistent watch stream
 feature?
 
 One-time watch is very inconvenient for us.
 We need to get a stream of notifications in order. We are talking thousands
 of clients.
 Since notification can happen in bulk, we need to set a watch first, and
 once we get a callback,
 we need to check periodically what happens to the children of the watch
 nodes.
 
 You can consider this as a automatic server-side watch registration
 feature: I'd call this sticky watch.
 I think it is easier to implement this as a sticky watch.
 
 Another question is if it is possible to coalesce predefined number of
 changes for watch callbacks.
 We have observed that if there are many changes in children node, clients
 gets different number of
 messages in bulks. This makes the architecture of our application from
 event-driven to polling for a while.
 Thus pretty much the same reasoning behind the first question.
 
 Any comments welcome.
 
 Thank you.
 
 Chang