Sure, well said. Don't get me wrong when I'm kidding Ted: I strongly believe that Kafka is misusing ZooKeeper for their event processing, although I don't know the details, just read the linked post on their wiki. Details might be important here, because at first glance it seemed to me that they expect a functionality in ZooKeeper which they suppose to provide.
Back to the subscriber topic: I'm planning to resurrect the patch that Jordan is referring to. I think it would be a great new feature in 3.6 or 4.0. Andor -----Original Message----- From: Ted Dunning <ted.dunn...@gmail.com> Reply-To: dev@zookeeper.apache.org To: dev@zookeeper.apache.org Subject: Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum Date: Tue, 13 Aug 2019 12:00:36 -0700 On Tue, Aug 13, 2019 at 10:41 AM Andor Molnar <an...@apache.org> wrote: > Hmmm... and what should Kafka do if it wants to see all events? > UseRabbitMQ? :) Kafka should not be using ZK to store messages. And synchronization ofconfiguration doesn't require message passing. Read the original ZK paper [1] to understand that coordination and reliablestate synchronization in ZK is not message passing. The point is aboutproviding primitives to allow a read cache of roughly coherent values withgood bounds on the semantics. Kafka should use ZK to coordinate who ismaster and should use its own mechanisms to pass messages. A key point isthe performance that can be gained by using a cache coherence strategyrather than an "all updates" strategy. Look at the Omega scheduler[2] paper to see the motivation behind thesmall-steps style of scheduling and resource allocation. Seriously, it you are doing things like designating where replicas go andwho has the baton, it is critical to fast forward to things as they are nowrather than things as they were. When the load hits the fan, it isimportant to be able to ignore the water under the bridge. This sort of topic arises in lots of other places. People complain that itis hard to get synchronized and atomic updates to multiple files in adistributed file system. Moral: don't use a file system when asynchronization tool like ZK is needed. People complain about message passing systems that they are slow in termsof total write speed (can't write 100's of GB/s) and don't allow native andefficient reading of state from a large data structure. Moral: don't use amessage store when you really want a file store. People complain the ZK's watches only give an update to latest state sothey have to put messages into independent znodes and performance and scalegoes to crap. Moral: don't use a synchronization system to store messages. Getting architectural basics close to write goes a long way. [1] https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf [2] https://ai.google/research/pubs/pub41684
