Great to see you working on this ! I would be great to have such feature, as it is the first step to a 'standalone' BookKeeper mode
Some complementary ideas/first look questions: - the document does not talk about security, IMHO we have at least to cover authentication and TLS, it would be great to leverage existing AuthPlugins, as they are based on exchanging byte[] (as SASL wants) - do we have some kind of "bootstrap servers list" configuration option ? the list should be complete or just a subset of bookies ? at connection the client could discover the list of other bookies - will the client connect to only one bookie at a time ? how we will deal with errors ? - should the bookie write on ZK metadata its gRPC endpoint info ? (this will be useful for a bookie to tell about other bookies to the connected clients) - the bookie will be somehow a proxy for zookeeper, I think that the 'watch' part is the more complex, we will have to deal with reconnections, errors....maybe it is worth to write more detail about this Minor issues: - Maybe you can consider using ledgerId and not ledger_id, like in LedgerMetadataFormat we are using lastEntryId -In the "motivation" part you write that the fact the having more clients than the number of bookies would be a problem for zookeeper, actually zookeeper is very good at dealing with a huge number of clients. Actually I am always running clusters with 3-5 bookies and 10-100 writing clients and this has never given troubles Future: - as bookies will be proxies maybe we should take care not to overwhelm a bookie with too many clients - iteration on ledgers, sometimes the clients enumerates ledgers but it is not interested in having all of them, as we are using the bookie as proxy maybe some kind of "filter" (at least on custom metadata) would be create to limit the number of returned items. Other point I don't know gRPC but it does not seems to be very clear how to 'stop' the iteration -- Enrico 2017-09-05 15:10 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > Hi all, > > I have just posted a proposal to remove zookeeper dependency from > bookkeeper client, to make bookkeeper client a thin client: > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/ > BP-16%3A+remove+zookeeper+dependency+from+bookkeeper+client > > > BookKeeper uses zookeeper for service discovery (discovering the available > bookies in the cluster), metadata management (storing all the metadata for > ledgers). However it exposes the metadata storage directly to the clients, > making bookkeeper client a very thick client. It also exposes some > problems. > > This BP explores the possibility of eliminating zookeeper completely from > client side, to produce a thin bookkeeper client. > > I will send a patch as soon as we agree on the proposal. > > > Thanks. > > -Jia >
