On Wed, Sep 6, 2017 at 1:07 PM, Enrico Olivelli <eolive...@gmail.com> wrote:
> Off topic curiosity... Jia and Sijie, do you think we are going to drop ZK > from DL too? > Yes. That's the goal - 1) for large deployment, we are trying to overcome the limitation of zookeeper; 2) for smaller deployments, it will make deployment much easier, you just need to deploy a cluster of bookies. once it is done, you can use ledger api or log stream api to access the bookkeeper cluster. Both DL and BK are metadata storage pluggable. They have very clear interfaces on defining metadata operations. So it is straightforward to use a different metadata storage. > Enrico > > On mer 6 set 2017, 19:51 Enrico Olivelli <eolive...@gmail.com> wrote: > > > > > > > On mer 6 set 2017, 18:25 Sijie Guo <guosi...@gmail.com> wrote: > > > >> On Sep 6, 2017 4:57 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote: > >> > >> Thank you Sijie and Jia for your comments and explanations, > >> answers inline > >> > >> 2017-09-06 2:23 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > >> > >> > Thanks a lot Enrico and Sijie for your comments and information on > this. > >> > > >> > On Tue, Sep 5, 2017 at 9:31 PM, Enrico Olivelli <eolive...@gmail.com> > >> > wrote: > >> > > >> > > Great to see you working on this ! > >> > > I would be great to have such feature, as it is the first step to a > >> > > 'standalone' BookKeeper mode > >> > > > >> > > Some complementary ideas/first look questions: > >> > > - the document does not talk about security, IMHO we have at least > to > >> > cover > >> > > authentication and TLS, it would be great to leverage existing > >> > AuthPlugins, > >> > > as they are based on exchanging byte[] (as SASL wants) > >> > > > >> > [Jia] It is a good idea. We left the security part for now for a few > >> > reasons. 1) Make this BP more focus on removing zookeeper dependencies > >> from > >> > client. 2) It is introduced as a separated implementation of existing > >> > interfaces. So it won’t impact existing security story. And for > sure, > >> We > >> > will add the security part later after this. > >> > > >> > >> > >> I am fine, I am only afraid that we won't be able to support it in the > >> (near) future, > >> maybe you could just only cite the security story and add some reference > >> to > >> how we would deal with it in future > >> > >> > >> The new ledger manager will be first marked as experimental, until it is > >> stable and have security feature. > >> > >> How does that sound? > >> > > > > Ok > > > >> > >> > >> > >> > > >> > - do we have some kind of "bootstrap servers list" configuration > option > >> ? > >> > > the list should be complete or just a subset of bookies ? at > >> connection > >> > the > >> > > client could discover the list of other bookies > >> > > > >> > [Jia] Yes, we will have a `clientBootstrapBookies` settings in the > >> server > >> > set. It can be a list of bookies or just simple a DNS over the > bookies. > >> > Will add this to the BP > >> > > >> > - will the client connect to only one bookie at a time ? how we will > >> deal > >> > > with errors ? > >> > > > >> > [Jia] It will connect the the list of bootstrap servers. gPRC will > load > >> > balance the requests and manage the connection errors. > >> > > >> > - should the bookie write on ZK metadata its gRPC endpoint info ? > (this > >> > > will be useful for a bookie to tell about other bookies to the > >> connected > >> > > clients) > >> > > > >> > [Jia]No, it won’t. We don’t see a strong reason to add it. Especially > >> > eventually we may eliminate zookeeper completely. > >> > It can be a fixed port `3281`, or in a scheduler-based environment, it > >> is > >> > very easy to have a load balancer sitting in front of those bookies. > >> > > >> > >> I think a fixed port is not a good way. > >> You will not be able to run more than one bookie on a single host. > >> > >> We should support: > >> - configurable port > >> - ephemeral port for tests > >> > >> > >> I think what Jia means is a configurable port, but it is a relatively > >> fixed > >> port, which client doesn't discover this port from zookeeper. > >> > > > > Very good > > > >> > >> > >> Ideally I would like to have the local transport option, in order to > have > >> a > >> single JVM, but this is not a blocker problem, as we are running gRPC on > >> netty it should be feasible or we can create some kind of short-circut > >> between the client and the Bookie > >> > >> > >> GRPC supports inprocess channel. So you don't need to use the low level > >> netty settings. > >> > > > > Great > > > > So it sounds all good to me thanks > > > > Enrico > > > > > >> > >> I am OK for not writing this to the bookie metadata, leaving up to the > >> client have a configured list of bookies enabled to metadata operations > >> > >> > >> > >> > >> > > >> > - the bookie will be somehow a proxy for zookeeper, I think that the > >> > > 'watch' part is the more complex, we will have to deal with > >> > reconnections, > >> > > errors....maybe it is worth to write more detail about this > >> > > > >> > [Jia] The `watch` API is using the `streaming` rpc in gRPC. It is a > >> > straightforward proxy behavior, if a connection is broken, the client > >> will > >> > simply retry on watching again. > >> > > >> > > >> > > Minor issues: > >> > > - Maybe you can consider using ledgerId and not ledger_id, like in > >> > > LedgerMetadataFormat we are using lastEntryId > >> > > > >> > [Jia] Thanks, It is a protobuf style. The protobuf will convert > >> `ledger_id` > >> > to `ledgerId`. We don’t need to worry about this. > >> > > >> > >> got it, thanks > >> > >> > >> > > >> > > >> > > -In the "motivation" part you write that the fact the having more > >> clients > >> > > than the number of bookies would be a problem for zookeeper, > actually > >> > > zookeeper is very good at dealing with a huge number of clients. > >> > Actually I > >> > > am always running clusters with 3-5 bookies and 10-100 writing > clients > >> > and > >> > > this has never given troubles > >> > > >> > [Jia] :) Seems “10-100 writing clients” is not “a huge number of > >> clients”. > >> > > >> > >> OK, I agree with you an Sijie, I have no experience of larger clusters > >> > >> > >> > > >> > > > >> > > >> > > >> > > >> > > Future: > >> > > - as bookies will be proxies maybe we should take care not to > >> overwhelm > >> a > >> > > bookie with too many clients > >> > > > >> > [Jia] First, gRPC is based on Netty, the protocol is http2, so the > >> > connection is multiplexed. We don’t need to worry about connection > >> count. > >> > Second, all the bookies are treated equally for the metadata > operations, > >> > gRPC will load balancing the requests across the bookies. We don’t > need > >> to > >> > worry about some bookies are overwhelmed. > >> > > >> > >> gRPC sounds great > >> > >> > >> > > >> > > >> > > - iteration on ledgers, sometimes the clients enumerates ledgers but > >> it > >> > is > >> > > not interested in having all of them, as we are using the bookie as > >> proxy > >> > > maybe some kind of "filter" (at least on custom metadata) would be > >> create > >> > > to limit the number of returned items. Other point I don't know gRPC > >> but > >> > it > >> > > does not seems to be very clear how to 'stop' the iteration > >> > > > >> > [Jia] Thanks, We can add it later. For now, we would like to focus on > >> > adding the features the ledger manager needs. > >> > > >> > >> Yup > >> > >> -- Enrico > >> > >> > >> > > >> > > > >> > > -- Enrico > >> > > > >> > > > >> > > 2017-09-05 15:10 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > >> > > > >> > > > Hi all, > >> > > > > >> > > > I have just posted a proposal to remove zookeeper dependency from > >> > > > bookkeeper client, to make bookkeeper client a thin client: > >> > > > > >> > > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/ > >> > > > BP-16%3A+remove+zookeeper+dependency+from+bookkeeper+client > >> > > > > >> > > > > >> > > > BookKeeper uses zookeeper for service discovery (discovering the > >> > > available > >> > > > bookies in the cluster), metadata management (storing all the > >> metadata > >> > > for > >> > > > ledgers). However it exposes the metadata storage directly to the > >> > > clients, > >> > > > making bookkeeper client a very thick client. It also exposes some > >> > > > problems. > >> > > > > >> > > > This BP explores the possibility of eliminating zookeeper > completely > >> > from > >> > > > client side, to produce a thin bookkeeper client. > >> > > > > >> > > > I will send a patch as soon as we agree on the proposal. > >> > > > > >> > > > > >> > > > Thanks. > >> > > > > >> > > > -Jia > >> > > > > >> > > > >> > > >> > > -- > > > > > > -- Enrico Olivelli > > > -- > > > -- Enrico Olivelli >
