Any thoughts or comments on this. :) If not, would like to mark this BP approved. And we will prepare an initial PR for detailed discussion and comments.
On Wed, Sep 13, 2017 at 4:49 PM, Sijie Guo <guosi...@gmail.com> wrote: > On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > > > > > > > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > > > >> Thanks a lot for your time Yiming and Enrico. :) > >> > >> Regarding the security, we could do it in a separate BP, and make this > BP > >> more focus on filling up the useful endpoints. How about it? > >> > > > > OK, but please consider that the more utility you add to the endpoint the > > more users will want to enable it and so security will be a blocker issue > > > > My take on this - we all know security is important for any communication > to bookies. > > However, security is a big different scope of problems to address. we > shouldn't put everything in one big BP. I'd suggest us focusing on the > problems > that a BP tends to address, defer other things to separate BPs. Otherwise > we can't land any changes quickly. > > > > > > > >> > >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eolive...@gmail.com> > >> wrote: > >> > >> > Hi Jia, > >> > I am OK with the idea of having standard HTTP API, this will help > >> > development of (non-Java) tools. > >> > > >> > It is not clear to me if we are going to add an http API useful for > >> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper. > >> > I am referring to the fact that in the proposal there are API calls to > >> > create ledgers and to read data. > >> > I think we should separate this two aspects and maybe it is better to > >> > address the 'bookie management' first, which is the work that Yiming > >> > started. > >> > > >> [jia] It is targeting for the admin portal. the existence of the ledger > >> api > >> is just to simplify debugging or operations. we can eliminate the > ‘create’ > >> endpoint. > >> > > > > OK so are you already thinking about specific tools to manage bookies > > without the bookie shell > > > > We would like to integrate this for schedulers (e.g. k8s). so we can deploy > and operate bookkeeper easily > in those environments. > > > > > > > >> > >> > >> > > >> > Another point very important to me is that if we are going to > introduce > >> > management operations via http we have to take into account security, > at > >> > least TLS and some kind of authentication. > >> > About TLS it is very simple to achieve this, every HTTP server > supports > >> TLS > >> > (https) > >> > About authentication the problem is not so simple, Kerberos on HTTPs > is > >> > very complex, but we need to introduce some auth mechanism. > >> > IMHO We can require the 'http server implementation' to implement > >> security > >> > but out of the box we have to supply basic support at least for one > >> > provider bundled in the distribution package. > >> > > >> [jia] The BP focuses on filling up the useful endpoints. The security > will > >> be a separate BP. > >> > >> > >> > > >> > Some API can return very large result sets like 'list ledgers', > actually > >> > the HTTP Server subsystem exchanges strings in memory, we will need to > >> > introduce some more smart way because it would be easy to bring down > the > >> > bookie just by calling that API multiple times concurrencly (it is > just > >> an > >> > example) > >> > > >> [jia] We could add pagination into all the `list` api. > >> > > > > Yes but it will be really tricky to implement, but please do it > > > > > >> > >> > > >> > IMHO The 'create ledger' API is not useful, due to the design of BK > you > >> > have to create a ledger and then write immediately to it, I think that > >> such > >> > an API should allow the client to stream the contents of the ledger in > >> the > >> > HTTP body at least. But I think that a more stateful http API needs to > >> be > >> > designed to implement a pure Http client > >> > > >> [jia] Thanks, we are not planning to implement an http client. we will > >> remove ‘create’ here. > >> > > > > OK, thanks > > > >> > >> > >> > > >> > Maybe we should add a more narrowed motivation and add the only useful > >> APIs > >> > to address those issues. > >> > For instance in my company we would like to start creating a > BookKeeper > >> Web > >> > UI, so we need some way to talk to bookies and exchange data, but in > >> this > >> > case I am interested in asking to bookies their status and the > effective > >> > contents of the bookie > >> > > >> [jia] The BP proposes adding a standard naming convention for adding > admin > >> endpoints. Feel free to propose the endpoints you would like to appear > in > >> the http admin portal. > >> > > > > The ones you wrote are enough complete for me, I would like to have > > read-only operation in order to have a global view on the status of the > > system. > > > > -- Enrico > > > > > > > >> > >> > >> > > >> > -- Enrico > >> > > >> > > >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>: > >> > > >> > > Sure, I think the current HTTP endpoints in Twitter are only > designed > >> for > >> > > Twitter specific, such as check quorum loss, check rack/region > >> diversity. > >> > > So the endpoints convention in Twitter are not the same as in the > >> > proposal. > >> > > I think it would be great to have an agreement on the API naming > >> design, > >> > so > >> > > I like the API design in the proposal, I think the proposal looks > >> good to > >> > > me. > >> > > > >> > > Besides, we're currently only using GET functionalities in Twitter, > >> but I > >> > > notice there're a lot of POST and PUT APIs in the proposal which > could > >> > > change the bookie state or trigger some heavy workload. These APIs > >> looks > >> > a > >> > > bit risky to me if we don't have any authentication enabled (in > >> Twitter). > >> > > > >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <guosi...@gmail.com> > >> wrote: > >> > > > >> > > > + Yiming > >> > > > > >> > > > Yiming, if you have time, please take a look at this BP. let's see > >> if > >> > > > there are any conflicts with those you added for autorecovery. > >> > > > > >> > > > - Sijie > >> > > > > >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zhaiji...@gmail.com> > >> wrote: > >> > > > > >> > > >> Hi all, > >> > > >> > >> > > >> Based on Github #278 <https://github.com/apache/boo > >> kkeeper/pull/278>, > >> > > I > >> > > >> have just posted a proposal regarding define BookKeeper public > http > >> > > >> endpoints: > >> > > >> > >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP- > >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints > >> > > >> > >> > > >> > >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278> > >> > introduces > >> > > >> BookKeeper Http Endpoint module. However there are only two > >> endpoints, > >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in > >> #278. > >> > > In > >> > > >> order to fully leverage the http modules, The goal of this BP is > to > >> > add > >> > > >> more endpoints to this modules. > >> > > >> > >> > > >> Any comments are welcome and appreciated. > >> > > >> > >> > > >> > >> > > >> Thanks. > >> > > >> > >> > > >> -Jia > >> > > >> > >> > > > > >> > > > > >> > > > >> > > >> > > > > >
