A great step to move forward. BP-29 and BP-30 along with reorganizing ZK will help the BK to shape perfect MDS abstraction. While BP-30 is ambitious, it is a perfect way to start ambitious projects. :)
JV On Wed, Feb 7, 2018 at 6:49 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > It is very interesting! Thank you. > I will look into it soon > > Enrico > > Il mer 7 feb 2018, 15:24 Sijie Guo <guosi...@gmail.com> ha scritto: > > > Hi all, > > > > I started a proposal of contributing a table (aka key/value) service > > component as a contrib module to the bookkeeper community. This BP > together > > with other BPs I sent last week forms the idea of how we can do on > > improving metadata management in bookkeeper (I will talk a bit more at > the > > end of this email). > > > > **why it was developed** > > > > Two main categories of use cases were driving the need of a key/value > like > > service. > > > > One is metadata storage, bookkeeper needs a key/value like storage > > (currently it is zookeeper) to store the ledger's metadata, systems built > > on top of bookkeeper like distributedlog/pulsar also follow the pattern > > that bookkeeper is using. They all need a key/value like storage to store > > their metadata. We all know zookeeper is the bottleneck of the > scalability. > > And it is also an issue marker to production systems (based on my biased > > production experiences). > > > > The other one is state storage in real-time/streaming > > analytics/computation. In streaming analytics, the computation jobs > usually > > process streaming data. they usually need to store some sort of state of > > the computation operators into a storage and serve the computation state > as > > final results for queries. Those state are usually represented in > key/value > > forms, and usually backed by wal. BookKeeper has been used in this area > via > > distributedlog/pulsar for storing and serving log / streaming data. It is > > ideal for bookkeeper also able to store and serve state data for the sake > > of unification, simplification and also reducing the complexity of > > deployment and operations. > > > > Hence we prototyped/developed a table service component as an add-on to > > bookkeeper. We'd like to contribute this as a contrib module to > bookkeeper > > and continue the development, integration and evaluation in the > bookkeeper > > community. > > > > We hope this can be like bookkeeper in zookeeper. bookkeeper was a > contrib > > module in zookeeper, and it is developed in the community and grown into > > what it is now. > > > > **how it is aligned with metadata storage** > > > > BP-28, BP-29 and BP-30. They are related at some extend. > > > > BP-28 is more a cleanup proposal to carry-on Jia's work (on service > > discovery interfaces). This is to produce a clean metadata api module, > > define a clean dependency between > > bookkeeper implementation and metadata service, and allow we really > plugin > > different > > metadata services without touching/changing bookkeeper implementation. > > > > BP-29 and BP-30 can be thought as two different metadata service > > implementation based > > on the metadata api contract defined in BP-28. > > > > BP-29 is to use Etcd as the metadata service, while BP-30 is to have a > > built-in key/value service as the metadata service. Both BP-29 and BP-30 > > have pros and cons. However they > > are not against to each other. Allowing two concurrent approaches will > help > > us understand > > more on metadata management in bookkeeper and its ecosystem (e.g. dlog, > > pulsar), which > > will lead the project head in a healthy direction. > > > > **Proposed Changes** > > > > This proposal is to propose this table service as a contrib module under > > `stream` directory just as how we handle `dlog`. We can mark it as > > "preview"/"alpha" in 4.7 and continue the development of this module in > > bookkeeper community. > > > > The details of the proposal can be found in the google doc attached > below: > > > > > > https://docs.google.com/document/d/155xAwWv5IdOitHh1NVMEwCMGgB28M > 3FyMiQSxEpjE-Y/edit#heading=h.56rbh52koe3f > > > > Please take a look. Comments are welcome. > > > > - Sijie > > > > > -- > > > -- Enrico Olivelli > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
