Il mar 20 feb 2018, 10:57 Sijie Guo <guosi...@gmail.com> ha scritto: > Hi all, > > Hope you guys have gone through the BP. > > I've also made the repo of this table service that we have been working on > public. https://github.com/streamlio/stream-storage > > I would like to call a vote for making this work as a contrib module in > bookkeeper for 4.7 as a preview, so we can continue the development of this > idea for bookkeeper metadata storage in subsequent releases. Like how > bookkeeper was developed from a contrib module in zookeeper before. > > Here is the PR for adding this BP to bookkeeper_proposals page: > https://github.com/apache/bookkeeper/pull/1185 > > Please take a look and vote your opinions. >
So I have to put my +1 on github or will you start an email thread? > > - Sijie > > > > > > > > > > > On Sun, Feb 11, 2018 at 5:07 PM, Sijie Guo <guosi...@gmail.com> wrote: > > > Thanks JV and Encrico. > > > > I would like to include this as a contrib in bookkeeper for 4.7 like > > bookkeeper was grown from a contrib in zookeeper before. > > > > So if the idea sounds good to you guys, and if you guys think this is > > aligned with bookkeeper roadmap, let’s try to move this forward with a > > contrib module in bookkeeper and continue the development in bookkeeper. > > > > If there is no major concerns, I would like to call a vote for this week. > > > > Sijie > > > > > > On Thu, Feb 8, 2018 at 12:01 AM Venkateswara Rao Jujjuri < > > jujj...@gmail.com> wrote: > > > >> A great step to move forward. BP-29 and BP-30 along with reorganizing ZK > >> will help the BK to shape perfect MDS abstraction. > >> While BP-30 is ambitious, it is a perfect way to start ambitious > projects. > >> :) > >> > >> JV > >> > >> On Wed, Feb 7, 2018 at 6:49 AM, Enrico Olivelli <eolive...@gmail.com> > >> wrote: > >> > >> > It is very interesting! Thank you. > >> > I will look into it soon > >> > > >> > Enrico > >> > > >> > Il mer 7 feb 2018, 15:24 Sijie Guo <guosi...@gmail.com> ha scritto: > >> > > >> > > Hi all, > >> > > > >> > > I started a proposal of contributing a table (aka key/value) service > >> > > component as a contrib module to the bookkeeper community. This BP > >> > together > >> > > with other BPs I sent last week forms the idea of how we can do on > >> > > improving metadata management in bookkeeper (I will talk a bit more > at > >> > the > >> > > end of this email). > >> > > > >> > > **why it was developed** > >> > > > >> > > Two main categories of use cases were driving the need of a > key/value > >> > like > >> > > service. > >> > > > >> > > One is metadata storage, bookkeeper needs a key/value like storage > >> > > (currently it is zookeeper) to store the ledger's metadata, systems > >> built > >> > > on top of bookkeeper like distributedlog/pulsar also follow the > >> pattern > >> > > that bookkeeper is using. They all need a key/value like storage to > >> store > >> > > their metadata. We all know zookeeper is the bottleneck of the > >> > scalability. > >> > > And it is also an issue marker to production systems (based on my > >> biased > >> > > production experiences). > >> > > > >> > > The other one is state storage in real-time/streaming > >> > > analytics/computation. In streaming analytics, the computation jobs > >> > usually > >> > > process streaming data. they usually need to store some sort of > state > >> of > >> > > the computation operators into a storage and serve the computation > >> state > >> > as > >> > > final results for queries. Those state are usually represented in > >> > key/value > >> > > forms, and usually backed by wal. BookKeeper has been used in this > >> area > >> > via > >> > > distributedlog/pulsar for storing and serving log / streaming data. > >> It is > >> > > ideal for bookkeeper also able to store and serve state data for the > >> sake > >> > > of unification, simplification and also reducing the complexity of > >> > > deployment and operations. > >> > > > >> > > Hence we prototyped/developed a table service component as an add-on > >> to > >> > > bookkeeper. We'd like to contribute this as a contrib module to > >> > bookkeeper > >> > > and continue the development, integration and evaluation in the > >> > bookkeeper > >> > > community. > >> > > > >> > > We hope this can be like bookkeeper in zookeeper. bookkeeper was a > >> > contrib > >> > > module in zookeeper, and it is developed in the community and grown > >> into > >> > > what it is now. > >> > > > >> > > **how it is aligned with metadata storage** > >> > > > >> > > BP-28, BP-29 and BP-30. They are related at some extend. > >> > > > >> > > BP-28 is more a cleanup proposal to carry-on Jia's work (on service > >> > > discovery interfaces). This is to produce a clean metadata api > module, > >> > > define a clean dependency between > >> > > bookkeeper implementation and metadata service, and allow we really > >> > plugin > >> > > different > >> > > metadata services without touching/changing bookkeeper > implementation. > >> > > > >> > > BP-29 and BP-30 can be thought as two different metadata service > >> > > implementation based > >> > > on the metadata api contract defined in BP-28. > >> > > > >> > > BP-29 is to use Etcd as the metadata service, while BP-30 is to > have a > >> > > built-in key/value service as the metadata service. Both BP-29 and > >> BP-30 > >> > > have pros and cons. However they > >> > > are not against to each other. Allowing two concurrent approaches > will > >> > help > >> > > us understand > >> > > more on metadata management in bookkeeper and its ecosystem (e.g. > >> dlog, > >> > > pulsar), which > >> > > will lead the project head in a healthy direction. > >> > > > >> > > **Proposed Changes** > >> > > > >> > > This proposal is to propose this table service as a contrib module > >> under > >> > > `stream` directory just as how we handle `dlog`. We can mark it as > >> > > "preview"/"alpha" in 4.7 and continue the development of this module > >> in > >> > > bookkeeper community. > >> > > > >> > > The details of the proposal can be found in the google doc attached > >> > below: > >> > > > >> > > > >> > > https://docs.google.com/document/d/155xAwWv5IdOitHh1NVMEwCMGgB28M > >> > 3FyMiQSxEpjE-Y/edit#heading=h.56rbh52koe3f > >> > > > >> > > Please take a look. Comments are welcome. > >> > > > >> > > - Sijie > >> > > > >> > > >> > > >> > -- > >> > > >> > > >> > -- Enrico Olivelli > >> > > >> > >> > >> > >> -- > >> Jvrao > >> --- > >> First they ignore you, then they laugh at you, then they fight you, then > >> you win. - Mahatma Gandhi > >> > > > -- -- Enrico Olivelli
