Il gio 27 lug 2017, 17:27 Venkateswara Rao Jujjuri <[email protected]> ha scritto:
> We have inMemoryMetaStore.java already to store metadata locally isn't it? > there are some tests that make use of that too. So if you don't need ZK for > consensus > and you can use this inMemoryMetaStore for metadata store. > MetadataStore should share data between clients and server, as InMemoryMetaStore is only in memory it is not useful. We need a metadatastore on the bookie, with data persistence and memory bounds, and on the client side we will access those metadata thru the bookie. Enrico > On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <[email protected]> wrote: > > > it might be better to call it 'local model' or 'standalone mode', rather > > than 'zookeeper-less'. > > > > On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <[email protected]> > > wrote: > > > > > Hi, > > > I would like to share this idea with you before starting a real design > > > effort for this new configuration. > > > > > > > Sometimes I have to run Bookie + BookKeeper inside the same JVM, > > > essentially because I want my applications to run both in "cluster > mode" > > > and in "single instance mode". > > > I have some other use case of embedding the Bookie, but this is not the > > > scope of this email. > > > > > > In order to run BookKeeper we need ZooKeeper and running ZooKeeper > inside > > > the same process is tricky and you know, it is not the suggested > > > configuration, due to several aspects (out of the scope of this email > > > too....) > > > > > > I would like to make it run without ZooKeeper at all. > > > This idea will support my use case (Bookie + Client inside the same > JVM) > > > and the single Bookie use case, which is a feature that could be useful > > in > > > general. > > > Some other projects, like HBase, have "single server" vs "multi server" > > > setup, and so I think it will be a good idea for the project and for > > > gathering more interest and use cases. > > > > > > We are using ZooKeeper for (at least) these purposes: > > > - bookie discovery > > > - ledger metadata management > > > - auditor leader election > > > - global configuration settings (like autorecovery > enabled/disabled....) > > > > > > my case is to have only one Bookie, so things are really simpler, no > > > auditor, no shared configuration, no underreplication/overreplication, > > no > > > need for 'discovery' > > > > > > In order to address the bookie discovery we can just have some > > client-side > > > configuration to set the address of the Bookie and have some special > > > PlacementPolicy which only returns the same bookie. > > > > > > The big thing is to implement the ledger metadata management, because > > both > > > the Bookie and the client must deal with it. > > > My idea is to add to the Bookie protocol all the functions to access > > > LedgersMetadata (create/list/update with CAS.....), this RPCs will call > > the > > > underlying LedgerManager > > > > > > > I think this is in general very good approach to take. It will make the > > bookie client not depends on metadata store directly and delegating the > > heavy work to the server (bookie side). Once the client is thin, it is > easy > > to implement clients in other languages. > > > > However, I do not think it is necessarily for your use case. You can make > > this as a separate topic. For RPC, I would suggest exploring gPRC (it is > > protobuf + netty), that would be much better for metadata rpcs rather > than > > implementing the logic in bookie protocol. > > > > > > > > > > On the client side we will not use ZK but execute RPCs on the Bookie > > > endpoint (a special LedgerManager which do not use ZK but uses RPCs) > > > > > > On the Bookie side we will have a local ledger metadata management > which > > > persists data on disk, we could use existing Key-Value stores, like > > RocksDB > > > or MapDB, or what ever we prefer, we need something which supports CAS > > and > > > eventually let us put bounds on used memory. > > > > > > > In your use case, you might be able to just implement a ledger manager > > using an embedded k/v. > > > > > > > > > > Does this make sense to you ? > > > > > > -- Enrico > > > > > > > > > -- > Jvrao > --- > First they ignore you, then they laugh at you, then they fight you, then > you win. - Mahatma Gandhi > -- -- Enrico Olivelli
