> Let me clarify the 3 accesses for a close operation. > > 1) first read the ledger metadata. (1 metadata read) > 2) if the ledger state is in open state, write ledger metadata to > IN_RECOVERY state. (1 metadata write) > 3) do ledger recovery procedure > 4) update last entry id and change state to CLOSED. (1 metadata write) > > I am not very sure about whether we could get ride of step 2). But at > least, we still have 1 read and 1 write for a close operation. (NOTE: close > here is not create a ledger and close it. it is the *close* when open to > recover) You cannot get rid of 2) without sacrificing correctness.
> The big problem for current bookkeeper is applications (either Hedwig or > managed ledger) needs to find extra places to record the ledger ids > generated by bookkeeper. It is not efficient and also duplicates the > metadata storage, especially for a system facing large number of topics or > ledgers, it is Achilles' Heel. I think a better way to handle this would be just to scale the metadata storage along with the system. Part of the problem here is that hedwig is being used in a way which is quite different to what it was first designed for. Originally, it was meant to have a number of long lived subscriptions, over which a lot of data travelled. Now the load has flipped to a large number of short lived subscriptions, over which relatively little data travels. This means that the disk capacity of the bookies isn't fully used. So how about using that for metadata also? We can embed some other or write our own. Anything we would do would need to be persistent, support CAS, and replicated. It would be a fairly hefty project, but it would give us horizontal scalability and reduce the number of moving parts required to provide this scalability. For writing our own, we could have a metadata store that sits inside the bookie, sharing the journal and snapshotting every so often, so it should barely affect performance. > I took care of re-using existing concepts with minmum changes to extend the > api to provide more flexibility and efficiency for applications. I don't > think it violates the principle to make the system a bare minmum. For > example, I don't add cursor concept in ledger. The ledger is still could be > read many times. How to shrink a ledger depends on applications. Hedwig or > managed ledger would record their cursors (subscriber states) and shrink a > ledger when they decided to do that. The ledger can still be read many times, but you have removed the guarantee that what is read each time will be the same thing. -Ivan
