see comments in line. On Fri, Jan 18, 2013 at 12:54 AM, Flavio Junqueira <[email protected]>wrote:
> When you say that 'create a ledger and record the ledger id in some other > places' is bad, I assume that you're referring to the number of metadata > accesses. The pattern itself is not bad; BookKeeper provides a ledger > abstraction, the application designer decides the best way of using those > ledgers. Storing the ledgers elsewhere is a common pattern indeed, but > again, managed ledgers have been proposed in my understanding to cover > common patterns like this one. > metadata accesses is one thing. leaking ledger id and causing zombie metadata is another thing. for managed ledger, I explain it in my previous email. > > About the use of names instead of longs, I believe one reason we did it is > that we didn't initially expect users to manipulate ledgers directly, so > programs manipulate the ledgers. Consequently, doing things like comparing > two ledger ids is more efficient with longs. Granted that for > administration perhaps human readable names could be easier. > > > Removing metadata accessing means > > you could not change ledgers, a better way to do that is to 'shrink' a > > ledger. > > I think this is a great point, but can we actually review why publishes > are causing metadata accesses? This should only happen upon topic creation, > acquisition, or when rolling the log, no? For the first one and for the > particular application I believe we are discussing, we shouldn't be > creating and destroying topics as frequently as we are. Topic acquisition > is a problem in the case of crashes, but for topic acquisition, we'll need > at least one access to the metadata anyway, although we could amortize an > access across many ledgers if we have groups. For the last one, I believe > we can increase the frequency of rolling. > reasons explained in previous email. one more sentence about 'ledger id', for 'ledger id' style, there are two metadata, one is ledger metadata and the other one is application metadata (e.g. hedwig persistence info). Even you could group ledger metadata, if you don't group application metadata, it is still 'linear'. if we could consolidate two metadata into one, then applying some kind of group ideas, it might really get rid of 'linear'. > > -Flavio > > On Jan 18, 2013, at 7:56 AM, Sijie Guo <[email protected]> wrote: > > > As I pointed, the style 'create a ledger and record the ledger id in some > > other places' is bad, especially changing ledger would cause 2 metadata > > writes and it is in the publish path, which caused latency spike (the > > latency spike depends on the two metadata writes latency). The reason > why I > > proposed using 'ledger name', 're-open' and 'shrink', is trying to remove > > any metadata accesses during publishes. Removing metadata accessing means > > you could not change ledgers, a better way to do that is to 'shrink' a > > ledger. > > > > -Sijie > > > > On Thu, Jan 17, 2013 at 7:26 AM, Ivan Kelly <[email protected]> wrote: > > > >> Also, I don't think the shrink operation is necessary. The aim here is > >> to avoid the metadata spike, yes? If so, we can still roll the ledger > >> when it gets to a certain capacity, and this is unlikely to happen on > >> a spike. > >> > >
