I am thinking about changing it. I am going to send draft plan to you. It would be great for you to validate and change notes. Yixue
On Thu, Oct 18, 2012 at 12:22 AM, Ivan Kelly (JIRA) <[email protected]> wrote: > > [ > https://issues.apache.org/jira/browse/BOOKKEEPER-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478750#comment-13478750] > > Ivan Kelly commented on BOOKKEEPER-432: > --------------------------------------- > > We'd have to change to point at which entries are added to the index cache > also. It does add some complexity to the change, but it still keeps all the > changes in the entrylogger for now. Are you working on this at the moment? > I was thinking of picking this up next week. > > > Improve performance of entry log range read per ledger entries > > --------------------------------------------------------------- > > > > Key: BOOKKEEPER-432 > > URL: > https://issues.apache.org/jira/browse/BOOKKEEPER-432 > > Project: Bookkeeper > > Issue Type: Improvement > > Components: bookkeeper-server > > Affects Versions: 4.2.0 > > Environment: Linux > > Reporter: Yixue (Andrew) Zhu > > Labels: patch > > > > We observed random I/O reads when some subscribers fall behind (on some > topics), as delivery needs to scan the entry logs (thru ledger index), > which are interleaved with ledger entries across all ledgers being served. > > Essentially, the ledger index is a non-clustered index. It is not > effective when a large number of ledger entries need to be served, which > tend to be scattered around due to interleaving. > > Some possible improvements: > > 1. Change the ledger entries buffer to use a SkipList (or other > suitable), sorted on (ledger, entry sequence). When the buffer is flushed, > the entry log is written out in the already-sorted order. > > The "active" ledger index can point to the entries buffer (SkipList), > and fixed up with entry-log position once latter is persisted. > > Or, the ledger index can be just rebuilt on demand. The entry log file > tail can have index attached (light-weight b-tree, similar with big-table). > We need to track per ledger which log files contribute entries to it, so > that in-memory index can be rebuilt from the tails of corresponding log > files. > > 2. Use affinity concept to make ensembles of ledgers (belonging to same > topic) as identical as possible. This will help above 1. be more effective. > > > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira > -- Best Regards, yixue
